Configuration

s3mirror reads one config file, in YAML or JSON, with four sections: source, destination, performance and sync. You pass it with --config; there’s no default path and no environment variables, so without --config s3mirror runs with built-in placeholders that point at s3.source.example.com and fail the connection check. Config keys lists every key with its default; this page is about putting a file together.

The file format

The format comes from the extension: .yaml or .yml is read as YAML, .json as JSON. Anything else stops with Error: Unsupported config format. Use .yaml, .yml or .json and exit code 1. The file has to hold a mapping at the top level. An empty YAML file is read as an empty mapping, which leaves every default in place, placeholders included.

The same config in JSON:

config.json

{
  "source": {
    "endpoint_url": "https://s3.source.example.com",
    "aws_access_key_id": "SOURCE_ACCESS_KEY",
    "aws_secret_access_key": "SOURCE_SECRET_KEY",
    "verify_ssl": true
  },
  "destination": {
    "endpoint_url": "https://s3.destination.example.com",
    "aws_access_key_id": "DEST_ACCESS_KEY",
    "aws_secret_access_key": "DEST_SECRET_KEY",
    "verify_ssl": true
  }
}

How your file is merged

s3mirror starts from its defaults and merges your file over them one section at a time. Within a section, a key you set replaces the default and a key you leave out keeps it. So this sets max_workers and leaves the other four performance keys at their defaults:

performance:
  max_workers: 8

The merge goes one level deep, and then the result is checked. A section or key s3mirror doesn’t know, such as a misspelt delete_extranous, or a value of the wrong type stops it before it connects, with exit code 1 and every problem listed, so a typo can’t quietly leave deletion on. Run --show-config after any change to see what s3mirror will really use:

s3mirror --config config.yaml --show-config

It prints the merged configuration as JSON, with "false" and "true" strings read as booleans, after --workers and --no-delete are applied, with both secret keys replaced by ***REDACTED*** and the access key IDs shortened to their first and last four characters, and exits 0 without connecting to anything. A value s3mirror can’t read is reported as a config error instead, with exit code 1.

The two endpoints

source and destination take the same five keys. endpoint_url is the service’s S3 API address, including the scheme and any port:

config.yaml

source:
  endpoint_url: "https://s3.eu-west-2.amazonaws.com"
  aws_access_key_id: "SOURCE_ACCESS_KEY"
  aws_secret_access_key: "SOURCE_SECRET_KEY"
  region_name: "eu-west-2"
  verify_ssl: true

destination:
  endpoint_url: "https://minio.internal.example:9000"
  aws_access_key_id: "DEST_ACCESS_KEY"
  aws_secret_access_key: "DEST_SECRET_KEY"
  verify_ssl: "/etc/ssl/certs/internal-ca.pem"

region_name defaults to us-east-1. Many S3-compatible services ignore it, but it’s part of every request signature, so set it to whatever region the provider tells you to use. On AWS, use the regional endpoint and the matching region. The destination’s region_name is also where s3mirror creates any bucket that’s missing there: outside us-east-1 it’s sent as the bucket’s location constraint, as AWS requires.

s3mirror uses path-style addressing for both endpoints, so the endpoint URL is the service’s root, never a bucket’s own hostname.

The keys are passed to boto3 exactly as written. If you leave one out, its placeholder default stays and the connection check fails. If you set both keys on a side to null, boto3 falls back to its usual credential chain for that client (environment variables, ~/.aws/credentials, an instance role). The chain is the same for both sides, so that only makes sense for one of them. Setting just one of the pair to null is a config error.

verify_ssl defaults to true, which checks the endpoint’s certificate against the system’s trusted CAs. For a private CA or a self-signed certificate, set it to the path of the CA bundle, as the destination above does; Safety explains when false is acceptable.

Choosing what’s mirrored

Every source bucket the source credential can list is mirrored, under the same name, unless you exclude it:

sync:
  exclude_buckets:
    - scratch-bucket
    - temporary-exports

Names must match exactly; there are no patterns or prefixes. exclude_buckets has to be a list, even for one bucket, and a plain string is a config error. Excluded buckets are skipped on both sides, so their destination copies are neither updated nor deleted from. The run logs Excluding 2 bucket(s): scratch-bucket, temporary-exports when any of them are on the source.

There’s no way to mirror only part of a bucket, or to rename a bucket on the way across. To mirror a few buckets out of many, give the source credential access to only those, since s3mirror mirrors whatever ListBuckets returns.

Deletion

sync:
  delete_extraneous: false

delete_extraneous defaults to true, so leave it out and runs delete destination objects the source doesn’t have. A bare false and a quoted "false" both turn it off, and a value that’s neither true nor false, such as 0 or "no", stops s3mirror with a config error rather than being guessed at. --no-delete turns deletion off for one run whatever the file says. Read Deletion before you leave it on.

Tuning

The defaults copy 20 objects at once, upload anything from 8 MiB up in 8 MiB parts with up to ten threads each, and keep up to 50 connections open per endpoint:

performance:
  max_workers: 20
  multipart_threshold: 8388608
  multipart_chunksize: 8388608
  max_concurrency: 10
  max_pool_connections: 50

All five are whole numbers of at least 1; sizes are in bytes. Some starting points:

• Many small objects: raise max_workers, and raise max_pool_connections with it so each worker can keep a connection.
• A few very large objects: max_workers matters less than max_concurrency and multipart_chunksize, because each large object is split into parts and the parts run in parallel. S3 allows 10,000 parts per upload, which at the default 8 MiB is about 78 GiB, so for a larger object s3mirror raises the part size to the smallest whole number of MiB that fits: a 100 GiB object goes up in 11 MiB parts, a 1 TiB one in 105 MiB parts. You don’t need to change multipart_chunksize for big objects, only if you want bigger parts for all of them.
• A rate-limited or shared endpoint: lower max_workers, or pass --workers for one run.
• Memory: each large object being uploaded can hold up to ten parts in memory at once, so with 20 workers all on large objects and 8 MiB parts that’s up to about 1.6 GiB. Lower max_workers or multipart_chunksize on a small machine. Raised part sizes raise it too: the 1 TiB object above can hold about 1 GiB on its own.

Several mirrors

One config file describes one source and one destination. For several pairs, write one file per pair and run s3mirror once for each, one after another, so they don’t compete for bandwidth. Give each its own --log-file.