How mirroring works

One s3mirror run step by step, from the connection check to the exit code, and how the work is spread across threads.

A run is one pass over the source’s buckets. s3mirror keeps nothing between runs: every run lists both sides again and decides from those listings alone, so you can stop it, change the config and run it again without cleaning anything up.

One run

Connection check. It calls ListBuckets on the source and then on the destination. If either call fails, it logs the error code and message and exits 1 without changing anything. The destination’s bucket list isn’t used for anything else, but the call still has to succeed.
Bucket discovery. It lists the source’s buckets again and drops any named in exclude_buckets. The rest are handled one at a time, in the order the source returned them, which on most providers is alphabetical. If nothing is left, it logs No buckets found to sync and goes straight to the summary.
Destination bucket. For each bucket it calls HeadBucket at the destination. If that fails for any reason, it calls CreateBucket with the same name. If the create fails, the bucket is skipped, counted as an error, and the run moves on to the next one.
Listings. It lists every object in the bucket on both sides with ListObjectsV2, page by page, keeping each key and its size.
Comparison. A source object is copied if the destination has no object with that key, or has one of a different size. With deletion on, every destination key the source doesn’t have is marked for deletion. Change detection and Deletion go through both rules.
Copies. The copies run in parallel. When they’ve all finished, failed or not, the deletions run, also in parallel.
Summary. After the last bucket it prints how many buckets it processed and created, how many objects it copied and deleted, the bytes transferred and, when there were any, the average throughput and the error count. It exits 1 if it counted any errors and 0 otherwise.

How an object is copied

Each copy is a GetObject on the source whose body is streamed into an upload at the destination with boto3’s upload_fileobj. Nothing is written to local disk, but every byte goes through the machine running s3mirror, so its network link carries each object twice: once in from the source and once out to the destination. There’s no server-side copy, even when both endpoints are the same service.

Objects at or above multipart_threshold (8 MiB by default) are uploaded as multipart uploads in parts of multipart_chunksize. The upload only sets the bucket and key, so the destination object gets the destination’s defaults for everything else, such as storage class and encryption.

Both clients sign requests with Signature Version 4 and use path-style addressing (https://endpoint/bucket/key), which most S3-compatible services expect. botocore’s adaptive retry mode retries throttled and failed requests up to three times, so each request gets four attempts in all before s3mirror sees the error.

Concurrency

Three settings in the performance section decide how much runs at once:

Setting	Default	What it limits
`max_workers`	`20`	Objects copied, or deleted, at the same time within a bucket
`max_concurrency`	`10`	Threads each object’s upload can use for its parts
`max_pool_connections`	`50`	Open HTTP connections each of the two clients keeps

Buckets are always handled one after another; only the objects within a bucket run in parallel. Large objects are read from the source in parts and held in memory until they’re uploaded, so raising max_workers, max_concurrency and multipart_chunksize together raises memory use as well as throughput. --workers overrides max_workers for one run.

What a run counts as an error

The error count, and so the exit code, covers objects that failed to copy, objects that failed to delete, and destination buckets that couldn’t be created. A failed object listing isn’t counted: it logs the error and carries on as if the bucket were empty. That has consequences for deletion, which Deletion explains, and it means a run can report COMPLETED SUCCESSFULLY and exit 0 when a bucket wasn’t mirrored at all.

Read on

How mirroring worksChange detectionThe key-and-size rule, and what it doesn't notice.How mirroring worksDeletionWhich objects are deleted, when, and how to stop it.How mirroring worksSafetyPermissions, TLS and checks before a run that can delete.ReferenceLoggingEvery line a run prints, and where it goes.