How mirroring works

A run is one pass over the source’s buckets. s3mirror keeps nothing between runs: every run lists both sides again and decides from those listings alone, so you can stop it, change the config and run it again without cleaning anything up.

One run

1. Connection check. It calls ListBuckets on the source and then on the destination. If either call fails, it logs the error code and message and exits 1 without changing anything. The destination’s bucket list isn’t used for anything else, but the call still has to succeed.

2. Bucket discovery. It lists the source’s buckets again and drops any named in exclude_buckets. The rest are handled one at a time, in the order the source returned them, which on most providers is alphabetical. If nothing is left, it logs No buckets found to sync and goes straight to the summary. If this second ListBuckets call fails, that’s counted as an error and it goes straight to the summary too.

3. Destination bucket. For each bucket it calls HeadBucket at the destination. Only a 404, meaning the bucket doesn’t exist, leads to a CreateBucket with the same name, in the destination’s region_name. If HeadBucket fails any other way, such as a 403 for a bucket the credential can’t see, or the create fails, the bucket is skipped, counted as an error, and the run moves on to the next one.

4. Listings. It lists every object in the bucket on both sides with ListObjectsV2, page by page, keeping each key and its size. If either listing fails, on any page, the bucket is skipped with nothing copied or deleted, counted as an error, and the run moves on to the next one.

5. Comparison. A source object is copied if the destination has no object with that key, or has one of a different size. With deletion on, every destination key the source doesn’t have is marked for deletion. Change detection and Deletion go through both rules.

6. Copies. The copies run in parallel. When they’ve all finished, failed or not, the deletions run, also in parallel.

7. Summary. After the last bucket it prints how many buckets it processed and created, how many objects it copied and deleted, the bytes transferred and, when there were any, the average throughput and the error count. It exits 1 if it counted any errors and 0 otherwise.

How an object is copied

Each copy is a GetObject on the source whose body is streamed into an upload at the destination with boto3’s upload_fileobj. Nothing is written to local disk, but every byte goes through the machine running s3mirror, so its network link carries each object twice: once in from the source and once out to the destination. There’s no server-side copy, even when both endpoints are the same service.

Objects at or above multipart_threshold (8 MiB by default) are uploaded as multipart uploads in parts of multipart_chunksize, or bigger parts for an object that would otherwise need more than S3’s limit of 10,000. The upload only sets the bucket and key, so the destination object gets the destination’s defaults for everything else, such as storage class and encryption.

Both clients sign requests with Signature Version 4 and use path-style addressing (https://endpoint/bucket/key), which most S3-compatible services expect. botocore’s adaptive retry mode retries throttled and failed requests up to three times, so each request gets four attempts in all before s3mirror sees the error.

Concurrency

Three settings in the performance section decide how much runs at once:

Setting	Default	What it limits
max_workers	20	Objects copied, or deleted, at the same time within a bucket
max_concurrency	10	Threads each object’s upload can use for its parts
max_pool_connections	50	Open HTTP connections each of the two clients keeps

Buckets are always handled one after another; only the objects within a bucket run in parallel. Large objects are read from the source in parts and held in memory until they’re uploaded, so raising max_workers, max_concurrency and multipart_chunksize together raises memory use as well as throughput. --workers overrides max_workers for one run.

What a run counts as an error

The error count, and so the exit code, covers objects that failed to copy, objects that failed to delete, destination buckets that couldn’t be checked or created, buckets whose objects couldn’t be listed on either side, and a failed listing of the source’s buckets. A bucket that couldn’t be checked, created or listed is skipped, so a run that reports COMPLETED SUCCESSFULLY and exits 0 mirrored every bucket it found. Deletion explains why a failed listing never leads to deletions.

Read on

How mirroring worksChange detectionThe key-and-size rule, and what it doesn't notice.How mirroring worksDeletionWhich objects are deleted, when, and how to stop it.How mirroring worksSafetyPermissions, TLS and checks before a run that can delete.ReferenceLoggingEvery line a run prints, and where it goes.