How mirroring worksSafety
Safety
The permissions each side needs, TLS verification, where the secrets live, and the checks to make before a run that can delete.
s3mirror writes to the destination and, with deletion on, removes from it. It can’t change anything at the source: it only lists and reads there. The risks are all on the destination side, and most of them come down to three things: the credentials you give it, whether TLS is verified, and what you check before letting it delete.
Permissions
Section titled “Permissions”Give s3mirror its own credential on each side, with only what a mirror needs. In AWS IAM terms:
| Side | Actions | Used for |
|---|---|---|
| Source | s3:ListAllMyBuckets | The connection check and bucket discovery |
| Source | s3:ListBucket on each mirrored bucket | Listing its objects |
| Source | s3:GetObject on each mirrored bucket’s objects | Reading each object it copies |
| Destination | s3:ListAllMyBuckets | The connection check |
| Destination | s3:ListBucket on each mirrored bucket | HeadBucket and listing its objects; without it, the bucket is skipped |
| Destination | s3:CreateBucket | Only if s3mirror should create missing buckets |
| Destination | s3:PutObject on each mirrored bucket’s objects | Single and multipart uploads |
| Destination | s3:AbortMultipartUpload on each mirrored bucket’s objects | Cleaning up a multipart upload that failed |
| Destination | s3:DeleteObject on each mirrored bucket’s objects | Only if deletion is on |
Other providers name these differently, but most offer the same split between listing, reading,
writing and deleting. s3mirror never writes to the source, but a read-only source credential still
matters: if the source and destination sections are ever swapped by mistake, it’s what stops the
run overwriting and deleting your real data. A destination credential without s3:DeleteObject turns deletion into
logged errors rather than lost data, and each refused delete is counted, so the run exits 1.
Make sure the source credential can list every bucket it can see. A bucket that shows up in
ListBuckets but can’t be listed is skipped, with nothing copied or deleted, and counted as an
error on every run; Deletion has the
details.
verify_ssl defaults to true for both endpoints, so s3mirror checks each endpoint’s certificate
against the system’s trusted CAs and refuses to connect if it doesn’t verify. That’s what any hosted
provider needs. For a private CA, the value is passed straight to boto3 as its verify argument,
so a path to a CA bundle works too. For a self-signed certificate, the certificate file itself
usually works as the bundle:
destination: endpoint_url: "https://minio.internal.example:9000" aws_access_key_id: "DEST_ACCESS_KEY" aws_secret_access_key: "DEST_SECRET_KEY" verify_ssl: "/etc/ssl/certs/internal-ca.pem"verify_ssl: false turns the check off for that endpoint, so s3mirror accepts any certificate,
including one presented by something in the middle. Use it only when you can’t get the CA bundle,
and only on a network you trust. Each run then logs
⚠ TLS certificate verification is off for the source (or destination) once, after the banner,
in place of the warning urllib3 would otherwise print for every request.
Secrets
Section titled “Secrets”Both secret keys live in the config file in plain text. Keep it readable only by the account that
runs s3mirror (chmod 600), and out of version control.
The secret keys are never logged, and --show-config replaces them with ***REDACTED***. Access
key IDs aren’t secret in the same way, but they’re still worth keeping out of a pasted log or an
issue: on AWS, anyone with an AWS account of their own can look up which account a full key ID
belongs to. So debug output, log files and --show-config show only the first and last four
characters, such as AKIA...MPLE. That’s enough to tell which key is in use: on AWS the first four say what kind of key it is (AKIA
for a long-term key, ASIA for a temporary one) and the last four are what aws configure list
shows. A key ID shorter than 20 characters, as MinIO and some other services allow, is shown as
***, because four and four would be most of it. The placeholders, such as SOURCE_ACCESS_KEY, are
shown in full, since they aren’t credentials and seeing one means the key was never set.
Before a run that can delete
Section titled “Before a run that can delete”- Run with
--no-delete --debugand read the output: the bucket list, the object counts on each side and theActions:line for each bucket. - Add any bucket the destination should keep to itself, or that’s temporary or test-only, to
exclude_buckets. Buckets that exist only at the destination are already safe. - Turn on versioning at the destination, or keep an independent backup of anything you can’t
lose. s3mirror’s deletions are plain
DeleteObjectcalls. - Try new endpoints and credential changes against a non-production bucket first.
- For scheduled runs, use
--log-fileand alert on a non-zero exit code.