#development #devops #golang #tools
If you need to either duplicate or sync one space on Digital Ocean with another, there's a great tool called rclone which makes this very straightforward.
To get started, you first need to install it:
brew install rclone
apt get rclone
This will give you access to a CLI tool named
The next step is to configure it to access your DO spaces. To do this, you'll need to create an API key for your DO account.
Once you did that, create a config directory and file:
mkdir -p ~/.config/rclone nano ~/.config/rclone/rclone.conf
Once that is done, add the following configuration to the file:
1[my-spaces] 2type = s3 3provider = DigitalOcean 4env_auth = false 5access_key_id = <access_key_id> 6secret_access_key = <secret_access_key> 7endpoint = ams3.digitaloceanspaces.com 8acl = private
This basically allows us to connect to the storage engine and use it as a source.
Now, let's you have a bucket called
my-source-bucket which you want to duplicate as
my-target-bucket, you can run
the subcommand called
rclone sync my-spaces:my-source-bucket my-spaces:my-target-bucket
This will create a full copy of
my-target-bucket if it doesn't exist yet. If it does, it will only copy the files that are changed. It's explained in the documentation as follows:
Sync the source to the destination, changing the destination only. Doesn't transfer files that are identical on source and destination, testing by size and modification time or MD5SUM. Destination is updated to match source, including deleting files if necessary (except duplicate objects, see below). If you don't want to delete files from destination, use the copy command instead.
If you want to see what it does before actually doing the copy / sync, you can use the
rclone sync my-spaces:my-source-bucket my-spaces:my-target-bucket --dry-run
If you want progress reports, you can use the
rclone sync my-spaces:my-source-bucket my-spaces:my-target-bucket --progress
Note that this also allows you to sync from one provider to another (e.g. from AWS S3 to DO Spaces or the other way around).
As rclone supports many different cloud providers, you need to check the documentation if you want to use another storage engine.
PS: in case you're interested, rclone is written in Go.
If this post was enjoyable or useful for you, please share it! If you have comments, questions, or feedback, you can email my personal email. To get new posts, subscribe use the RSS feed.