Let’s explore the differences between S3
sync command. We will look into various areas and discuss the difference.
- Use Case
cp – Copy all the files to the new location. It will not check if there is an existing file and will just overwrite with the file from source location. The files at the new location will stay even if it does not exist in the source location.
sync – Will check the new location if the file already exist. This operation will only copy the file and update it if the file is not found or updated compared to source location. There is also an option to delete files not found in the source location.
cp – Will read all the files from the source location and write into the new location.
sync – Will scan the new location and only overwrite the file from the source location if the file is newly created or updated(via file size and modified timestamp comparison).
From the looks base on the operation, sync is much more efficiency as it will save time by not copying the files that is the same between source and new location.
cp – We can use
cp if we want to just copy a few files from source location to new location and no need them to be at the same state.
sync – We can use
sync is we need the folders and files to be in the same state between source location and new location
S3 cp and sync Examples
Please ensure that you have the latest version of AWS CLI installed by referring to AWS documentation.
# Copy all the files from 1 folder to the other recursively aws s3 cp s3://my-bucket-old/folder s3://my-bucket-new/folder --recursive # Sync the 2 locations while removing files that does not exist in source location aws s3 sync s3://my-bucket-old/folder s3://my-bucket-new/folder --delete
Although it may seems that
sync is much more efficient compare to
cp command, deciding on which to use depends on the use case. If only copy a few files, we can use
cp. If we need to have the 2 location having the same state, then we will use