Difference between revisions of "IT/AWS/S3"
From Lewis Consultancy Wiki
Jump to navigationJump to search (Created page with "=Tag objects in an S3 bucket based on size= It's desirable to avoid archiving small files as not efficient. One approach is to tag the file based on then archive to glacier ba...") |
|||
| Line 8: | Line 8: | ||
cat /tmp/fullfilelist.txt | awk '{ if ($3> 5000000) print $4}' > /tmp/largefilenames.txt | cat /tmp/fullfilelist.txt | awk '{ if ($3> 5000000) print $4}' > /tmp/largefilenames.txt | ||
| − | ==Tag files (set tag archive=yes == | + | ==Tag files ( set tag archive=yes ) == |
cat /tmp/largefilenames.txt | while read fn; do aws s3api put-object-tagging --bucket inview-nas1 --key "$fn" --tagging '{"TagSet": [{ "Key": "archive", "Value": "yes" }]}'; done | cat /tmp/largefilenames.txt | while read fn; do aws s3api put-object-tagging --bucket inview-nas1 --key "$fn" --tagging '{"TagSet": [{ "Key": "archive", "Value": "yes" }]}'; done | ||
Revision as of 09:00, 1 September 2021
Contents
Tag objects in an S3 bucket based on size
It's desirable to avoid archiving small files as not efficient. One approach is to tag the file based on then archive to glacier based on tag.
Get full filelist of bucket and store
aws s3 ls s3://bucketname/path --recursive > /tmp/fullfilelist.txt
Select only files greater than x bytes
cat /tmp/fullfilelist.txt | awk '{ if ($3> 5000000) print $4}' > /tmp/largefilenames.txt
Tag files ( set tag archive=yes )
cat /tmp/largefilenames.txt | while read fn; do aws s3api put-object-tagging --bucket inview-nas1 --key "$fn" --tagging '{"TagSet": [{ "Key": "archive", "Value": "yes" }]}'; done
