In this article we will see how to automate back from our local machine to AWS S3 with sync command. Requirements:AWS Cli should be configured on Local Linux MachineInternet Connectivity [Of course] In our previous posts we saw how to install and configure AWS Cli on local machine. If you are don’t know how to do it please check my previous posts. Steps to Automate Backup: Create S3 Bucket where we would be sending our data for backup Using AWS Cli to issue the command to backup Using Cron to schedule our backup So let’s start: If you remember from our last post we already created S3 bucket by the name of tekco2020 as you can see in below image, we can use it to backup our data. Folder or directory on our local computer which we would like to backup is /opt/tekco-backup root@Red-Dragon:/opt/tekco-backup# pwd /opt/tekco-backup root@Red-Dragon:/opt/tekco-backup# ls fb-https.txt itpings-curl.txt password-curl-header.txt tekco.net-https-info.txt itpings-curl-header.txt password-2-curl-header.txt tekco.net-https-info2.txt As we can see above that there is data available in our folder tekco-backup Now issue the below command to start backing up data to AWS S3 bucket root@Red-Dragon:/opt/tekco-backup# aws s3 sync . s3://tekco2020 upload: ./itpings-curl.txt to s3://tekco2020/itpings-curl.txt upload: ./itpings-curl-header.txt to s3://tekco2020/itpings-curl-header.txt upload: ./password-curl-header.txt to s3://tekco2020/password-curl-header.txt upload: ./password-2-curl-header.txt to s3://tekco2020/password-2-curl-header.txt upload: ./tekco.net-https-info2.txt to s3://tekco2020/tekco.net-https-info2.txt upload: ./tekco.net-https-info.txt to s3://tekco2020/tekco.net-https-info.txt upload: ./fb-https.txt to s3://tekco2020/fb-https.txt root@Red-Dragon:/opt/tekco-backup# We can see that the files were successfully uploaded , let’s confirm it from AWS S3 Console Great, we can see that our data is in the bucket. Now let’s create a file and rerun the same commad. root@Red-Dragon:/opt/tekco-backup# touch sal.txt root@Red-Dragon:/opt/tekco-backup# aws s3 sync . s3://tekco2020 upload: ./sal.txt to s3://tekco2020/sal.txt As we can see , this time only new file got copied to S3 bucket , lets confirm it from AWS S3 console. Great ! Now to Automate the backup task we will setup cron. For this demo i will set it up to backup every 1 min. So let’s do it. root@Red-Dragon:/opt/tekco-backup# crontab -e Add the following lines to cron */1 * * * * /usr/local/bin/aws s3 sync /opt/tekco-backup/ s3://tekco2020 Save and quit and list with the following command root@Red-Dragon:/opt/tekco-backup# crontab -l m h dom mon dow command */1 * * * * /usr/local/bin/aws s3 sync /opt/tekco-backup/ s3://tekco2020 Now restart cron , copy some files to tekco-backup folder and wait for a min to see if the backup starts automatically. root@Red-Dragon:/opt/tekco-backup# systemctl restart cron root@Red-Dragon:/opt/tekco-backup# systemctl status cron ● cron.service – Regular background program processing daemon Loaded: loaded (/lib/systemd/system/cron.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2020-10-06 16:40:11 +06; 5s ago Docs: man:cron(8) Main PID: 18618 (cron) Tasks: 1 (limit: 18972) Memory: 520.0K CGroup: /system.slice/cron.service └─18618 /usr/sbin/cron -f Oct 06 16:40:11 Red-Dragon systemd[1]: Started Regular background program processing daemon. Oct 06 16:40:11 Red-Dragon cron[18618]: (CRON) INFO (pidfile fd = 3) Oct 06 16:40:11 Red-Dragon cron[18618]: (CRON) INFO (Skipping @reboot jobs — not system startup) root@Red-Dragon:/opt/tekco-backup# touch mynewfile-aftercron.txt Now after 1 min , let’s check our bucket root@Red-Dragon:/opt/tekco-backup# aws s3 ls s3://tekco2020 PRE .aptitude/ PRE .aws/ PRE .cache/ PRE .config/ PRE .dbus/ PRE .local/ PRE .ssh/ PRE .synaptic/ 2020-10-06 16:39:05 11147 .bash_history 2020-10-06 16:39:05 3106 .bashrc 2020-10-06 16:39:10 31 .lesshst 2020-10-06 16:39:10 161 .profile 2020-10-06 16:39:11 75 .selected_editor 2020-10-06 16:39:13 12103 .viminfo 2020-01-04 21:39:25 70522 Kubernetes-Install.pdf 2020-10-06 15:27:26 148119 fb-https.txt 2020-10-06 15:27:26 384 itpings-curl-header.txt 2020-10-06 15:27:26 53772 itpings-curl.txt 2020-10-06 16:59:06 0 mynewfile-aftercron.txt 2020-10-06 15:27:26 242 password-2-curl-header.txt 2020-10-06 15:27:26 187 password-curl-header.txt 2020-10-06 16:29:05 0 sal.txt 2020-10-06 15:27:26 71968 tekco.net-https-info.txt 2020-10-06 15:27:26 71968 tekco.net-https-info2.txt Perfect, we can see our cron is working and file has been copied. Lets add one more file “dragon.txt” and check after 1 min from AWS S3 Console root@Red-Dragon:/opt/tekco-backup# touch dragon.txt Great our Automatic Backup is working every , minute , you can adjust it as per your requiremets. Thanks, Salman A. Francishttps://www.youtube.com/linuxkinghttps://www.tekco.net
Connecting S3 from AWS CLI Version 2
In this post we will see What is S3 How to Create User in AWS How to Configure AWS Cli Version 2 to list S3 buckets and content So Let’s start: S3:Stands for Simple Storage Service. It is a Cloud based Storage Service which can hold unlimited amount of data and the data can be retrieved anytime over the web.This data is stored in a general graphic locations called regions.These regions contain Availability zones. These AZ’s are isolated facilities.**Note: Data in S3 is replicated to 3 AZ’s and Amazon handles this automatically.S3 is object based storage e.g it stores files , jpegs, pictures, pdf but not run applications. For apps we use Block devices such as EBS. These objects are stored in bucket and buckets are created within the region. Object can be upto 5 TB in size. We can set permission on objects as well as on the buckets.Storage Types:Standard S3S3 IA [ Infrequent Access]S3 Single Zone Glacier Now to setup AWS Cli to access S3 content: 1) Create an IAM User2) Give this user S3 -Access let’s make him S3Admin3) Get his Access key and Secret key from Security Credentials Creating an IAM user:To create an IAM user login to AWS Console and type IAM in the search bar as shown below and click on IAM. Now Click on Users as show below Now click on Add user Then give details as show below Note: You can Check Access Key box here or later. Now Click Next and then Click on Create group. Now give group a name e.g “tekco-s3 ” as show below and from policy filter type S3 and select AmazonS3FullAccess Then click Next and click on create group. Once you have clicked on create the group, you will be presented with the following screen. If all is as per your desire click finish. Click next and you will receive a success message for your new user. Now once the user is created, click on Security Credentials and , then create “Access Key” Once you click on Create access key , You will see Access key and Security key. Please save these keys and move to next step of Configuring AWS Cli Version 2 Now Let’s Configure our AWS Cli Version 2 To configure aws cli to access S3 go to your linux terminal and type the following: root@Red-Dragon:/home/salman# aws configure It will ask you to provide with Access key, Secret key and the region ( You have all the information when you created the user earlier) AWS Access Key ID [None]: AKIAYQYYFIDYUMHTUMHC AWS Secret Access Key [None]: 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 Default region name [None]: us-east-1 Default output format [None]: root@Red-Dragon:/home/salman# Thats it, our AWS Cli is configured for S3, to confirm that issue the following commands: root@Red-Dragon:/home/salman# aws s3 ls 2019-11-28 00:08:55 itpings2000 2020-01-04 21:37:18 tekco2020 root@Red-Dragon:/home/salman# aws s3 ls s3://tekco2020 2020-01-04 21:39:25 70522 Kubernetes-Install.pdf We can see that the content is listed and our bucket tekco2020 contains Kubernetes-install.pdf , let’s verify this by checking S3 from AWS S3 console As we can see above it does shows the same content as seen from Cli. Let’s create a bucket from AWS CLI with aws s3 mb command. root@Red-Dragon:/home/salman# aws s3 mb s3://tekco-demo-2020 make_bucket: tekco-demo-2020 root@Red-Dragon:/home/salman# aws s3 ls 2019-11-28 00:08:55 itpings2000 2020-10-01 23:47:17 tekco-demo-2020 2020-01-04 21:37:18 tekco2020 Let’s go to our AWS S3 Console and check if the bucket is being created. Perfect, we have successfully created our bucket from AWS Cli. Thanks, Salman Francishttps://www.youtube.com/linuxkinghttps://www.tekco.net
