Working with CGP Storage via Linux/FreeBSD CLI

Installing Google Cloud SDK on FreeBSD:

This is easily done via package or ports:

pkg install google-cloud-sdk

Installing Google Cloud SDK on Debian/Ubuntu:

Follow the instructions here which are summarized below

Add the Google Cloud SDK as a package source:

echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list

Install required dependencies:

sudo apt install apt-transport-https ca-certificates gnupg

Add Google Cloud public key:

curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
 

Install Google Cloud SDK:

sudo apt update
sudo apt install google-cloud-sdk

Prepping gCloud:

If a proxy server is required, set gcloud to use it:

gcloud config set proxy/type http
gcloud config set proxy/address 10.10.10.100
gcloud config set proxy/port 3128

Configure gCloud.  This will spit out a URL to paste in to browser, which will return an authorization code

gcloud init

This will generate an encrypted file ~/.gsutil/credstor that will be used for authentication.  To re-authenticate:

gcloud auth login

To switch to a different project:

glcoud config set project <PROJECT_ID>

To switch to a different account:

gcloud config set account

To use a service account:

gcloud auth activate-service-account <ACCOUNT_EMAIL> --key-file=<JSON_KEY_FILE>

CLI commands for working with Google Cloud Storage

List existing buckets

gsutil ls

Create a storage bucket called ‘mybucket’

gsutil mb gs://mybucket

Get information about a bucket called ‘mybucket’

gsutil ls -L -b gs://mybucket/

Upload a single file to the bucket

gsutil cp myfile gs://mybucket/

Upload a directory and its contents to a bucket

gsutil cp -r folder1 gs://code-j5-org/

List contents of a bucket

gsutil ls -r gs://mybucket/

Download a file called ‘testfile.png’ in ‘folder1’

gsutil cp gs://mybucket/folder1/testfile.png

Delete multiple files in a folder

gsutil rm gs://mybucket/folder1/*.png

Delete a folder and all its contents

gsutil rm -r gs://mybucket/folder1

Delete a bucket, if bucket is empty

gsutil rb gs://mybucket

Delete a bucket all all files

gsutil rm -r gs://mybucket

Accessing buckets via HTTPS

asdflkj

curl -X POST --data-binary @[OBJECT_LOCATION] \
-H "Authorization: Bearer [OAUTH2_TOKEN]" \
-H "Content-Type: [OBJECT_CONTENT_TYPE]" \
"https://storage.googleapis.com/upload/storage/v1/b/[BUCKET_NAME]/o?uploadType=media&name=[OBJECT_NAME]"

To download files, buckets can be accessed at https://<bucket name>.storage.googleapis.com/path   For example,

curl https://mybucket.storage.googleapis.com/folder1/testfile.png

Within GCP for subnets that have “Private google access”, this DNS name will always resolve to 199.36.153.8-11

Advertisement

Using AWS S3 Storage from Linux CLI

Start by installing aws-shell, then run the configure command to enter key and region information:

sudo apt install aws-shell
aws configure

To list files in a bucket called ‘mybucket’:

aws s3 ls s3://mybucket

To upload a single file:

aws s3 cp /tmp/myfile.txt s3://mybucket/

To upload all files in a directory with a certain extension:

aws s3 cp /tmp/ s3://mybucket/ --recursive --exclude '*' --include '*.txt'

To recursively upload contents of a directory:

aws s3 cp /tmp/mydir/ s3://mybucket/ --recursive

To delete a single file:

aws s3 rm s3://mybucket/myfile.text

To empty a bucket (delete all files, but keep bucket):

aws s3 rm s3://mybucket --recursive

 

Adding a swap file to a t2.nano in AWS running Ubuntu 18

I recently moved my Cacti/Rancid/Ansible Linux VM to a t2.nano in AWS. With only 500 MB of RAM, I knew there would be some performance limitations, but  what I didn’t realize is by default, the instance had no swap configured.  A MariaDB server consumes ~200 MB of memory when running, and sure enough, mysqld died after a few days uptime:

Apr 20 15:42:20 nettools kernel: [351649.590161] Out of memory: Kill process 27535 (mysqld) score 491 or sacrifice child
Apr 20 15:42:20 nettools kernel: [351649.598181] Killed process 27535 (mysqld) total-vm:1168184kB, anon-rss:240496kB, file-rss:0kB, shmem-rss:0kB
Apr 20 15:42:20 nettools kernel: [351649.676624] oom_reaper: reaped process 27535 (mysqld), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

So I wanted to add a 1GB swap file so that any memory-heavy processes would be happy and stable.  It was easy enough to find a blog post that outlined creating the swapfile:

# Become root
sudo su

# Create an empty 1 GB (1MB x 1024) file called /swap.img
dd if=/dev/zero of=/swap.img bs=1M count=1024

# Set recommended permissions
chmod 600 /swap.img

# Convert it to usable swap
mkswap /swap.img

Many of these posts were neglecting how to make the swap activated automatically at boot time.  To do so, add this line to bottom of /etc/fstab

/swap.img swap swap defaults 0 0

The swap file can be activated immediately with this command:

swapon -a

Or, give it a test reboot and verify it’s being activated automatically at startup:

ubuntu@linux:~$ top
top - 16:55:04 up 18 min,  1 user,  load average: 0.00, 0.01, 0.03
Tasks: 108 total,   1 running,  71 sleeping,   0 stopped,   0 zombie
%Cpu(s): 10.3 us,  6.0 sy,  0.0 ni, 80.8 id,  3.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :   491200 total,    21040 free,   329668 used,   140492 buff/cache
KiB Swap:  1048572 total,   992764 free,    55808 used.   140596 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                                                                                    
  905 mysql     20   0 1166460 160616  10508 S  4.7 32.7   0:04.90 mysqld                                                                                                     
 1781 www-data  20   0  283268  29696  18944 S  1.7  6.0   0:00.05 php                                                                                                        
 1785 www-data  20   0  289512  30488  18688 S  1.7  6.2   0:00.05 php                                                                                                        
   35 root      20   0       0      0      0 S  1.0  0.0   0:00.45 kswapd0                                                                                                    
  967 www-data  20   0  481904  22936  18432 S  0.3  4.7   0:00.16 apache2                                                                                                    
    1 root      20   0  225264   8408   6792 S  0.0  1.7   0:02.56 systemd