Category: Linux

Installing AWS CLI Tools v2 on FreeBSD 12.4

J5Leave a comment

I upgraded my FreeBSD VM from 11 to 12 last weekend. Installing Google Cloud SDK was no problem; just use the FreeBSD package:

pkg install google-cloud-sdk

But for AWS CLI tools, there’s only a package for AWS CLI Tools version 1 . CLI Tools version 2 has been out for 3 years now, so we really don’t want to still be using v1.

Usually there’s a simple work-around: since AWS CLI tools is mostly Python scripts, you can install it via PIP. First, verify the version of Python installed, then install PIP3 for that version:

python -V
Python 3.9.16

pkg install py39-pip

Then install AWS CLI Tools v2 via PIP:

pip install awscliv2

But when we go to complete the install, we get this error:

awscliv2 --install
09:27:26 - awscliv2 - ERROR - FreeBSD amd64 is not supported, use docker version

This is because AWS CLI v2 does rely on a binary, and is only compiled for Linux. We can work around this by enabling Linux emulation.

Activating Linux Emulation in FreeBSD

First, add the following lines to /etc/rc.conf

linux_enable="YES"

Then either run this command, or simply reboot:

service linux start

Also install the CentOS 7 base from packages:

pkg install linux_base-c7

Completing install of AWS CLI Tools v2

Add the following line near the bottom of /usr/local/lib/python3.9/site-packages/awscliv2/installers.py to allow it to support FreeBSD:

    if os_platform == "FreeBSD" and arch == "amd64":
        return install_linux_x86_64()

Now we can complete the install successfully:

arnie@freebsd:~ % awscliv2 --install
10:07:13 - awscliv2 - INFO - Installing AWS CLI v2 for Linux
10:07:13 - awscliv2 - INFO - Downloading package from https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip to /tmp/tmpan_dsh2c.zip
10:07:17 - awscliv2 - INFO - Extracting /tmp/tmpan_dsh2c.zip to to /tmp/tmpnwvl45tn
10:07:26 - awscliv2 - INFO - Installing /tmp/tmpnwvl45tn/aws/install to /home/arnie/.awscliv2
10:08:37 - awscliv2 - INFO - Now awsv2 will use this installed version
10:08:37 - awscliv2 - INFO - Running now to check installation: awsv2 --version

Verify the binary and libraries are installed correctly:

~/.awscliv2/v2/current/dist/aws --version
aws-cli/2.11.10 Python/3.11.2 Linux/3.2.0 exe/x86_64.centos.7 prompt/off

You’ll probably want to include this directory in your path. Since I use TCSH, I do this by adding this line to ~/.cshrc:

set path = (/sbin /bin /usr/sbin /usr/bin /usr/local/sbin /usr/local/bin $HOME/bin $HOME/.awscliv2/v2/2.11.10/dist)

You’re now ready to configure aws cli tools v2. Run this command:

aws configure

Or, just manually setup the files ~/.aws/config and ~/.aws/credentials. Then try out a command. 

aws s3 ls

Use the AWS_PROFILE and AWS_REGION environment variable to override the defaults configured in ~/.aws/config

Advertisement

Using GCP Ops Agent to view Squid Logs

J5Leave a comment

The VMs were deployed via Terraform using instance templates, managed instance groups, and an internal TCP/UDP load balancer with a forwarding rule for port 3128. Debian 11 (Bullseye) was selected as the OS because it has a low memory footprint while still offering an nice pre-packaged version of Squid version 4.

The first problem is the older stackdriver agent isn’t compatible with Debian 11. So I had to install the newer one. I chose to just add these lines to my startup script, pulling the script directly from a bucket to avoid the requirement of Internet access:

gsutil cp gs://public-j5-org/add-google-cloud-ops-agent-repo.sh /tmp/
bash /tmp/add-google-cloud-ops-agent-repo.sh --also-install

After re-deploying the VMs, I ssh’d in and verified the Ops agent was installed and running:

sudo systemctl status google-cloud-ops-agent"*"

google-cloud-ops-agent-opentelemetry-collector.service - Google Cloud Ops Agent - Metrics Agent
     Loaded: loaded (/lib/systemd/system/google-cloud-ops-agent-opentelemetry-collector.service; static)
     Active: active (running) since Fri 2023-02-10 22:18:17 UTC; 18min ago
    Process: 4317 ExecStartPre=/opt/google-cloud-ops-agent/libexec/google_cloud_ops_agent_engine -service=otel -in /etc/google-cloud-ops-agent/config.yaml -logs ${LOGS_DIRECTORY} (code=exited, status=0/>
   Main PID: 4350 (otelopscol)
      Tasks: 7 (limit: 1989)
     Memory: 45.7M
        CPU: 1.160s

After waiting a couple minutes, I still didn’t see anything, so I downloaded and ran their diagnostic script:

gsutil cp gs://public-j5-org/diagnose-agents.sh /tmp/ && bash /tmp/diagnose-agents.sh

This was confusing because while it didn’t show any errors, the actual log was dumped to disk in a sub-directory of /var/tmp/google-agents/. and did indicate a problem in the agent-info.txt file:

API Check - Result: FAIL, Error code: LogApiPermissionErr, Failure:
 Service account is missing the roles/logging.logWriter role., Solution: Add the roles/logging.logWriter role to the Google Cloud service account., Res
ource: https://cloud.google.com/stackdriver/docs/solutions/agents/ops-agent/authorization#create-service-account

And this made sense, because in order for Ops Agent to function, it needs these two IAM roles enabled for the service account:

  • Monitoring > Monitoring Metric Writer.
  • Logging > Logs Writer.

Here’s a Terraform snippet that will do that:

# Add required IAM permissions for Ops Agents
locals {
  roles = ["logging.logWriter", "monitoring.metricWriter"]
}
resource "google_project_iam_member" "default" {
  for_each = var.service_account_email != null ? toset(local.roles) : {}
  project  = var.project_id
  member   = "serviceAccount:${var.service_account_email}"
  role     = "roles/${each.value}"
}

Within a few minutes of adding these, data started showing up in the graphs.

Configure Squid for HTTPS on Debian VM

J5Leave a comment

Verify we’re running the latest version of Debian

lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 11 (bullseye)
Release:	11
Codename:	bullseye

Become root

sudo su

Update packages

apt update && apt upgrade -y

Install the Squid package that has openssl configured and enabled

apt install squid-openssl

Create a local CA, using a 4096-bit key and SHA-2 hashing. This one is good for the next 10 years

openssl req -new -newkey rsa:4096 -sha256 -days 3653 -nodes -x509 -keyout /etc/squid/CA.key -out /etc/squid/CA.crt

Combine the key and cert in to a single file for convenience

cat CA.key CA.crt > CA.pem

Initialize the directory used for minted certs and set permissions so squid owns it

/usr/lib/squid/security_file_certgen -c -s /var/spool/squid/ssl_db -M 4MB
chown -R proxy:proxy /var/spool/squid

Finally, configure Squid to use HTTPS

http_port 3128 ssl-bump cert=/etc/squid/CA.pem generate-host-certificates=on options=NO_SSLv3
ssl_bump bump all

Restart Squid

service squid restart

Test connections by configuring 3128. Note the certificate from the CA, good for 10 years:

export https_proxy=http://localhost:3128

curl -v --cacert CA.crt  https://teapotme.com 

* Uses proxy env variable https_proxy == 'http://localhost:3128'
*   Trying ::1:3128...
* Connected to localhost (::1) port 3128 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to teapotme.com:443
> CONNECT teapotme.com:443 HTTP/1.1
> Host: teapotme.com:443
> User-Agent: curl/7.74.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< 
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: CA.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=teapotme.com
*  start date: Nov  6 04:03:48 2022 GMT
*  expire date: Nov  6 04:03:48 2032 GMT
*  subjectAltName: host "teapotme.com" matched cert's "teapotme.com"
*  issuer: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd; CN=localhost
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: teapotme.com
> User-Agent: curl/7.74.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Mark bundle as not supporting multiuse
< HTTP/1.1 418 I'm a teapot
< Server: nginx
< Date: Sun, 06 Nov 2022 04:08:13 GMT
< Content-Type: application/json
< Content-Length: 483
< X-Cache: MISS from test-1
< X-Cache-Lookup: MISS from test-1:3128
< Via: 1.1 test-1 (squid/4.13)
< Connection: keep-alive
< 
{
    "host": "teapotme.com",
    "user-agent": "curl/7.74.0",
    "x-forwarded-for": "::1, 35.233.234.155, 172.17.0.1",
    "x-forwarded-proto": "https",
}

Git clone / pull / push fails with ‘no mutual signature algorithm’ on Ubuntu 22 to GCP Cloud Source

J5Leave a comment

I created a new Ubuntu 22 VM a few weeks ago and noticed when trying a git pull or git push to a GCP Cloud Source Repo, I wasn’t having any luck when using SSH:

cd myrepo/
git pull
myusername@myorg.com@source.developers.google.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

The SSH key was a standard RSA with the public key uploaded to Cloud Source SSH Keys, so there was no obvious reason why it wasn’t working.

Next step was try and get some type of debug or error message as to why the public key exchange wasn’t working. Newer versions of Git can turn on SSH debugging by setting the GIT_SSH_COMMAND environment variable, so I did that:

export GIT_SSH_COMMAND="ssh -vvv"

When re-running the git pull request, I get some somewhat useful debugs back:

debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/j5/.ssh/id_rsa RSA SHA256:JBgC+R4Ozel+YI+7oEv1UOf9/jLqGBhysN8bpoEDbPU
debug1: send_pubkey_test: no mutual signature algorithm

The ‘no mutual signature algorithm’ indicated one side didn’t like the signing algorithm. I did a Google and found this article which indicates that Ubuntu 22 doesn’t allow RSA by default. I can’t change the setting on the Cloud Source side, so on the Ubuntu 22 client, I did this as a quick work-around:

echo "PubkeyAcceptedKeyTypes +ssh-rsa" > /etc/ssh/ssh_config.d/enable_rsa.conf

And now the git pull/push works without issue.

An alternate solution is instead use Elliptic Curve DSA rather than RSA. To generate a new ECDSA key:

ssh-keygen -t ecdsa
cat ~/.ssh/id_ecdsa.pub

Then copy/paste the key in to the SSH Key Manager. This will be easier to copy/paste then RSA since it’s shorter.

Clearing out /var/spool/clientmqueue in FreeBSD

J5Leave a comment

My FreeBSD VM with its 10GB virtual hard disk ran out of space today. The primary culprit was /var/spool/clientmqueue consuming nearly 3GB of space:

# du -d 1 /var/spool/
8	/var/spool/output
4	/var/spool/opielocks
2955904	/var/spool/clientmqueue
4	/var/spool/dma
4	/var/spool/lpd
4	/var/spool/lock
4	/var/spool/mqueue
2955936	/var/spool/

But when trying to just delete the files, I got “argument list too long”:

# rm -f /var/spool/clientmqueue/*
/bin/rm: Argument list too long.

In the Google search I learned something interesting: find has a -delete option. This worked well:

# find /var/spool/clientmqueue -name '*' -delete

Rancid: no matching key exchange method found. Their offer: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1

J5Leave a comment

Time to move Rancid to a newer VM again, this time it’s Ubuntu 20. Hit a snag when I tried a test clogin run:

$ clogin myrouter
Unable to negotiate with 1.2.3.4 port 22: no matching key exchange method found.  Their offer: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1

OpenSSH removed SHA-1 from the defaults a while back, which makes sense since the migration to SHA-2 began several years ago. So looks like SSH is trying to use SHA-2 but the Cisco Router is defaulting to SHA-1, and something has to give in order for negotiation to succeed.

My first thought was to tell the Cisco router to use SHA-2, and this is possible for the MAC setting:

Router(config)#ip ssh server algorithm mac ?
  hmac-sha1      HMAC-SHA1 (digest length = key length = 160 bits)
  hmac-sha1-96   HMAC-SHA1-96 (digest length = 96 bits, key length = 160 bits)
  hmac-sha2-256  HMAC-SHA2-256 (digest length = 256 bits, key length = 256 bits)
  hmac-sha2-512  HMAC-SHA2-512 (digest length = 512 bits, key length = 512 bits

Router(config)#ip ssh server algorithm mac hmac-sha2-256 hmac-sha2-512
Router(config)#do sh ip ssh | inc MAC       
MAC Algorithms:hmac-sha2-256,hmac-sha2-512

But not for key exchange, which apparently only supports SHA-1:

Router(config)#ip ssh server algorithm kex ?
  diffie-hellman-group-exchange-sha1  DH_GRPX_SHA1 diffie-hellman key exchange algorithm
  diffie-hellman-group14-sha1         DH_GRP14_SHA1 diffie-hellman key exchange algorithm

Thus, the only option is to change the setting on the client. SSH has CLI options for Cipher and Mac:

-c : sets cipher (encryption) list.

-m: sets mac (authentication) list

One quick solution is tell the SSH client to support the Kex Exchange by adding this line to the /etc/ssh/ssh_config file:

KexAlgorithms +diffie-hellman-group14-sha1

But, I wanted to change the setting only for Rancid and not SSH in general, hoping that Cisco adds SHA-2 key exchange soon. I found out it is possible to set SSH options in the .cloginrc file. The solution is this:

add  sshcmd  *  {ssh\  -o\ KexAlgorithms=+diffie-
hellman-group14-sha1}

Clogin is now successful:

$ clogin myrouter
spawn ssh -oKexAlgorithms=+diffie-hellman-group14-sha1 -c aes128-ctr,aes128-cbc,3des-cbc -x -l myusername myrouter
Password:
Router#_

By the way, I stayed away from diffie-hellman-group-exchange-sha1 as it’s considered insecure, whereas diffie-hellman-group14-sha1 was considered deprecated but still widely deployed and still “strong enough”, probably thanks to its 2048-bit key length.

Sidenote: this only affects Cisco IOS-XE devices. The Cisco ASA ships with this in the default configuration:

ssh key-exchange group dh-group14-sha256

Install Terraform on Debian 10 (Buster) when a proxy is required

J5Leave a comment 
# Setup proxy, if required
sudo bash -c 'echo "Acquire::http::Proxy \"http://10.0.0.9:3128\";" > /etc/apt/apt.conf.d/99http-proxy'

# Set environment variables to be used by Curl
export http_proxy=http://10.0.0.9:3128
export https_proxy=http://10.0.0.9:3128

Now install Terraform

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -

sudo apt-get install software-properties-common

sudo apt-add-repository "deb [arch=$(dpkg --print-architecture)] https://apt.releases.hashicorp.com $(lsb_release -cs) main"

sudo apt update
sudo apt upgrade
sudo apt install terraform

Migrating to MaxMind GeoIP2 for Python3

J5Leave a comment

With Python2 now EOL, one of my tasks was to replace old python2/geolite2 code with python3/geoip. This does require a subscription to MaxMind to either make the calls via web or download static database files, which fortunately was a option.

Installing Python3 GeoIP2 package

On Ubuntu 20:

  • apt install python3-pip
  • pip3 install geoip2

On FreeBSD 11.4:

  • pkg install python3
  • pkg install py37-pip
  • pip install geoip2

Verify successful install

% python3
Python 3.7.8 (default, Aug  8 2020, 01:18:05) 
[Clang 8.0.0 (tags/RELEASE_800/final 356365)] on freebsd11
Type "help", "copyright", "credits" or "license" for more information.
>>> import geoip2.database
>>> help(geoip2.database.Reader) 
Help on class Reader in module geoip2.database:

Sample Python Script

#!/usr/bin/env python3

import sys
import geoip2.database

ipv4_address = input("Enter an IPv4 address: ")

with geoip2.database.Reader('/var/db/GeoIP2-City.mmdb') as reader:
    try:
        response = reader.city(ipv4_address)
    except:
        sys.exit("No info for address: " + ipv4_address)
    if response:
        lat = response.location.latitude
        lng = response.location.longitude
        city = response.city.name
        print("lat: {}, lng: {}, city: {}".format(lat, lng, city))

Ubuntu 20.04: “Unable to locate package python-pip”

J5Leave a comment

I started the upgrade from Ubuntu 18.04 to 20.04 today and got a bit of a surprise when trying to install PIP so I could use the MaxMind geolite2 package:

root@ubuntu-rr58:/home/me# apt install python-pip
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package python-pip

The root problem here is Python 2 went EoS in January 2020 and does not ship with Ubuntu 20.  But, there is a hack to load certain Python 2 packages…

First, install python3-pip:

apt install python3-pip

Then try to install the python2 packages you’re looking for:

pip3 install python-geoip
pip3 install python-geoip-geolite2

Now, install Python 2.7:

sudo apt install python2

In your script, use sys.path.insert to add the Python3 packages directory.

#!/usr/bin/env python2

from __future__ import print_function
import sys
sys.path.insert(1, '/usr/local/lib/python3.8/dist-packages/')
from geoip import geolite2

 

A better solution for this particular issue was migrate from geoip-geolite2 to , which is fully python3.

Working with CGP Storage via Linux/FreeBSD CLI

J5Leave a comment

Installing Google Cloud SDK on FreeBSD:

This is easily done via package or ports:

pkg install google-cloud-sdk

You may also wish to install the Python modules:

pkg install py39-google-api-python-client
pkg install py39-google-cloud-storage

Installing Google Cloud SDK on Debian/Ubuntu:

Follow the instructions here which are summarized below

Add the Google Cloud SDK as a package source:

echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list

Install required dependencies:

sudo apt install apt-transport-https ca-certificates gnupg

Add Google Cloud public key:

curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
 

Install Google Cloud SDK:

sudo apt update
sudo apt install google-cloud-sdk

Prepping gCloud:

If a proxy server is required, set gcloud to use it:

gcloud config set proxy/type http
gcloud config set proxy/address 10.10.10.100
gcloud config set proxy/port 3128

Configure gCloud.  This will spit out a URL to paste in to browser, which will return an authorization code

gcloud init

This will generate an encrypted file ~/.gsutil/credstor that will be used for authentication.  To re-authenticate:

gcloud auth login

To switch to a different project:

glcoud config set project <PROJECT_ID>

To switch to a different account:

gcloud config set account

To use a service account:

gcloud auth activate-service-account <ACCOUNT_EMAIL> --key-file=<JSON_KEY_FILE>

CLI commands for working with Google Cloud Storage

List existing buckets

gsutil ls

Create a storage bucket called ‘mybucket’

gsutil mb gs://mybucket

Get information about a bucket called ‘mybucket’

gsutil ls -L -b gs://mybucket/

Upload a single file to the bucket

gsutil cp myfile gs://mybucket/

Upload a directory and its contents to a bucket

gsutil cp -r folder1 gs://code-j5-org/

List contents of a bucket

gsutil ls -r gs://mybucket/

Download a file called ‘testfile.png’ in ‘folder1’

gsutil cp gs://mybucket/folder1/testfile.png

Delete multiple files in a folder

gsutil rm gs://mybucket/folder1/*.png

Delete a folder and all its contents

gsutil rm -r gs://mybucket/folder1

Delete a bucket, if bucket is empty

gsutil rb gs://mybucket

Delete a bucket all all files

gsutil rm -r gs://mybucket

Accessing buckets via HTTPS

asdflkj

curl -X POST --data-binary @[OBJECT_LOCATION] \
-H "Authorization: Bearer [OAUTH2_TOKEN]" \
-H "Content-Type: [OBJECT_CONTENT_TYPE]" \
"https://storage.googleapis.com/upload/storage/v1/b/[BUCKET_NAME]/o?uploadType=media&name=[OBJECT_NAME]"

To download files, buckets can be accessed at https://<bucket name>.storage.googleapis.com/path   For example,

curl https://mybucket.storage.googleapis.com/folder1/testfile.png

Within GCP for subnets that have “Private google access”, this DNS name will always resolve to 199.36.153.8-11