Category: Linux

Configure Squid for HTTPS on Debian VM

Verify we’re running the latest version of Debian

lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 11 (bullseye)
Release:	11
Codename:	bullseye

Become root

sudo su

Update packages

apt update && apt upgrade -y

Install the Squid package that has openssl configured and enabled

apt install squid-openssl

Create a local CA, using a 4096-bit key and SHA-2 hashing. This one is good for the next 10 years

openssl req -new -newkey rsa:4096 -sha256 -days 3653 -nodes -x509 -keyout /etc/squid/CA.key -out /etc/squid/CA.crt

Combine the key and cert in to a single file for convenience

cat CA.key CA.crt > CA.pem

Initialize the directory used for minted certs and set permissions so squid owns it

/usr/lib/squid/security_file_certgen -c -s /var/spool/squid/ssl_db -M 4MB
chown -R proxy:proxy /var/spool/squid

Finally, configure Squid to use HTTPS

http_port 3128 ssl-bump cert=/etc/squid/CA.pem generate-host-certificates=on options=NO_SSLv3
ssl_bump bump all

Restart Squid

service squid restart

Test connections by configuring 3128. Note the certificate from the CA, good for 10 years:

export https_proxy=http://localhost:3128

curl -v --cacert CA.crt  https://teapotme.com 

* Uses proxy env variable https_proxy == 'http://localhost:3128'
*   Trying ::1:3128...
* Connected to localhost (::1) port 3128 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to teapotme.com:443
> CONNECT teapotme.com:443 HTTP/1.1
> Host: teapotme.com:443
> User-Agent: curl/7.74.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< 
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: CA.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=teapotme.com
*  start date: Nov  6 04:03:48 2022 GMT
*  expire date: Nov  6 04:03:48 2032 GMT
*  subjectAltName: host "teapotme.com" matched cert's "teapotme.com"
*  issuer: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd; CN=localhost
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: teapotme.com
> User-Agent: curl/7.74.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Mark bundle as not supporting multiuse
< HTTP/1.1 418 I'm a teapot
< Server: nginx
< Date: Sun, 06 Nov 2022 04:08:13 GMT
< Content-Type: application/json
< Content-Length: 483
< X-Cache: MISS from test-1
< X-Cache-Lookup: MISS from test-1:3128
< Via: 1.1 test-1 (squid/4.13)
< Connection: keep-alive
< 
{
    "host": "teapotme.com",
    "user-agent": "curl/7.74.0",
    "x-forwarded-for": "::1, 35.233.234.155, 172.17.0.1",
    "x-forwarded-proto": "https",
}
Advertisement

Git clone / pull / push fails with ‘no mutual signature algorithm’ on Ubuntu 22 to GCP Cloud Source

I created a new Ubuntu 22 VM a few weeks ago and noticed when trying a git pull or git push to a GCP Cloud Source Repo, I wasn’t having any luck when using SSH:

cd myrepo/
git pull
myusername@myorg.com@source.developers.google.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

The SSH key was a standard RSA with the public key uploaded to Cloud Source SSH Keys, so there was no obvious reason why it wasn’t working.

Next step was try and get some type of debug or error message as to why the public key exchange wasn’t working. Newer versions of Git can turn on SSH debugging by setting the GIT_SSH_COMMAND environment variable, so I did that:

export GIT_SSH_COMMAND="ssh -vvv"

When re-running the git pull request, I get some somewhat useful debugs back:

debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/j5/.ssh/id_rsa RSA SHA256:JBgC+R4Ozel+YI+7oEv1UOf9/jLqGBhysN8bpoEDbPU
debug1: send_pubkey_test: no mutual signature algorithm

The ‘no mutual signature algorithm’ indicated one side didn’t like the signing algorithm. I did a Google and found this article which indicates that Ubuntu 22 doesn’t allow RSA by default. I can’t change the setting on the Cloud Source side, so on the Ubuntu 22 client, I did this as a quick work-around:

echo "PubkeyAcceptedKeyTypes +ssh-rsa" > /etc/ssh/ssh_config.d/enable_rsa.conf

And now the git pull/push works without issue.

An alternate solution is instead use Elliptic Curve DSA rather than RSA. To generate a new ECDSA key:

ssh-keygen -t ecdsa
cat ~/.ssh/id_ecdsa.pub

Then copy/paste the key in to the SSH Key Manager. This will be easier to copy/paste then RSA since it’s shorter.

Clearing out /var/spool/clientmqueue in FreeBSD

My FreeBSD VM with its 10GB virtual hard disk ran out of space today. The primary culprit was /var/spool/clientmqueue consuming nearly 3GB of space:

# du -d 1 /var/spool/
8	/var/spool/output
4	/var/spool/opielocks
2955904	/var/spool/clientmqueue
4	/var/spool/dma
4	/var/spool/lpd
4	/var/spool/lock
4	/var/spool/mqueue
2955936	/var/spool/

But when trying to just delete the files, I got “argument list too long”:

# rm -f /var/spool/clientmqueue/*
/bin/rm: Argument list too long.

In the Google search I learned something interesting: find has a -delete option. This worked well:

# find /var/spool/clientmqueue -name '*' -delete

Rancid: no matching key exchange method found. Their offer: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1

Time to move Rancid to a newer VM again, this time it’s Ubuntu 20. Hit a snag when I tried a test clogin run:

$ clogin myrouter
Unable to negotiate with 1.2.3.4 port 22: no matching key exchange method found.  Their offer: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1

OpenSSH removed SHA-1 from the defaults a while back, which makes sense since the migration to SHA-2 began several years ago. So looks like SSH is trying to use SHA-2 but the Cisco Router is defaulting to SHA-1, and something has to give in order for negotiation to succeed.

My first thought was to tell the Cisco router to use SHA-2, and this is possible for the MAC setting:

Router(config)#ip ssh server algorithm mac ?
  hmac-sha1      HMAC-SHA1 (digest length = key length = 160 bits)
  hmac-sha1-96   HMAC-SHA1-96 (digest length = 96 bits, key length = 160 bits)
  hmac-sha2-256  HMAC-SHA2-256 (digest length = 256 bits, key length = 256 bits)
  hmac-sha2-512  HMAC-SHA2-512 (digest length = 512 bits, key length = 512 bits

Router(config)#ip ssh server algorithm mac hmac-sha2-256 hmac-sha2-512
Router(config)#do sh ip ssh | inc MAC       
MAC Algorithms:hmac-sha2-256,hmac-sha2-512

But not for key exchange, which apparently only supports SHA-1:

Router(config)#ip ssh server algorithm kex ?
  diffie-hellman-group-exchange-sha1  DH_GRPX_SHA1 diffie-hellman key exchange algorithm
  diffie-hellman-group14-sha1         DH_GRP14_SHA1 diffie-hellman key exchange algorithm

Thus, the only option is to change the setting on the client. SSH has CLI options for Cipher and Mac:

-c : sets cipher (encryption) list.

-m: sets mac (authentication) list

But the option for Key Exchange can only be configured via the /etc/ssh/sshd_config file with this line:

KexAlgorithms +diffie-hellman-group14-sha1

I wanted to change the setting only for Rancid and not SSH in general, hoping that Cisco adds SHA-2 key exchange soon. I found out it is possible to set SSH options in the .cloginrc file. The solution is this:

add  sshcmd  *  {ssh\  -o\ KexAlgorithms=+diffie-
hellman-group14-sha1}

Clogin is now successful:

$ clogin myrouter
spawn ssh -oKexAlgorithms=+diffie-hellman-group14-sha1 -c aes128-ctr,aes128-cbc,3des-cbc -x -l myusername myrouter
Password:
Router#_

By the way, I stayed away from diffie-hellman-group-exchange-sha1 as it’s considered insecure, whereas diffie-hellman-group14-sha1 was considered deprecated but still widely deployed and still “strong enough”, probably thanks to its 2048-bit key length.

Sidenote: this only affects Cisco IOS-XE devices. The Cisco ASA ships with this in the default configuration:

ssh key-exchange group dh-group14-sha256

Install Terraform on Debian 10 (Buster) when a proxy is required

# Setup proxy, if required
sudo bash -c 'echo "Acquire::http::Proxy \"http://10.0.0.9:3128\";" > /etc/apt/apt.conf.d/99http-proxy'

# Set environment variables to be used by Curl
export http_proxy=http://10.0.0.9:3128
export https_proxy=http://10.0.0.9:3128

Now install Terraform

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -

sudo apt-get install software-properties-common

sudo apt-add-repository "deb [arch=$(dpkg --print-architecture)] https://apt.releases.hashicorp.com $(lsb_release -cs) main"

sudo apt update
sudo apt upgrade
sudo apt install terraform

Migrating to MaxMind GeoIP2 for Python3

With Python2 now EOL, one of my tasks was to replace old python2/geolite2 code with python3/geoip. This does require a subscription to MaxMind to either make the calls via web or download static database files, which fortunately was a option.

Installing Python3 GeoIP2 package

On Ubuntu 20:

  • apt install python3-pip
  • pip3 install geoip2

On FreeBSD 11.4:

  • pkg install python3
  • pkg install py37-pip
  • pip install geoip2

Verify successful install

% python3
Python 3.7.8 (default, Aug  8 2020, 01:18:05) 
[Clang 8.0.0 (tags/RELEASE_800/final 356365)] on freebsd11
Type "help", "copyright", "credits" or "license" for more information.
>>> import geoip2.database
>>> help(geoip2.database.Reader) 
Help on class Reader in module geoip2.database:

Sample Python Script

#!/usr/bin/env python3

import sys
import geoip2.database

ipv4_address = input("Enter an IPv4 address: ")

with geoip2.database.Reader('/var/db/GeoIP2-City.mmdb') as reader:
    try:
        response = reader.city(ipv4_address)
    except:
        sys.exit("No info for address: " + ipv4_address)
    if response:
        lat = response.location.latitude
        lng = response.location.longitude
        city = response.city.name
        print("lat: {}, lng: {}, city: {}".format(lat, lng, city))

Ubuntu 20.04: “Unable to locate package python-pip”

I started the upgrade from Ubuntu 18.04 to 20.04 today and got a bit of a surprise when trying to install PIP so I could use the MaxMind geolite2 package:

root@ubuntu-rr58:/home/me# apt install python-pip
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package python-pip

The root problem here is Python 2 went EoS in January 2020 and does not ship with Ubuntu 20.  But, there is a hack to load certain Python 2 packages…

First, install python3-pip:

apt install python3-pip

Then try to install the python2 packages you’re looking for:

pip3 install python-geoip
pip3 install python-geoip-geolite2

Now, install Python 2.7:

sudo apt install python2

In your script, use sys.path.insert to add the Python3 packages directory.

#!/usr/bin/env python2

from __future__ import print_function
import sys
sys.path.insert(1, '/usr/local/lib/python3.8/dist-packages/')
from geoip import geolite2

 

A better solution for this particular issue was migrate from geoip-geolite2 to , which is fully python3.

Working with CGP Storage via Linux/FreeBSD CLI

Installing Google Cloud SDK on FreeBSD:

This is easily done via package or ports:

pkg install google-cloud-sdk

Installing Google Cloud SDK on Debian/Ubuntu:

Follow the instructions here which are summarized below

Add the Google Cloud SDK as a package source:

echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list

Install required dependencies:

sudo apt install apt-transport-https ca-certificates gnupg

Add Google Cloud public key:

curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
 

Install Google Cloud SDK:

sudo apt update
sudo apt install google-cloud-sdk

Prepping gCloud:

If a proxy server is required, set gcloud to use it:

gcloud config set proxy/type http
gcloud config set proxy/address 10.10.10.100
gcloud config set proxy/port 3128

Configure gCloud.  This will spit out a URL to paste in to browser, which will return an authorization code

gcloud init

This will generate an encrypted file ~/.gsutil/credstor that will be used for authentication.  To re-authenticate:

gcloud auth login

To switch to a different project:

glcoud config set project <PROJECT_ID>

To switch to a different account:

gcloud config set account

To use a service account:

gcloud auth activate-service-account <ACCOUNT_EMAIL> --key-file=<JSON_KEY_FILE>

CLI commands for working with Google Cloud Storage

List existing buckets

gsutil ls

Create a storage bucket called ‘mybucket’

gsutil mb gs://mybucket

Get information about a bucket called ‘mybucket’

gsutil ls -L -b gs://mybucket/

Upload a single file to the bucket

gsutil cp myfile gs://mybucket/

Upload a directory and its contents to a bucket

gsutil cp -r folder1 gs://code-j5-org/

List contents of a bucket

gsutil ls -r gs://mybucket/

Download a file called ‘testfile.png’ in ‘folder1’

gsutil cp gs://mybucket/folder1/testfile.png

Delete multiple files in a folder

gsutil rm gs://mybucket/folder1/*.png

Delete a folder and all its contents

gsutil rm -r gs://mybucket/folder1

Delete a bucket, if bucket is empty

gsutil rb gs://mybucket

Delete a bucket all all files

gsutil rm -r gs://mybucket

Accessing buckets via HTTPS

asdflkj

curl -X POST --data-binary @[OBJECT_LOCATION] \
-H "Authorization: Bearer [OAUTH2_TOKEN]" \
-H "Content-Type: [OBJECT_CONTENT_TYPE]" \
"https://storage.googleapis.com/upload/storage/v1/b/[BUCKET_NAME]/o?uploadType=media&name=[OBJECT_NAME]"

To download files, buckets can be accessed at https://<bucket name>.storage.googleapis.com/path   For example,

curl https://mybucket.storage.googleapis.com/folder1/testfile.png

Within GCP for subnets that have “Private google access”, this DNS name will always resolve to 199.36.153.8-11

Using AWS S3 Storage from Linux CLI

Start by installing aws-shell, then run the configure command to enter key and region information:

sudo apt install aws-shell
aws configure

To list files in a bucket called ‘mybucket’:

aws s3 ls s3://mybucket

To upload a single file:

aws s3 cp /tmp/myfile.txt s3://mybucket/

To upload all files in a directory with a certain extension:

aws s3 cp /tmp/ s3://mybucket/ --recursive --exclude '*' --include '*.txt'

To recursively upload contents of a directory:

aws s3 cp /tmp/mydir/ s3://mybucket/ --recursive

To delete a single file:

aws s3 rm s3://mybucket/myfile.text

To empty a bucket (delete all files, but keep bucket):

aws s3 rm s3://mybucket --recursive

 

Adding a swap file to a t2.nano in AWS running Ubuntu 18

I recently moved my Cacti/Rancid/Ansible Linux VM to a t2.nano in AWS. With only 500 MB of RAM, I knew there would be some performance limitations, but  what I didn’t realize is by default, the instance had no swap configured.  A MariaDB server consumes ~200 MB of memory when running, and sure enough, mysqld died after a few days uptime:

Apr 20 15:42:20 nettools kernel: [351649.590161] Out of memory: Kill process 27535 (mysqld) score 491 or sacrifice child
Apr 20 15:42:20 nettools kernel: [351649.598181] Killed process 27535 (mysqld) total-vm:1168184kB, anon-rss:240496kB, file-rss:0kB, shmem-rss:0kB
Apr 20 15:42:20 nettools kernel: [351649.676624] oom_reaper: reaped process 27535 (mysqld), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

So I wanted to add a 1GB swap file so that any memory-heavy processes would be happy and stable.  It was easy enough to find a blog post that outlined creating the swapfile:

# Become root
sudo su

# Create an empty 1 GB (1MB x 1024) file called /swap.img
dd if=/dev/zero of=/swap.img bs=1M count=1024

# Set recommended permissions
chmod 600 /swap.img

# Convert it to usable swap
mkswap /swap.img

Many of these posts were neglecting how to make the swap activated automatically at boot time.  To do so, add this line to bottom of /etc/fstab

/swap.img swap swap defaults 0 0

The swap file can be activated immediately with this command:

swapon -a

Or, give it a test reboot and verify it’s being activated automatically at startup:

ubuntu@linux:~$ top
top - 16:55:04 up 18 min,  1 user,  load average: 0.00, 0.01, 0.03
Tasks: 108 total,   1 running,  71 sleeping,   0 stopped,   0 zombie
%Cpu(s): 10.3 us,  6.0 sy,  0.0 ni, 80.8 id,  3.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :   491200 total,    21040 free,   329668 used,   140492 buff/cache
KiB Swap:  1048572 total,   992764 free,    55808 used.   140596 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                                                                                    
  905 mysql     20   0 1166460 160616  10508 S  4.7 32.7   0:04.90 mysqld                                                                                                     
 1781 www-data  20   0  283268  29696  18944 S  1.7  6.0   0:00.05 php                                                                                                        
 1785 www-data  20   0  289512  30488  18688 S  1.7  6.2   0:00.05 php                                                                                                        
   35 root      20   0       0      0      0 S  1.0  0.0   0:00.45 kswapd0                                                                                                    
  967 www-data  20   0  481904  22936  18432 S  0.3  4.7   0:00.16 apache2                                                                                                    
    1 root      20   0  225264   8408   6792 S  0.0  1.7   0:02.56 systemd