Migrating Terraform State Files to Workspaces in an AWS S3 Bucket

Just as I did with GCP a few weeks ago, I needed to circle back and migrate my state files to a cloud storage bucket. This done mainly to centralize the storage location automatically and thus lower the chance of a state file loss or corruption.

Previously, I’d been separating the state files using the -state parameter. I then use a different input file and state file for each environment like this:

terraform apply -var-file=env1.tfvars -state=env1.tfstate
terraform apply -var-file=env2.tfvars -state=env2.tfstate
terraform apply -var-file=env3.tfvars -state=env3.tfstate

To instead store the state files in an AWS S3 bucket, create a backend.tf file with this content:

terraform {
  backend "s3" {
    bucket               = "my-bucket-name"
    workspace_key_prefix = "tf-state"
    key                  = "terraform.tfstate"
    region               = "us-west-1"
  }
}

This will use a bucket named ‘my-bucket-name’ in AWS region us-west-1. Each workspace will store its state file in tfstate/<WORKSPACE_NAME>/terraform.tfstate

Note: if workspace_key_prefix is not specified, the directory ‘env:‘ will be created and used.

Since the backend has changed, I have to run this:

terraform init -reconfigure

I then have to copy the local state files to the correct location that the workspace will be using. This is easiest done with the AWS CLI tool, which will automatically create the sub-directory if it doesn’t exist.

aws s3 cp env1.tfstate s3://my-bucket-name/tf-state/env1/terraform.tfstate
aws s3 cp env2.tfstate s3://my-bucket-name/tf-state/env2/terraform.tfstate
aws s3 cp env3.tfstate s3://my-bucket-name/tf-state/env3/terraform.tfstate

Now I’m ready to run an terraform plan/apply and verify the new state file location is being used.

$ terraform workspace new env1
Created and switched to workspace "env1"!

$ terraform apply -var-file=env1.tfvars

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

$ terraform workspace new env2
Created and switched to workspace "env2"!

$ terraform apply -var-file=env2.tfvars

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

An alternate way to do this migration is enable workspaces first, then migrate the backend to S3.

$ terraform workspace new env1
Created and switched to workspace "env1"!

$ mv env1.tfstate terraform.tfstate.d/env1/terraform.tfstate

$ terraform apply -var-file=env1.tfvars

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Then create the backend.tf file and run terraform init -reconfigure. You’ll then be prompted to move the state files to S3:

$ terraform init -reconfigure
Initializing modules...

Initializing the backend...
Do you want to migrate all workspaces to "s3"?

Enter a value: yes

$ terraform apply -var-file=env1.tfvars

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Getting web server variables and query parameters in different Python Frameworks

As I explore different ways of doing Web programming in Python via different Frameworks, I kept finding the need to examine HTTP server variables, specifically the server hostname, path, and query string. The method to do this varies quite a bit by framework.

Given the following URL: http://www.mydomain.com:8000/derp/test.py?name=Harry&occupation=Hit%20Man

I want to create the following variables with the following values:

server_host is ‘www.mydomain.com’
server_port is 8000
path is ‘/derp/test.py’
query_params is this dictionary: {‘name’: ‘Harry’, ‘occupation’: ‘Hit Man’}

Old School CGI

cgi.FieldStorage() is the easy way to do this, but it returns a list of tuples, and must be converted to a dictionary.

#!/usr/bin/env python3

if __name__ == "__main__":

    import os, cgi

    server_host = os.environ.get('HTTP_HOST', 'localhost')
    server_port = os.environ.get('SERVER_PORT', 80)
    path = os.environ.get('SCRIPT_URL', '/')
    query_params = {}
    _ = cgi.FieldStorage()
    for key in _:
        query_params[key] = str(_[key].value)

Note this will convert all values to strings. By default, cgi.FieldStorage() create numberic values as int or float.

WSGI

Similar to CGI, but environment variables get passed simply in a dictionary as the first parameter. There is no need to load the os module.

def application(environ, start_response):

    from urllib import parse

    server_host = environ.get('HTTP_HOST', 'localhost')
    server_port = environ.get('SERVER_PORT', 80)
    path = environ.get('SCRIPT_URL', '/')
    query_params = {}
    if '?' in environ.get('REQUEST_URI', '/'):
        query_params = dict(parse.parse_qsl(parse.urlsplit(environ['REQUEST_URI']).query))

Since the CGI Headers don’t exist, urllib.parse can be used to analyze the REQUEST_URI environment variable in order to form the dictionary.

Flask

Flask makes this very easy. The only real trick comes with path; the ‘/’ gets removed, so it must be re-added

from flask import Flask, request

app = Flask(__name__)

# Route all possible paths here
@app.route("/", defaults={"path": ""})
@app.route('/<string:path>')
@app.route("/<path:path>")

def index(path):
      
    [server_host, server_port] = request.host.split(':')
    path =  "/" + path
    query_params = request.args

FastAPI

This one’s a slightly different because the main variable to examine actually a QueryParams object with is a form of MultiDict

from fastapi import FastAPI, Request

app = FastAPI()

# Route all possible paths here
@app.get("/")
@app.get("/{path:path}")
def root(path, req: Request):

    [server_host, server_port] = req.headers['host'].split(':')
    path = "/" + path
    query_params = dict(req.query_params)

AWS Lambda

Lambda presents a dictionary called ‘event’ to the handler and it’s simply a matter of grabbing the right keys:

def lambda_handler(event, context):

    server_host = event['headers']['host']
    server_port = event['headers']['X-Forwarded-Port']
    path = event['path']
    query_params = event['queryStringParameters']

If multiValueheaders are enabled, some of the variables come in as lists, which in turn may have a list as values, even if there’s only one item.

    server_host = event['multiValueHeaders']['host'][0]
    query_params = {}
    for _ in event["multiValueQueryStringParameters"].items():
        query_params[_[0]] = _[1][0]

Working with HTTP Requests & Responses in Python

http.client

http client is very fast, but also low-level, and takes a few more lines of code than the others.

import http.client
import ssl

try:
    conn = http.client.HTTPSConnection("www.hightail.com", port=443, timeout=5, context=ssl._create_unverified_context())
    conn.request("GET", "/en_US/theme_default/images/hightop_250px.png")
    resp = conn.getresponse()
    if 301 <= resp.status <= 302:
        print("Status: {}\nLocation: {}\n".format(resp.status,resp.headers['Location']))
    else:
        print("Status: {}\nContent-Type: {}\n".format(resp.status, resp.headers['Content-Type']))

except Exception as e:
    print("Status: 500\nContent-Type: text/plain\n\n{}".format(e))

Requests

Requests is a 3rd party, high level library. It does have a simpler format to use, but is much slower than http.client and is not natively supported on AWS Lambda.

import requests

url = "http://www.yousendit.com"
try:
    resp = requests.get(url, params = {}, timeout = 5, allow_redirects = False)
    if 301 <= resp.status_code <= 302:
        print("Status: {}\nLocation: {}\n".format(resp.status_code,resp.headers['Location']))
    else:
        print("Status: {}\nContent-Type: {}\n".format(resp.status_code, resp.headers['Content-Type']))

except Exception as e:
    print("Status: 500\nContent-Type: text/plain\n\n{}".format(e))

AWS or GCP IPSec Tunnels with BGP routing on a FortiGate software version 6.x

To use BGP routing on an AWS or GCP VPN connection, the tunnel interface needs to have its IP address assigned as a /32 and then the remote IP specified:

config system interface
    edit "GCP"
        set vdom "root"
        set ip 169.254.0.2 255.255.255.255
        set type tunnel
        set remote-ip 169.254.0.1 255.255.255.255
        set interface "wan1"
    next
end

BGP can be configured under the GUI in Network -> BGP in most cases, but the CLI has additional options. Here’s an example config for a peer 169.254.0.1 with ASN 64512, announcing the 192.168.1.0/24 prefix.

config router bgp
    set as 65000
    set router-id 192.168.1.254
    set keepalive-timer 10
    set holdtime-timer 30
    set scan-time 15
    config neighbor
       edit "169.254.0.1"
           set remote-as 64512
       next
    end
    config network
        edit 1
            set prefix 192.168.1.0 255.255.255.0
        next
    end

Using AWS S3 Storage from Linux CLI

Start by installing aws-shell, then run the configure command to enter key and region information:

sudo apt install aws-shell
aws configure

To list files in a bucket called ‘mybucket’:

aws s3 ls s3://mybucket

To upload a single file:

aws s3 cp /tmp/myfile.txt s3://mybucket/

To upload all files in a directory with a certain extension:

aws s3 cp /tmp/ s3://mybucket/ --recursive --exclude '*' --include '*.txt'

To recursively upload contents of a directory:

aws s3 cp /tmp/mydir/ s3://mybucket/ --recursive

To delete a single file:

aws s3 rm s3://mybucket/myfile.text

To empty a bucket (delete all files, but keep bucket):

aws s3 rm s3://mybucket --recursive

Adding a swap file to a t2.nano in AWS running Ubuntu 18

I recently moved my Cacti/Rancid/Ansible Linux VM to a t2.nano in AWS. With only 500 MB of RAM, I knew there would be some performance limitations, but what I didn’t realize is by default, the instance had no swap configured. A MariaDB server consumes ~200 MB of memory when running, and sure enough, mysqld died after a few days uptime:

Apr 20 15:42:20 nettools kernel: [351649.590161] Out of memory: Kill process 27535 (mysqld) score 491 or sacrifice child
Apr 20 15:42:20 nettools kernel: [351649.598181] Killed process 27535 (mysqld) total-vm:1168184kB, anon-rss:240496kB, file-rss:0kB, shmem-rss:0kB
Apr 20 15:42:20 nettools kernel: [351649.676624] oom_reaper: reaped process 27535 (mysqld), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

So I wanted to add a 1GB swap file so that any memory-heavy processes would be happy and stable. It was easy enough to find a blog post that outlined creating the swapfile:

# Become root
sudo su

# Create an empty 1 GB (1MB x 1024) file called /swap.img
dd if=/dev/zero of=/swap.img bs=1M count=1024

# Set recommended permissions
chmod 600 /swap.img

# Convert it to usable swap
mkswap /swap.img

Many of these posts were neglecting how to make the swap activated automatically at boot time. To do so, add this line to bottom of /etc/fstab

/swap.img swap swap defaults 0 0

The swap file can be activated immediately with this command:

swapon -a

Or, give it a test reboot and verify it’s being activated automatically at startup:

ubuntu@linux:~$ top
top - 16:55:04 up 18 min,  1 user,  load average: 0.00, 0.01, 0.03
Tasks: 108 total,   1 running,  71 sleeping,   0 stopped,   0 zombie
%Cpu(s): 10.3 us,  6.0 sy,  0.0 ni, 80.8 id,  3.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :   491200 total,    21040 free,   329668 used,   140492 buff/cache
KiB Swap:  1048572 total,   992764 free,    55808 used.   140596 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                                                                                    
  905 mysql     20   0 1166460 160616  10508 S  4.7 32.7   0:04.90 mysqld                                                                                                     
 1781 www-data  20   0  283268  29696  18944 S  1.7  6.0   0:00.05 php                                                                                                        
 1785 www-data  20   0  289512  30488  18688 S  1.7  6.2   0:00.05 php                                                                                                        
   35 root      20   0       0      0      0 S  1.0  0.0   0:00.45 kswapd0                                                                                                    
  967 www-data  20   0  481904  22936  18432 S  0.3  4.7   0:00.16 apache2                                                                                                    
    1 root      20   0  225264   8408   6792 S  0.0  1.7   0:02.56 systemd

OpenVPN Server in AWS

Licensing costs start at just under $100 per user per year. For compute costs, these are common supported instance sizes in a typical region:

t2.micro 1 vCPU, 1 GB RAM, ~75 Mpbs = ~$100/yr
t2.small 1 vCPU, 2 GB RAM, ~125 Mbps = ~$200/yr
t2.medium 2 vCPU, 2 GB RAM, ~250 Mbps = ~ $400/yr
t2.large 2 vCPU, 8 GB RAM, ~500 Mbps = ~$800/yr

SSH to the IP address using the correct private SSH key and ‘openvpnas’ as the username

The setup wizard should start automatically. To run it again:

sudo su
/usr/bin/ovpn-init –ec2

To use the second (eth1) interface as the internal interface, get the IP address from AWS console and then edit /etc/netplan/50-cloud-init.yaml to add these lines ( (i.e. 192.168.101.123/255.255.255.0)

        eth1:
            dhcp4: no
            addresses: [192.168.101.123/24, ]

After saving the file, restart netplan and verify eth1 has the new IP address

sudo netplan apply
ifconfig eth1

To add internal static routes (for example, the RFC-1918 blocks) add these lines too:

            routes:
            - to: 192.168.0.0/16
                  via: 192.168.101.1
            - to: 172.16.0.0/12
                  via: 192.168.101.1
            - to: 10.0.0.0/8
                  via: 192.168.101.1

Then another restart netplan and verify the routes are working as entered

sudo netplan apply
netstat -rn

Set an initial password for the openvpn admin account via this command:

sudo passwd openvpn

Access the web gui at https://ip.address/admin/ logging in as openvpn with the password that was just set

Upgrading Checkpoint Management Server in AWS from R80.20 to R80.30

Unfortunately it is not possible to simply upgrade an existing CheckPoint management server in AWS. A new one must be built, with the database manually exported from the old instance and imported to the new one.

There is a CheckPoint Knowledge base article, but I found it to have several errors and also be confusing on which version of tools should be used.

Below is the process I used to go from R80.20 to R80.30

cd /home/admin
tar -zxvf Check_Point_R80.30_Gaia_SecurePlatform_above_R75.40_Migration_tools.tgz

Run the export job to create an archive of the database:

./migrate export --exclude-licenses /tmp/R8020Backup.tgz

Copy this .tgz file to the new R80.30 management server in /tmp

On the new management server, run the import job:

cd $FWDIR/bin/upgrade_tools
./migrate import /tmp/R8020Backup.tgz 
The import operation will eventually stop all Check Point services (cpstop)
Do you want to continue? (y/n) [n]? y

After a few minutes, the operation will complete and you’ll be prompted to start services again.

Finish by upgrading SmartConsole to R80.30 and connect to the new R80.30 server. I’ve noticed it to be very slow, but it will eventually connect and all the old gateways and policies will be there.

Launching Checkpoint Gateway in AWS

Instance types for R80.20 or higher:

Checkpoint Gateway: c5.large (2 vCPU / 4 GB) or c5.xlarge (4 vCPU / 8 GB)
Checkpoint Manager: m5.large ( 2vCPU / 8 GB) or m5.xlarge ( 4 vCPU / 16 GB)

Once launched, SSH in and set a temporary admin password:

$ ssh -i ~/.ssh/mykey.pem admin@10.10.10.123
This system is for authorized use only.
In order to configure your system, please access the Web UI and finish the First Time Wizard.

gw-f0633c> set user admin password
New password:
Verify new password:
gw-f0633c> save config

I also like to add routes so the Internal interface is accessible:

gw-f0633c> set static-route 10.0.0.0/8 nexthop gateway address 10.10.10.1 on
gw-f0633c> set static-route 172.16.0.0/12 nexthop gateway address 10.10.10.1 on
gw-f0633c> set static-route 192.168.0.0/16 nexthop gateway address 10.10.10.1 on
gw-f0633c> save config

Now access the GUI. In this case, https://10.10.10.123

Route Filtering and Aggregation in Hybrid Cloud scenarios (EIGRP -> BGP)

One of the challenges of cloud is route table limits . For example, AWS has a limit of 100 per table. This can pose a real challenge in hybrid cloud scenarios where the on-prem infrastructure can easily support hundreds or thousands of internal routes no problem, leaving you (aka “cloud guy”) responsible for performing filtering and aggregation.

Consider this scenario:

EIGRPtoBGPredistribution

The CSR1000v learns about 150 routes via EIGRP, mostly in RFC 1918 space:

D EX 10.4.0.0/16 [170/51307520] via 10.1.4.73, 00:05:02, Tunnel100
D EX 10.5.0.0/16 [170/51307520] via 10.1.4.61, 00:05:02, Tunnel100
D EX 10.6.8.0/22 [170/51307520] via 10.1.4.12, 00:05:02, Tunnel100
D EX 192.168.11.0/24 [170/52234240] via 10.1.4.88, 00:05:02, Tunnel100
D EX 192.168.22.0/23 [170/51829760] via 10.1.4.99, 00:05:02, Tunnel100
D EX 192.168.33.0/24 [170/51824640] via 10.1.4.123, 00:05:02, Tunnel100

So obviously we need need to do some filtering or summarization before passing the routes to the AWS route tables via BGP.

The quick and simple fix: summarize the 10.0.0.0/8 & 192.168.0.0/16 prefixes on the CSR1000v:

router bgp 65000
 bgp log-neighbor-changes
 !
  address-family ipv4 
  aggregate-address 10.0.0.0 255.0.0.0 summary-only  
  aggregate-address 192.168.0.0 255.255.0.0 summary-only
  redistribute eigrp 100
  neighbor 169.254.1.2 remote-as 65100
  neighbor 169.254.1.2 activate

Upon initial examination, this seems to work great. Only the aggregate routes are passed to the BGP neighbors:

CSR1000v#sh ip bgp nei 169.254.1.2 advertised-routes | inc (10\.|192\.168)
*>  10.0.0.0         0.0.0.0       32768 i
*>  192.168.0.0/16   0.0.0.0       32768 i

But there’s a nasty surprise when the EIGRP neighbors are reset. The “summary-only” option briefly stops working for about 20 seconds:

CSR1000v#sh ip bgp nei 169.254.1.2 advertised-routes | inc 10\.
*> 10.0.0.0      0.0.0.0              32768 i
*> 10.4.0.0/16   10.1.4.73  51307520  32768 ?
*> 10.5.0.0/16   10.1.4.61  51307520  32768 ?
*> 10.6.8.0/22   10.1.4.12  51307520  32768 ?
*> 10.7.12.0/22  10.1.4.52  51307520  32768 ?
*> 10.8.8.0/24   10.1.4.7   51307520  32768 ?
*> 10.9.0.0/24   10.1.4.41  51307520  32768 ?
*> 10.77.0.0/16  10.1.4.8   51312640  32768 ?

This exceeds the 100 route limit, and AWS will disable the BGP peering session for 5 minutes.

One fix is use filters instead of the “summary-only” option:

router bgp 65000
 bgp log-neighbor-changes
 !
 address-family ipv4
  aggregate-address 10.0.0.0 255.0.0.0
  aggregate-address 172.16.0.0 255.240.0.0
  aggregate-address 192.168.0.0 255.255.0.0
  redistribute eigrp 100
  distribute-list prefix SUMM_RFC_1918 out
!
ip prefix-list SUMM_RFC_1918 seq 10 deny 10.0.0.0/8 ge 9
ip prefix-list SUMM_RFC_1918 seq 20 deny 172.16.0.0/12 ge 13
ip prefix-list SUMM_RFC_1918 seq 30 deny 192.168.0.0/16 ge 17
ip prefix-list SUMM_RFC_1918 seq 99 permit 0.0.0.0/0 le 32

Another solution is simply don’t do EIGRP to BGP redistribution, and instead just advertise the RFC 1918 blocks with the network statement:

router bgp 65000
 bgp log-neighbor-changes
 !
 address-family ipv4 
  network 10.0.0.0
  network 172.16.0.0 mask 255.240.0.0
  network 192.168.0.0 mask 255.255.0.0
!
ip route 10.0.0.0 255.0.0.0 Null0 254
ip route 172.16.0.0 255.240.0.0 Null0 254
ip route 192.168.0.0 255.255.0.0 Null0 254