IOS-XE 16.6 (Everest) software upgrade on Cisco 3650 & 3850 series switches

Had a 3650 switch stack I needed to upgrade from IOS 16.6.5 to 16.6.8.  The software upgrade procedure is quite different from the 3550/3560/3750s I’d done in the past.

First step is simply download the .bin file to the first switch’s onboard flash drive:

Switch# copy http://mysite/cat3k_caa-universalk9.16.06.08.SPA.bin flash:

I then verify the signature:

Switch#verify flash:cat3k_caa-universalk9.16.06.08.SPA.bin
Starting image verification
Digital signature successfully verified in package cat3k_caa-rpbase.16.06.08.SPA.pkg
Digital signature successfully verified in package cat3k_caa-srdriver.16.06.08.SPA.pkg
Digital signature successfully verified in package cat3k_caa-webui.16.06.08.SPA.pkg
Digital signature successfully verified in package cat3k_caa-rpcore.16.06.08.SPA.pkg
Digital signature successfully verified in package cat3k_caa-guestshell.16.06.08.SPA.pkg
Digital signature successfully verified in file flash:cat3k_caa-universalk9.16.06.08.SPA.bin

The actual command to start the upgrade is quite a mouthful:

Switch#request platform software package install switch all file flash:cat3k_caa-universalk9.16.06.08.SPA.bin new auto-copy

--- Starting install local lock acquisition on switch 1 ---
Finished install local lock acquisition on switch 1
SUCCESS: Software provisioned. New software will load on reboot.
[2]: Finished install successful on switch 2
Checking status of install on [1 2]
[1 2]: Finished install in switch 1 2
SUCCESS: Finished install: Success on [1 2]

This distributes the software as .pkg files on each stack member.

To proceed with the actual upgrade I simply rebooted the entire stack. Ideally I would have done a rolling upgrade similar to how I did 3750s a few years ago, but could not determine if this is supported on the 3650/3850

Other useful commands:

To copy the image on the fly during the upgrade process:

Switch# request platform software package expand switch all file to flash: auto-copy

To clean up old packages to free up disk space:

Switch# request platform software package clean switch all file flash:


VPN from Cisco ISR with frontdoor and interior VRFs + HSRP to a Cisco ASA

So this week I was tasked with bringing up a site-to-site VPN between a Cisco ISR and Cisco ASA.   Every site-to-site VPN I’ve done over the last 10 years or so has been route-based.  But the older Cisco ASA code won’t support those, so it was back to isakmp profiles and crypto-maps.  The real trick was the ISR was using both frontdoor and interior VRFs, and HSRP for failover for added complexity.

Here’s was the information:

  • Site A (Cisco ISR 4431, IOS-XE version 16.6.8):
  •   public IP:, HSRP IP:, VRF: EXTERNAL
  •   private IP IP:, VRF: INTERNAL
  • Site B (Cisco ASA, software version 9.4):
  •   public IP:
  •   private network:

The configuration for the ISR starts with the ISAKMP policy, which basically sets encryption parameters.  This is global, and not VRF-aware:

crypto isakmp invalid-spi-recovery
crypto isakmp keepalive 10
crypto isakmp policy 1
 encr aes 256              ! 256-bit AES encrytpion
 authentication pre-share
 group 2                   ! 1024-bit key
 lifetime 86400            ! default setting
 hash sha                  ! default setting

The pre-shared key is set in the crypto keyring, with needs to be mapped to the frontdoor VRF (the VRF set on the public interface)

crypto keyring MY_KEYRING vrf EXTERNAL
 local-address EXTERNAL
 pre-shared-key address key xxxxxxxxxxxxxxx

Unlike route-based VPNs, an ISAKMP profile is required, which is VRF-aware.  Note the presence of the iVRF (internal one) on the “vrf” line:

crypto isakmp profile MY_ISAKMP_PROFILE
 keyring MY_KEYRING 
 match identity address EXTERNAL
 local-address EXTERNAL

For IPSec / Phase 2, a transform set of course should be defined.   Here I’m using Encapsulation Security Payload, AES 128-bit encryption with SHA-1 hashing, operating in IPSec tunnel mode:

crypto ipsec transform-set ESP_AES-128_SHA_TUNNEL esp-aes esp-sha-hmac 
 mode tunnel
crypto ipsec df-bit clear

Since this is not route-based, an ACL defines the “interesting” traffic to send via the tunnel:

ip access-list extended 101
 permit ip

Then a crypto map links this ACL with the ISAKMP profile and IPSec settings:

crypto map MY_CMAP 1 ipsec-isakmp 
 set peer
 set transform-set ESP_AES-128_SHA_TUNNEL 
 set pfs group5
 set isakmp-profile MY_ISAKMP_PROFILE
 match address 101
 lifetime 3600                ! default setting

Finally, the crypto map is applied to the public interface.  If HSRP is being used between a pair of routers, a name for the HSRP group must be set, then added to the crypto map statement:

interface Port-channel1
 description Public Interface
 vrf forwarding EXTERNAL
 ip address
 ip nat outside
 standby delay minimum 30
 standby 1 ip
 standby 1 priority 254
 standby 1 preempt
 standby 1 name VPNHA
 load-interval 30
 no negotiation auto
 crypto map MY_CMAP redundancy VPNHA
ip route vrf EXTERNAL

I had missed the “redundancy” option on the crypto map in my initial config, which probably cause the traffic to source from the primary IP address rather than the HSRP one.  Running debug crypto isakmp, I would see this:

May 15 17:15:35.892: ISAKMP: (0):SA request profile is MY_ISAKMP_PROFILE
May 15 17:15:35.892: ISAKMP: (0):Created a peer struct for, peer port 500
May 15 17:15:35.892: ISAKMP: (0):New peer created peer = 0x80007FAD0B334DB8 peer_handle = 0x8000000080000012
May 15 17:15:35.892: ISAKMP: (0):Locking peer struct 0x80007FAD0B334DB8, refcount 1 for isakmp_initiator
May 15 17:15:35.892: ISAKMP: (0):local port 500, remote port 500
May 15 17:15:35.892: ISAKMP: (0):set new node 0 to QM_IDLE      
May 15 17:15:35.893: ISAKMP: (0):insert sa successfully sa = 80007FAD0B75C470
May 15 17:15:35.893: ISAKMP: (0):Can not start Aggressive mode, trying Main mode.
May 15 17:15:35.893: ISAKMP-ERROR: (0):No Cert or pre-shared address key. 
May 15 17:15:35.894: ISAKMP-ERROR: (0):construct_initial_message: Can not start Main mode

Since the debug wasn’t showing the source IP for this traffic, I spent several hours troubleshooting the ISAKMP profile and VRF statements, only to conclude they were in fact correct.


“No Vlan association for STP Interface Member 1.0” when upgrading F5 BigIP in AWS from 13.1.1 to

After upgrading several of our AWS Bigip-VEs in AWS from 13.1.1 to without issue, I had big problems with a pair this afternoon.  The first one took forever to boot up, and when it did, was complaining about incomplete configuration.

I’ve seen this before and know it usually means it couldn’t migrate a certain section of the configuration file, and rather than ignoring it, it just can’t load anything.  This is what was showing up in /var/log/ltm and on console:

May 4 17:47:53 bigip warning mcpd[18134]: 01070932:4: Pending local Interface from cluster.: 1.0, configuration ignored
May 4 17:47:53 bigip warning mcpd[18134]: 01070932:4: Pending Interface: 1.0, configuration ignored
May 4 17:47:53 bigip err mcpd[18134]: 01070523:3: No Vlan association for STP Interface Member 1.0.
May 4 17:47:53 bigip emerg load_config_files: "/usr/bin/tmsh -n -g load sys config partitions all base " - failed. -- 01070523:3: No Vlan association for STP Interface Member 1.0. Error: failed to reset strict operations; disconnecting from mcpd. Will reconnect on next command.

The problem was the original 13.1.1 configuration file had this:

net interface 1.0 {

media-fixed 10000T-FD


This is really an error from the get-go, since interface 1.0 is the eth0 / management interface and shouldn’t be in the “net interface” section.

My work around was to reset to factory config, then re-create Self IPs and re-sync the cluster.  Alternately, the configuration file could be modified to simply remove the offending lines.

Since I did not see this configuration in any other F5 BigIP-VEs, I’d suspect it was mistakenly inserted in to the 13.1.1-0.0.4 AMI by F5 that I’d launched last summer.

Ubuntu 20.04: “Unable to locate package python-pip”

I started the upgrade from Ubuntu 18.04 to 20.04 today and got a bit of a surprise when trying to install PIP so I could use the MaxMind geolite2 package:

root@ubuntu-rr58:/home/me# apt install python-pip
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package python-pip

The root problem here is Python 2 went EoS in January 2020 and does not ship with Ubuntu 20.  But, there is a hack to load certain Python 2 packages…

First, install python3-pip:

apt install python3-pip

Then try to install the python2 packages you’re looking for:

pip3 install python-geoip
pip3 install python-geoip-geolite2

Now, install Python 2.7:

sudo apt install python2

In your script, use sys.path.insert to add the Python3 packages directory.

#!/usr/bin/env python2

from __future__ import print_function
import sys
sys.path.insert(1, '/usr/local/lib/python3.8/dist-packages/')
from geoip import geolite2


A better solution for this particular issue was migrate from geoip-geolite2 to , which is fully python3.