LACP config migration bug in IOS-XE 16.12.05

Not to publically shame Cisco or anything, but to publically shame Cisco…

We’re moving a site to a different physical location next week, so I wanted to do a round of patching beforehand to clear out any possible lingering software bugs. Our IOS-XE ISR 4431 routers and 3650 switch stack were on 16.12.04 which is very current, but I noticed 16.12.05 came out last week and while my general rule of thumb is never try a Cisco IOS version until it’s been out for at least a month, you’d think a release ending in a ‘5’ marked as MD would be good to go, right?

Oh…Cisco. Fool me once, shame on you. Fool me twice? We can’t get fooled again…,

So the router came up find with no errors and seemed to check out fine, but I soon realized the DMVPN tunnel showed no EIGRP neighbors and ARP showed all entries incomplete. I ultimately had to come in through a console backdoor and noticed the core switch had suspended the LACP bundle, even though the router reported interface Port-Channel1 up/up. Huh.

Looking closer at the router configuration, I soon realized the problem. What had previously been a working LACP configuration in IOS-XE 16.12.04 had now become broken a forced EtherChannel on 16.12.05:

interface GigabitEthernet0/0/0
 no ip address
 negotiation auto
 channel-group 1 
!
interface GigabitEthernet0/0/1
 no ip address
 negotiation auto
 channel-group 1 
!

Thus, the core switch (Nexus 93180YC-EX) which was still configured as LACP passive had rightfully suspended the bundle.

Recreating the bundles as LACP made them functional again:

conf t
interface GigabitEthernet0/0/0
 no channel-group 1 
!
interface GigabitEthernet0/0/1
 no channel-group 1 
 channel-group 1 mode active
!
interface GigabitEthernet0/0/0
 channel-group 1 mode active
!

Ironically, LACP support is one of the reasons I had retired our working-perfectly-fine 2921s in favor of the 4431s.

Once again, this example proves three important points:

IOS-XE, despite being out for like 10 years, is still buggy
Code developed non-MD trains is being slipped in to the MD train and bringing new bugs along with it. In this case, https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvw74609
Cisco’s branding of “MD” is meaningless anyway, because maintenance releases clearly are not undergoing adequate regression testing

Cisco ISR G2 to CheckPoint R80.30 IKEv1 VPN woes

I had previously done Cisco router to CheckPoint R80.30 gateway VPNs before without issue, but for whatever reason could not even establish phase 1 for this one. CheckPoint R80 VPN communities default to AES-256, SHA-1, Group 2, and 1-day timetime which is easy to match on the Cisco with this config:

crypto keyring mycheckpoint
 local-address GigabitEthernet0/0
 pre-shared-key address 192.0.2.190 key abcdefghij1234567890
!
crypto isakmp policy 100
 encr aes 256
 authentication pre-share
 group 2
 hash sha          ! <--- default value
 lifetime 864000   ! <--- default value
!

After verifying connectivity, doing packet captures, and multiple reboots on on both ends, IKE simply would not come up. On the Cisco ISR, debug crypto isakmp wasn’t especially helpful:

Jun 18 11:06:17.085: ISAKMP: (0):purging SA., sa=3246F97C, delme=3246F97C
Jun 18 11:06:17.285: ISAKMP: (0):SA request profile is (NULL)
Jun 18 11:06:17.285: ISAKMP: (0):Created a peer struct for 35.245.62.190, peer port 500
Jun 18 11:06:17.285: ISAKMP: (0):New peer created peer = 0x2CE62C3C peer_handle = 0x80000005
Jun 18 11:06:17.285: ISAKMP: (0):Locking peer struct 0x2CE62C3C, refcount 1 for isakmp_initiator
Jun 18 11:06:17.285: ISAKMP: (0):local port 500, remote port 500
Jun 18 11:06:17.285: ISAKMP: (0):set new node 0 to QM_IDLE
Jun 18 11:06:17.285: ISAKMP: (0):insert sa successfully sa = 2CE620E8
Jun 18 11:06:17.285: ISAKMP: (0):Can not start Aggressive mode, trying Main mode.
Jun 18 11:06:17.285: ISAKMP: (0):found peer pre-shared key matching 192.0.2.190
Jun 18 11:06:17.285: ISAKMP: (0):constructed NAT-T vendor-rfc3947 ID
Jun 18 11:06:17.285: ISAKMP: (0):constructed NAT-T vendor-07 ID
Jun 18 11:06:17.285: ISAKMP: (0):constructed NAT-T vendor-03 ID
Jun 18 11:06:17.285: ISAKMP: (0):constructed NAT-T vendor-02 ID
Jun 18 11:06:17.285: ISAKMP: (0):Input = IKE_MESG_FROM_IPSEC, IKE_SA_REQ_MM
Jun 18 11:06:17.285: ISAKMP: (0):Old State = IKE_READY New State = IKE_I_MM1
Jun 18 11:06:17.285: ISAKMP: (0):beginning Main Mode exchange
Jun 18 11:06:17.285: ISAKMP-PAK: (0):sending packet to 192.0.2.190 my_port 500 peer_port 500 (I) MM_NO_STATE
Jun 18 11:06:17.285: ISAKMP: (0):Sending an IKE IPv4 Packet.
Jun 18 11:06:17.369: ISAKMP-PAK: (0):received packet from 192.0.2.190 dport 500 sport 500 Global (I) MM_NO_STATE
Jun 18 11:06:17.369: ISAKMP-ERROR: (0):Couldn't find node: message_id 2303169274
Jun 18 11:06:17.369: ISAKMP-ERROR: (0):(0): Unknown Input IKE_MESG_FROM_PEER, IKE_INFO_NOTIFY: state = IKE_I_MM1
Jun 18 11:06:17.369: ISAKMP: (0):Input = IKE_MESG_FROM_PEER, IKE_INFO_NOTIFY
Jun 18 11:06:17.369: ISAKMP: (0):Old State = IKE_I_MM1 New State = IKE_I_MM1

The CheckPoint gave a more “useful” error:

Main Mode Failed to match proposal: Transform: AES-256, SHA1, Group 2 (1024 bit); Reason: Wrong value for: Authentication Method

This seemed to imply the CheckPoint was expecting certificate-based authentication rather than PSK. In traditional mode, the gateway is set by default for certificate only. But it’s not clear how this is configured in newer versions.

After poking around settings for quite a while, I simply deleted the VPN community in CheckPoint SmartConsole and re-created it. The connection then popped up immediately.

¯\_(ツ)_/¯

IOS-XE 16.6 (Everest) software upgrade on Cisco 3650 & 3850 series switches

Had a 3650 switch stack I needed to upgrade from IOS 16.6.5 to 16.6.8. The software upgrade procedure is quite different from the 3550/3560/3750s I’d done in the past.

First step is simply download the .bin file to the first switch’s onboard flash drive:

Switch# copy http://mysite/cat3k_caa-universalk9.16.06.08.SPA.bin flash:

I then verify the signature:

Switch#verify flash:cat3k_caa-universalk9.16.06.08.SPA.bin
Starting image verification
Digital signature successfully verified in package cat3k_caa-rpbase.16.06.08.SPA.pkg
Digital signature successfully verified in package cat3k_caa-srdriver.16.06.08.SPA.pkg
Digital signature successfully verified in package cat3k_caa-webui.16.06.08.SPA.pkg
Digital signature successfully verified in package cat3k_caa-rpcore.16.06.08.SPA.pkg
Digital signature successfully verified in package cat3k_caa-guestshell.16.06.08.SPA.pkg
Digital signature successfully verified in file flash:cat3k_caa-universalk9.16.06.08.SPA.bin

The actual command to start the upgrade is quite a mouthful:

Switch#request platform software package install switch all file flash:cat3k_caa-universalk9.16.06.08.SPA.bin new auto-copy

--- Starting install local lock acquisition on switch 1 ---
Finished install local lock acquisition on switch 1
....
SUCCESS: Software provisioned. New software will load on reboot.
[2]: Finished install successful on switch 2
Checking status of install on [1 2]
[1 2]: Finished install in switch 1 2
SUCCESS: Finished install: Success on [1 2]

This distributes the software as .pkg files on each stack member.

To proceed with the actual upgrade I simply rebooted the entire stack. Ideally I would have done a rolling upgrade similar to how I did 3750s a few years ago, but could not determine if this is supported on the 3650/3850

Other useful commands:

To copy the image on the fly during the upgrade process:

Switch# request platform software package expand switch all file http://mysite.com/mypath/cat3k_caa-universalk9.16.12.04.SPA.bin to flash: auto-copy

To clean up old packages to free up disk space:

Switch# request platform software package clean switch all file flash:

VPN from Cisco ISR with frontdoor and interior VRFs + HSRP to a Cisco ASA

So this week I was tasked with bringing up a site-to-site VPN between a Cisco ISR and Cisco ASA. Every site-to-site VPN I’ve done over the last 10 years or so has been route-based. But the older Cisco ASA code won’t support those, so it was back to isakmp profiles and crypto-maps. The real trick was the ISR was using both frontdoor and interior VRFs, and HSRP for failover for added complexity.

Here’s was the information:

Site A (Cisco ISR 4431, IOS-XE version 16.6.8):
public IP: 198.51.100.52/29, HSRP IP: 198.51.100.54, VRF: EXTERNAL
private IP IP: 10.111.111.1/24, VRF: INTERNAL
Site B (Cisco ASA, software version 9.4):
public IP: 203.0.113.105
private network: 10.222.222.0/24

The configuration for the ISR starts with the ISAKMP policy, which basically sets encryption parameters. This is global, and not VRF-aware:

crypto isakmp invalid-spi-recovery
crypto isakmp keepalive 10
!
crypto isakmp policy 1
 encr aes 256              ! 256-bit AES encrytpion
 authentication pre-share
 group 2                   ! 1024-bit key
 lifetime 86400            ! default setting
 hash sha                  ! default setting
!

The pre-shared key is set in the crypto keyring, with needs to be mapped to the frontdoor VRF (the VRF set on the public interface)

crypto keyring MY_KEYRING vrf EXTERNAL
 local-address 198.51.100.54 EXTERNAL
 pre-shared-key address 203.0.113.105 key xxxxxxxxxxxxxxx

Unlike route-based VPNs, an ISAKMP profile is required, which is VRF-aware. Note the presence of the iVRF (internal one) on the “vrf” line:

crypto isakmp profile MY_ISAKMP_PROFILE
 vrf INTERNAL
 keyring MY_KEYRING 
 match identity address 203.0.113.105 255.255.255.255 EXTERNAL
 local-address 198.51.100.54 EXTERNAL
!

For IPSec / Phase 2, a transform set of course should be defined. Here I’m using Encapsulation Security Payload, AES 128-bit encryption with SHA-1 hashing, operating in IPSec tunnel mode:

crypto ipsec transform-set ESP_AES-128_SHA_TUNNEL esp-aes esp-sha-hmac 
 mode tunnel
!
crypto ipsec df-bit clear

Since this is not route-based, an ACL defines the “interesting” traffic to send via the tunnel:

ip access-list extended 101
 permit ip 10.111.111.0 0.0.0.255 10.222.222.0 0.0.0.255
!

Then a crypto map links this ACL with the ISAKMP profile and IPSec settings:

crypto map MY_CMAP 1 ipsec-isakmp 
 set peer 203.0.113.105
 set transform-set ESP_AES-128_SHA_TUNNEL 
 set pfs group5
 set isakmp-profile MY_ISAKMP_PROFILE
 match address 101
 reverse-route
 lifetime 3600                ! default setting
!

Finally, the crypto map is applied to the public interface. If HSRP is being used between a pair of routers, a name for the HSRP group must be set, then added to the crypto map statement:

interface Port-channel1
 description Public Interface
 vrf forwarding EXTERNAL
 ip address 198.51.100.52 255.255.255.248
 ip nat outside
 standby delay minimum 30
 standby 1 ip 198.51.100.54
 standby 1 priority 254
 standby 1 preempt
 standby 1 name VPNHA
 load-interval 30
 no negotiation auto
 crypto map MY_CMAP redundancy VPNHA
!
ip route vrf EXTERNAL 0.0.0.0 0.0.0.0 198.51.100.49

I had missed the “redundancy” option on the crypto map in my initial config, which probably cause the traffic to source from the primary IP address rather than the HSRP one. Running debug crypto isakmp, I would see this:

May 15 17:15:35.892: ISAKMP: (0):SA request profile is MY_ISAKMP_PROFILE
May 15 17:15:35.892: ISAKMP: (0):Created a peer struct for 203.0.113.105, peer port 500
May 15 17:15:35.892: ISAKMP: (0):New peer created peer = 0x80007FAD0B334DB8 peer_handle = 0x8000000080000012
May 15 17:15:35.892: ISAKMP: (0):Locking peer struct 0x80007FAD0B334DB8, refcount 1 for isakmp_initiator
May 15 17:15:35.892: ISAKMP: (0):local port 500, remote port 500
May 15 17:15:35.892: ISAKMP: (0):set new node 0 to QM_IDLE      
May 15 17:15:35.893: ISAKMP: (0):insert sa successfully sa = 80007FAD0B75C470
May 15 17:15:35.893: ISAKMP: (0):Can not start Aggressive mode, trying Main mode.
May 15 17:15:35.893: ISAKMP-ERROR: (0):No Cert or pre-shared address key. 
May 15 17:15:35.894: ISAKMP-ERROR: (0):construct_initial_message: Can not start Main mode

Since the debug wasn’t showing the source IP for this traffic, I spent several hours troubleshooting the ISAKMP profile and VRF statements, only to conclude they were in fact correct.

Cisco IOS-XE SCP Server with RADIUS authentication

I’ve been wanting to try out SCP to copy IOS images to routers for a while, as I figured it would be faster and cleaner than FTP/TFTP. There’s essentially three tricks to getting it working..

Having the correct AAA permissions
Understanding the SCP syntax and file systems
Making the scp command from the router VRF aware, if required
16.6.7 or 16.9.4 or newer code. Performance on older IOS-XE versions is terrible

First, SSH has to be enabled and of course the SCP server must be activated

ip ssh version 2
ip scp server enable

After doing so, verify the router is accessible via SSH. If not, try generating a fresh key:

Router(config)#crypto key generate rsa modulus 2048

Now on to the AAA configuration. The important step is have accounts automatically go to their privilege level 15 without manually entering enable mode. This is done with the “aaa authorization exec” command:

aaa new-model
!
username admin privilege 15 password 7 XXXXXXX
!
aaa group server radius MyRadiusServer
 server-private 10.1.1.100 auth-port 1812 acct-port 1813 key 7 XXXXXXXX
 ip vrf forwarding MyVRF
!
aaa authentication login default local group MyRadiusServer
aaa authentication enable default none
aaa authorization config-commands
aaa authorization exec default local group MyRadiusServer if-authenticated

The RADIUS server will also need this vendor-specific attribute in the policy:

Vendor: Cisco
Name: Cisco-AV-Pair
Value: priv-lvl=15

If I SSH to the router using a RADIUS account, I should automatically see enable mode:

$ ssh billy@10.1.1.1
Password: 
Router#show privilege
Current privilege level is 15

I can now upload IOS images to a router with IP address 10.1.1.1 like this:

scp csr1000v-universalk9.16.06.06.SPA.bin billy@10.1.1.1:bootflash:/csr1000v-universalk9.16.06.06.SPA.bin

If copying images from the router where the egress interface is on a VRF, the source interface must be specified:

ip ssh source-interface GigabitEthernet0

And simply use the IOS copy command:

copy scp://billy@10.1.1.2:/csr1000v-universalk9.16.06.06.SPA.bin bootflash:

Note scp’s performance in IOS-XE 16.6.5, was very poor, but excellent in 16.6.7 and 16.9.4

VPNs to GCP using IKEv2 when your Cisco router is behind NAT

I was able to follow this tutorial but had to make a few adjustments. The main one is to configure the public IP address in the IKEv2 profile (see step 3 below).

Remember of course that the router will need UDP ports 500 & 4500 forwarded by the firewall, which also must support ESP passthrough.

1) Configure a global IKEv2 proposal and policy.

crypto ikev2 proposal MY_IKEV2_PROPOSAL 
 encryption aes-cbc-256 aes-cbc-192 aes-cbc-128
 integrity sha512 sha384 sha256
 group 24 16 14
!
crypto ikev2 policy MY_IKEV2_POLICY 
 proposal MY_IKEV2_PROPOSAL

2) And add the peer to the keyring:

crypto ikev2 keyring MY_KEYRING
 peer gcp1
  address 35.212.226.126
  pre-shared-key abcdef1234567890

3) Create a custom IKEv2 profile. Note the highlighted public IP address and also the lifetime and DPD interval settings.

crypto ikev2 profile GCP_IKEV2_PROFILE
 match address local interface GigabitEthernet0
 match identity remote address 0.0.0.0
 ! If router is behind NAT, set this to the public IP
 identity local address 203.0.113.222
 authentication remote pre-share
 authentication local pre-share
 keyring local MY_KEYRING
 lifetime 36000                       ! 10 hour SA lifetime
 dpd 60 5 periodic                    ! 1 minute keepalives
!

4) Configure a custom IPSec transform set and profile. This is 128-bit AES encryption with SHA-256 integrity:

! IPsec Settings
crypto ipsec transform-set ESP_AES128_SHA256 esp-aes esp-sha256-hmac
!
crypto ipsec profile GCP_IPSEC_PROFILE
 set security-association lifetime kilobytes disable
 set security-association lifetime seconds 10800
 set transform-set ESP_AES128_SHA256
 set pfs group14                          ! 2048-bit
 set ikev2-profile GCP_IKEV2_PROFILE
!

5) Finally, create the tunnel interface. Unlike the IKEv2 profile, this simply references the External interface, not the public IP:

interface Tunnel1
 ip address 169.254.0.2 255.255.255.252
 ip mtu 1460
 ip virtual-reassembly in 
 ip tcp adjust-mss 1420
 tunnel source GigabitEthernet0
 tunnel mode ipsec ipv4
 tunnel destination 35.212.226.126
 tunnel protection ipsec profile GCP_IPSEC_PROFILE
!

Troubleshooting

The SAs should look like this:

Router#show crypto ikev2 sa
IPv4 Crypto IKEv2 SA

Tunnel-id Local Remote fvrf/ivrf Status 
2 192.168.1.123/4500 35.212.226.126/4500 none/none READY 
Encr: AES-CBC, keysize: 128, Hash: SHA256, DH Grp:14, Auth sign: PSK, Auth verify: PSK
Life/Active Time: 36000/1226 sec

Router#show crypto ipsec sa peer 35.212.226.126

interface: Tunnel1
    Crypto map tag: Tunnel1-head-0, local addr 192.168.1.123

   protected vrf: (none)
   local  ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
   remote ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
   current_peer 35.212.226.126 port 4500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 45, #pkts encrypt: 45, #pkts digest: 45
    #pkts decaps: 58, #pkts decrypt: 58, #pkts verify: 58
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 0, #recv errors 0

     local crypto endpt.: 192.168.1.123, remote crypto endpt.: 35.212.226.126
     path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0
     current outbound spi: 0x962EDB69(2519653225)
     PFS (Y/N): N, DH group: none

     inbound esp sas:
      spi: 0x10B829B(17531547)
        transform: esp-aes esp-sha-hmac ,
        in use settings ={Tunnel UDP-Encaps, }
        conn id: 5, flow_id: Onboard VPN:5, sibling_flags 80000040, crypto map: Tunnel1-head-0
        sa timing: remaining key lifetime (sec): (14259)
        Kilobyte Volume Rekey has been disabled
        IV size: 16 bytes
        replay detection support: Y  replay window size: 1024
        Status: ACTIVE(ACTIVE)

     inbound ah sas:

     inbound pcp sas:

     outbound esp sas:
      spi: 0x962EDB69(2519653225)
        transform: esp-aes esp-sha-hmac ,
        in use settings ={Tunnel UDP-Encaps, }
        conn id: 6, flow_id: Onboard VPN:6, sibling_flags 80000040, crypto map: Tunnel1-head-0
        sa timing: remaining key lifetime (sec): (14259)
        Kilobyte Volume Rekey has been disabled
        IV size: 16 bytes
        replay detection support: Y  replay window size: 1024
        Status: ACTIVE(ACTIVE)

     outbound ah sas:

     outbound pcp sas:

IPSec VPN Spoke Router using FQDN authentication

In a previous post I’d covered doing site-to-site IPSec tunnels on Cisco routers when one or both devices are behind NAT. But what if multiple spoke routers have dynamic IP addresses? Or how about many behind the same NAT address?

The solution is to have the spoke routers authenticate using FQDN hostname rather than IP address. I’ve seen lots of examples which overly complicate how to do this. It’s actually pretty simple and only requires minor changes. Let’s assume the spoke routers have dynamic IPs and the hub has a static IP of 203.0.113.222…

IKEv1

Spoke Router

On spoke routers with IKEv1, simply replace the “crypto keyring” statement with “crypto isakmp peer” to use FQDN authentication and IKE aggressive mode, like this:

hostname spoke1
! 
no crypto keyring MyHub
crypto isakmp peer address 203.0.113.222 [vrf InternetVRFName]
 set aggressive-mode password XXXXXXXX
 set aggressive-mode client-endpoint fqdn spoke1.mydomain.com 
!

Note the fqdn hostname doesn’t necessarily need to match the router’s hostname

IKEv2

Spoke Router

Set the Hub’s IP address and pre-shared key in an IKEv2 keyring:

crypto ikev2 keyring MY_IKEV2_KEYRING
 peer MyHub
  address 203.0.113.222
  pre-shared-key MySecretKey1234    ! Must be 16 chars or longer

The identity (hostname) in the IKEv2 profile via the identity local line:

hostname spoke1
!
crypto ikev2 profile SPOKE_IKEV2_PROFILE
 match address local interface GigabitEthernet0/0
 match identity remote address 203.0.113.222 255.255.255.255
 identity local fqdn spoke1.spokedomain.com
 authentication remote pre-share
 authentication local pre-share
 keyring local MY_IKEV2_KEYRING
 dpd 20 2 periodic 
!

IPSec profile:

crypto ipsec profile SPOKE_IPSEC_PROFILE
 set security-association lifetime kilobytes disable
 set pfs group14
 set ikev2-profile SPOKE_IKEV2_PROFILE
!

Tunnel Interface and static route to 10.0.0.0/8:

interface Tunnel1
 ip address 169.254.1.100 255.255.255.0
 ip tcp adjust-mss 1379
 keepalive 10 3
 tunnel source GigabitEthernet0/0
 tunnel mode ipsec ipv4
 tunnel destination 203.0.113.222
 tunnel protection ipsec profile SPOKE_IPSEC_PROFILE
!
ip route 10.0.0.0 255.0.0.0 Tunnel1

Hub Router

Each Spoke will need an entry. Use identity in the entry, not hostname

crypto ikev2 keyring SPOKES_IKEV2_KEYRING
 peer spoke1
  identity fqdn spoke1.spokedomain.com
  pre-shared-key MySecretKey1234
 !

The IKEv2 profile will be a bit different. The domain is used to match multiple spokes:

crypto ikev2 profile HUB_IKEV2_PROFILE
 match address local interface GigabitEthernet0/0
 match identity remote fqdn domain spokedomain.com
 identity local fqdn myhub.hubdomain.com
 authentication remote pre-share
 authentication local pre-share
 keyring local SPOKES_IKEV2_KEYRING
 dpd 10 2 on-demand
 virtual-template 1

The IPSec profile is almost the same but is responder-only (passive):

crypto ipsec profile HUB_IPSEC_PROFILE
 set security-association lifetime kilobytes disable
 set pfs group14
 set ikev2-profile HUB_IKEV2_PROFILE
 responder-only

Rather than a regular tunnel interface, a virtual template one is used:

interface Virtual-Template1 type tunnel
 ip unnumbered Loopback0
 tunnel source GigabitEthernet0/0
 tunnel mode ipsec ipv4
 tunnel destination dynamic
 tunnel protection ipsec profile HUB_IPSEC_PROFILE

Route Filtering and Aggregation in Hybrid Cloud scenarios (EIGRP -> BGP)

One of the challenges of cloud is route table limits . For example, AWS has a limit of 100 per table. This can pose a real challenge in hybrid cloud scenarios where the on-prem infrastructure can easily support hundreds or thousands of internal routes no problem, leaving you (aka “cloud guy”) responsible for performing filtering and aggregation.

Consider this scenario:

EIGRPtoBGPredistribution

The CSR1000v learns about 150 routes via EIGRP, mostly in RFC 1918 space:

D EX 10.4.0.0/16 [170/51307520] via 10.1.4.73, 00:05:02, Tunnel100
D EX 10.5.0.0/16 [170/51307520] via 10.1.4.61, 00:05:02, Tunnel100
D EX 10.6.8.0/22 [170/51307520] via 10.1.4.12, 00:05:02, Tunnel100
D EX 192.168.11.0/24 [170/52234240] via 10.1.4.88, 00:05:02, Tunnel100
D EX 192.168.22.0/23 [170/51829760] via 10.1.4.99, 00:05:02, Tunnel100
D EX 192.168.33.0/24 [170/51824640] via 10.1.4.123, 00:05:02, Tunnel100

So obviously we need need to do some filtering or summarization before passing the routes to the AWS route tables via BGP.

The quick and simple fix: summarize the 10.0.0.0/8 & 192.168.0.0/16 prefixes on the CSR1000v:

router bgp 65000
 bgp log-neighbor-changes
 !
  address-family ipv4 
  aggregate-address 10.0.0.0 255.0.0.0 summary-only  
  aggregate-address 192.168.0.0 255.255.0.0 summary-only
  redistribute eigrp 100
  neighbor 169.254.1.2 remote-as 65100
  neighbor 169.254.1.2 activate

Upon initial examination, this seems to work great. Only the aggregate routes are passed to the BGP neighbors:

CSR1000v#sh ip bgp nei 169.254.1.2 advertised-routes | inc (10\.|192\.168)
*>  10.0.0.0         0.0.0.0       32768 i
*>  192.168.0.0/16   0.0.0.0       32768 i

But there’s a nasty surprise when the EIGRP neighbors are reset. The “summary-only” option briefly stops working for about 20 seconds:

CSR1000v#sh ip bgp nei 169.254.1.2 advertised-routes | inc 10\.
*> 10.0.0.0      0.0.0.0              32768 i
*> 10.4.0.0/16   10.1.4.73  51307520  32768 ?
*> 10.5.0.0/16   10.1.4.61  51307520  32768 ?
*> 10.6.8.0/22   10.1.4.12  51307520  32768 ?
*> 10.7.12.0/22  10.1.4.52  51307520  32768 ?
*> 10.8.8.0/24   10.1.4.7   51307520  32768 ?
*> 10.9.0.0/24   10.1.4.41  51307520  32768 ?
*> 10.77.0.0/16  10.1.4.8   51312640  32768 ?

This exceeds the 100 route limit, and AWS will disable the BGP peering session for 5 minutes.

One fix is use filters instead of the “summary-only” option:

router bgp 65000
 bgp log-neighbor-changes
 !
 address-family ipv4
  aggregate-address 10.0.0.0 255.0.0.0
  aggregate-address 172.16.0.0 255.240.0.0
  aggregate-address 192.168.0.0 255.255.0.0
  redistribute eigrp 100
  distribute-list prefix SUMM_RFC_1918 out
!
ip prefix-list SUMM_RFC_1918 seq 10 deny 10.0.0.0/8 ge 9
ip prefix-list SUMM_RFC_1918 seq 20 deny 172.16.0.0/12 ge 13
ip prefix-list SUMM_RFC_1918 seq 30 deny 192.168.0.0/16 ge 17
ip prefix-list SUMM_RFC_1918 seq 99 permit 0.0.0.0/0 le 32

Another solution is simply don’t do EIGRP to BGP redistribution, and instead just advertise the RFC 1918 blocks with the network statement:

router bgp 65000
 bgp log-neighbor-changes
 !
 address-family ipv4 
  network 10.0.0.0
  network 172.16.0.0 mask 255.240.0.0
  network 192.168.0.0 mask 255.255.0.0
!
ip route 10.0.0.0 255.0.0.0 Null0 254
ip route 172.16.0.0 255.240.0.0 Null0 254
ip route 192.168.0.0 255.255.0.0 Null0 254

IPSec VPNs on Cisco routers when both are behind NAT

IPSec VPNs or really any site-to-site VPN works best when at least one of the sides or better yet both have Public IP addresses. But what if one is behind NAT, or even both? It gets increasing tricky to configure the correct IP addresses for authentication, and forward correct ports on protocols. As I recently discovered, using IKEv2 and/or GRE further complicates things. Consider this setup:

Both routers are behind NAT/PAT firewalls without static 1-to-1 NATs configured. There are still some requirements though:

Both firewalls must allow for protocol 50 passthrough for IPSec, or protocol 47 passthough if using GRE, which most do
At least one side must be forwarding ports udp/500 (isakmp) and udp/4500 (nat-t) to the router’s internet-facing interface so the connection can be established
Both routers need crypto ipsec nat-transparency udp-encapsulation enabled, which is the default setting.

Let’s look at sample configs for each scenario. These assume 1921 ISR G2 routers but IOS-XE configs should be exactly the same.

IPSec with ISAKMP / IKEv1

The is the simplest way to do it since only public IPs need to be referenced.

1) The ISAKMP portion:

crypto isakmp invalid-spi-recovery
crypto isakmp disconnect-revoked-peers
crypto isakmp keepalive 10
crypto isakmp nat keepalive 900

! Policy supporting strong encryption
crypto isakmp policy 100
 encr aes 256               ! 256-bit AES encryption
 hash sha384                ! SHA-384 hashing
 authentication pre-share   ! Using pre-shared keys
 lifetime 28800             ! 28000 seconds = 8 hours
 group 14                   ! group 14 = 2048 bit key

! Backup policy supporting weaker encryption support for older devices
crypto isakmp policy 200
 encr aes                   ! 128-bit AES encryption
 hash sha                   ! SHA-1 hashing
 authentication pre-share   ! Using pre-shared keys
 lifetime 28800             ! 28000 seconds = 8 hours
 group 2                    ! group 2 = 1024 bit key

! FYI - default values are still des, sha, rsa-sig, 86400, group 1 :-O

2) And then pre-shared keys:

! Key for site 1 router
crypto keyring Site2
  local-address GigabitEthernet0/0
  pre-shared-key address 203.0.113.222 key 0 MySecretKey

! Key for site 2 router
crypto keyring Site1
 local-address GigabitEthernet0/0
 pre-shared-key address 198.51.100.111 key 0 MySecretKey

! Can also use "crypto isakmp key 0 MySecretKey address 1.2.3.4"

3) IPSec parameters. For encryption, I like to just use 128-bit AES with either SHA-256 or SHA-1 signing with a group 2 (1024-bit) key to make the tunnel negotiation quick as possible.

! Create some IPSec Transform sets for ESP & 128-bit AES
crypto ipsec transform-set ESP_AES128_SHA256 esp-aes esp-sha256-hmac 
 mode tunnel
crypto ipsec transform-set ESP_AES128_SHA esp-aes esp-sha-hmac 
 mode tunnel

! Create IPSec profile
crypto ipsec profile MY_IPSEC_PROFILE
 set transform-set ESP_AES128_SHA256 ESP_AES128_SHA 
 set pfs group2    ! group 2 = 1024-bit key

Optional step: since the “client” side isn’t reachable on port udp/500, the “server” side may be configured as a responder. This cuts down superfluous traffic, especially when the client is unreachable.

crypto ipsec profile IPSEC_PROFILE
 responder-only

5) The last step is build the tunnel interfaces. For the “client” side:

interface Tunnel1000
 ip address 169.254.0.1 255.255.255.252
 ip tcp adjust-mss 1379
 keepalive 10 3
 tunnel source GigabitEthernet0/0
 tunnel mode ipsec ipv4
 tunnel destination 203.0.113.222
 tunnel protection ipsec profile IPSEC_PROFILE

Server side is exactly the same but with different IP addresses:

interface Tunnel1000
 ip address 169.254.0.2 255.255.255.252
 tunnel destination 198.51.100.111

Doing debug crypto isakmp on the server side while the tunnels come up shows the public IP address of the client. Note the client’s random source ports.

ISAKMP (0): received packet from 198.51.100.111 dport 500 sport 14972 Global (R) MM_SA_SETUP
ISAKMP (1003): received packet from 198.51.100.111 dport 4500 sport 51597 Global (R) MM_KEY_EXCH
ISAKMP (1003): received packet from 198.51.100.111 dport 4500 sport 51597 Global (R) MM_KEY_EXCH
ISAKMP:(1003):SA has been authenticated with 198.51.100.111
ISAKMP:(1003):Detected port floating to port = 51597

We never see client’s private IP, but we do see the server side’s private IP at the end when the SA is finally built:

ISAKMP:(1003): Process initial contact,
bring down existing phase 1 and 2 SA's with local 192.168.2.222 remote 198.51.100.111 remote port 51597
ISAKMP: Trying to insert a peer 192.168.2.222/198.51.100.111/51597/, and inserted successfully

Can also see the other site’s private IP by examining the SAs once built:

Site1#show crypto isakmp peers 203.0.113.222
Peer: 203.0.113.222 Port: 4500 Local: 192.168.1.111
Phase1 id: 192.168.2.222

Site2#show crypto isakmp peers 198.51.100.111
Peer: 198.51.100.111 Port: 51597 Local: 192.168.2.222
Phase1 id: 192.168.1.111

IPSec with IKEv2

IKEv2 is new to me, but it was a surprise to see slightly different behavior when using NAT. Run through of the configuration:

1) Set some global IKEv2 parameters

crypto logging ikev2
crypto ikev2 nat keepalive 900
crypto ikev2 dpd 10 2 periodic

2) Create an IKEv2 Proposal and Policy

3) Keys are defined in the IKEv2 keyring, which has each site’s public IP address:

crypto ikev2 keyring MY_IKEV2_KEYRING
 ! Use this on site 1 router
 peer Site2
  address 203.0.113.222
  pre-shared-key MySecretKey1234    ! Must be 16 chars or longer
 ! Use this on site 2 router
 peer Site1
  address 198.51.100.111
  pre-shared-key MySecretKey1234    ! Must be 16 chars or longer

4) Create IKEv2 Profile

crypto ikev2 profile MY_IKEV2_PROFILE
 match address local interface GigabitEthernet0/0
 match identity remote address 192.168.0.0 255.255.0.0
 authentication remote pre-share
 authentication local pre-share
 keyring local MY_IKEV2_KEYRING
 dpd 20 2 periodic

This is where is gets weird because the “remote address” parameter needs to match the internal IP address(es) of the other sides. The simple and lazy approach is use 0.0.0.0 since that will match anything. The other option is use a different profile for each peer.

5) Add IKEv2 profile to the existing IPSec profile. Note that doing so shouldn’t break IKEv1 clients as the IKEv2 stuff should just be ignored.

crypto ipsec profile MY_IPSEC_PROFILE
 set transform-set ESP_AES128_SHA256_TUNNEL ESP_AES128_SHA1_TUNNEL 
 set pfs group2
 set ikev2-profile MY_IKEV2_PROFILE

In the SAs, we can see our private IP, but not the other side’s:

Site1#show crypto ikev2 sa remote 203.0.113.222
Tunnel-id Local                 Remote                fvrf/ivrf            Status
1         192.168.1.111/4500    203.0.113.222/4500    none/none            READY  
      Encr: AES-CBC, keysize: 128, Hash: SHA256, DH Grp:14, Auth sign: PSK, Auth verify: PSK
      Life/Active Time: 86400/30 sec

Site2#sh crypto ikev2 sa remote 198.51.100.111      
Tunnel-id Local                 Remote                fvrf/ivrf            Status 
1         192.168.2.222/4500    198.51.100.111/51597  none/none            READY  
      Encr: AES-CBC, keysize: 128, Hash: SHA256, DH Grp:14, Auth sign: PSK, Auth verify: PSK
      Life/Active Time: 86400/71 sec

BTW, if NAT-T has been disabled but is required by the other end, debug crypto ikev2 will show this:

Oct 12 19:53:08.620: IKEv2-ERROR:(SESSION ID = 1075,SA ID = 2): NAT is found but it is not supported.: NAT-T disabled via cli
Oct 12 19:53:08.620: IKEv2-ERROR:(SESSION ID = 1075,SA ID = 2):: NAT-T disabled via cli

IKEv2 Policies, Proposals, and Profiles on Cisco Routers

Just like “crypto isakmp policy”, the “crypto ikev2 policy” configuration is global and cannot be specified on a per-peer basis. If there’s a mismatch, “debug crypto ikev2 error” will show something like this:

IKEv2-ERROR:(SESSION ID = 685,SA ID = 1):Expected Policies: : Failed to find a matching policyProposal 1: AES-CBC-128 SHA256 SHA256 DH_GROUP_2048_MODP/Group 14

There are two solutions. The simplest is specify all possible encryption, integrity, and PFS parameters in a single proposal:

crypto ikev2 proposal MY_IKEV2_PROPOSAL 
 encryption aes-cbc-256 aes-cbc-192 aes-cbc-128 3des
 integrity sha512 sha384 sha256 sha1
 group 21 20 19 16 14 5 2

crypto ikev2 policy MY_IKEV2_POLICY 
 proposal MY_IKEV2_PROPOSAL

Alternately, write separate proposals, then list them in the policy by preference:

crypto ikev2 proposal HIGH 
 encryption aes-cbc-256 aes-cbc-192 aes-cbc-128 
 integrity sha512 sha384 sha256 
 group 21 20 19
crypto ikev2 proposal MEDIUM
 encryption aes-cbc-256 aes-cbc-192 aes-cbc-128 
 integrity sha256 sha1
 group 16 14
crypto ikev2 proposal LOW 
 encryption aes-cbc-128 3des
 integrity sha1 md5
 group 5 2
crypto ikev2 policy MY_IKEV2_POLICY 
 proposal HIGH
 proposal MEDIUM
 proposal LOW

It’s disappointing Cisco did not design things so policies could be associated with individual peers. Imagine a router terminating VPNs to different business partners, where Partner A insists on AES256/SHA512/Group16, while Partner B is still doing 3DES/MD5/Group2. You would write the policy with the most secure at the top, but there’s nothing to stop partner A from downgrading to partner B’s policy. It’s a security concern, plus takes extra time to negotiate.

Palo Alto Firewalls supports different ISAKMP policies on a per-IKE gateway basis and it’s one of the reasons I’ve really preferred them for Site-to-Site VPNs over the years.