Upping the IPv4 Unicast Route Limit on a Nexus 93180YC-EX

J5Leave a comment

We attempted to load a partial route table from CenturyLink (aka Level3) on a Cisco Nexus 93180YC-EX and found the switch threw IPFIB-SLOT1-2-UFIB_ROUTE_CREATE error messages starting at around 200,000 routes:

IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 9, 202.153.28.0/24, flags:0x0, intf:0xd001a, Error: FIB TCAM FULL For IP Routes(1129381967)
IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 9, 202.153.27.0/24, flags:0x0, intf:0xd001a, Error: FIB TCAM FULL For IP Routes(1129381967)
IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 9, 202.153.26.0/24, flags:0x0, intf:0xd001a, Error: FIB TCAM FULL For IP Routes(1129381967)

This command shed some insight on the problem:

MySwitch# show vdc MySwitch resource

Resource                   Min       Max       Used      Unused    Avail   
--------                   ---       ---       ----      ------    -----   
vlan                       16        4094      45        0         4049    
vrf                        2         4096      9         0         4087    
port-channel               0         511       14        0         497     
u4route-mem                248       248       2         246       246     
u6route-mem                96        96        1         95        95      
m4route-mem                58        58        1         57        57      
m6route-mem                8         8         1         7         7

So by default,  only 248 MB of the switch’s TCAM is allocated to IPv4 unicast routes.  This means in a typical 2 ISP deployment, it won’t be able to handle more than a couple hundred thousand routes.

In cases where the desired IPv4 route table exceeds this, a different template such as internet-peering can be set

MySwitch(config)# system routing ?
template-dual-stack-host-scale Dual Stack Host Scale
template-internet-peering Internet Peering
template-lpm-heavy LPM Heavy
template-mpls-heavy MPLS Heavy Scale
template-multicast-heavy Multicast Heavy Scale

This requires a reboot and will show a scary message about disabling multicast routing:

MySwitch(config)# system routing template-internet-peering 
Warning: The command will take effect after next reload.
Multicast is not supported in this profile
Increase the LPM scale by resetting multicast LPM max-scale to 0 using below CLI
hardware profile multicast max-limit lpm-entries 0
Note: This requires copy running-config to startup-config before switch reload.

After the reboot, the memory can be carved out to a larger amount

MySwitch(config)#vdc MySwitch
MySwitch(config-vdc)#limit-resource u4route-mem minimum 256 maximum 512
MySwitch(config)#exit

And now we have more TCAM allocated to IPv4 unicast routes:

MySwitch# show vdc MySwitch resource 

Resource                   Min       Max       Used      Unused    Avail   
--------                   ---       ---       ----      ------    -----   
vlan                       16        4094      45        0         4049    
vrf                        2         4096      9         0         4087    
port-channel               0         511       14        0         497     
u4route-mem                512       512       2         510       510     
u6route-mem                96        96        1         95        95      
m4route-mem                58        58        1         57        57      
m6route-mem                8         8         1         7         7

And now we’re able to take about 286k routes from CenturyLink no problem:

Neighbor   V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
4.15.16.9  4  3356  122187     289   297236    0    0 00:04:55 286770

Now the part I still don’t understand is there’s still the same amount of resources allocated to IPv6 unicast and multicast routes.  It’s also not totally clear what the total TCAM memory amount is, but I would assume 1-2 GB.

 

 

Jumbo Frames on Nexus 93180YC-EX, 5672UP, and perhaps others

J5Leave a comment

Cisco’s documentation implies that to enable jumbo frames on the 5K and 9K line, one must simply set mtu 9216 on the physical and logical L1/L2 interfaces:

Configure and Verify Maximum Transmission Unit on Cisco Nexus Platforms

However, currently working with the 93180YC-EX and previously worked with the 5672UP, I can tell you that both are actually based on the obscure 6K line.

And, per the 5672UP documentation, in order to get jumbo frames, you must do this additional step:

policy-map type network-qos jumbo
  class type network-qos class-default
          mtu 9216
system qos
  service-policy type network-qos jumbo

After applying this, also do a set mtu 9216 on the on the L3 SVIs:

Switch(config)#interface Vlan200
  no shutdown
  mtu 9216


Switch# show interface vl200
Vlan200 is up, line protocol is up, autostate enabled
  Hardware is EtherSVI, address is 70ea.1a44.d0a7
  Internet Address is 192.168.200.1/24
  MTU 9216 bytes, BW 1000000 Kbit, DLY 10 usec,

Switch# show interface et1/17
Ethernet1/17 is up
admin state is up, Dedicated Interface
  Belongs to Po17
  Hardware: 100/1000/10000/25000 Ethernet, address: 70ea.1a44.d0b8 (bia 70ea.1a44.d0b8)
  Description: Storage Filer
  MTU 9216 bytes, BW 10000000 Kbit, DLY 10 usec
  reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, medium is broadcast
  Port mode is trunk
  full-duplex, 10 Gb/s, media type is 10G

 

93180YC – Jumbo Frames

 

Upgrading Checkpoint Management Server in AWS from R80.20 to R80.30

J5Leave a comment

Unfortunately it is not possible to simply upgrade an existing CheckPoint management server in AWS.  A new one must be built, with the database manually exported from the old instance and imported to the new one.

There is a CheckPoint Knowledge base article, but I found it to have several errors and also be confusing on which version of tools should be used.

Below is the process I used to go from R80.20 to R80.30

Login to the old R80.20 server.  Download and extract the R80.30 tools:

cd /home/admin
tar -zxvf Check_Point_R80.30_Gaia_SecurePlatform_above_R75.40_Migration_tools.tgz

Run the export job to create an archive of the database:

./migrate export --exclude-licenses /tmp/R8020Backup.tgz

Copy this .tgz file to the new R80.30 management server in /tmp

On the new management server, run the import job:

cd $FWDIR/bin/upgrade_tools
./migrate import /tmp/R8020Backup.tgz 
The import operation will eventually stop all Check Point services (cpstop)
Do you want to continue? (y/n) [n]? y

After a few minutes, the operation will complete and you’ll be prompted to start services again.

Finish by upgrading SmartConsole to R80.30 and connect to the new R80.30 server.  I’ve noticed it to be very slow, but it will eventually connect and all the old gateways and policies will be there.

Cisco IOS-XE SCP Server with RADIUS authentication

J5Leave a comment

I’ve been wanting to try out SCP to copy IOS images to routers for a while, as I figured it would be faster and cleaner than FTP/TFTP.  There’s essentially three tricks to getting it working..

  1. Having the correct AAA permissions
  2. Understanding the SCP syntax and file systems
  3. Making the scp command from the router VRF aware, if required
  4. 16.6.7 or 16.9.4 or newer code.  Performance on older IOS-XE versions is terrible

First, SSH has to be enabled and of course the SCP server must be activated

ip ssh version 2
ip scp server enable

After doing so, verify the router is accessible via SSH.  If not, try generating a fresh key:

Router(config)#crypto key generate rsa modulus 2048

Now on to the AAA configuration.  The important step is have accounts automatically go to their privilege level 15 without manually entering enable mode.  This is done with the “aaa authorization exec” command:

aaa new-model
!
username admin privilege 15 password 7 XXXXXXX
!
aaa group server radius MyRadiusServer
 server-private 10.1.1.100 auth-port 1812 acct-port 1813 key 7 XXXXXXXX
 ip vrf forwarding MyVRF
!
aaa authentication login default local group MyRadiusServer
aaa authentication enable default none
aaa authorization config-commands
aaa authorization exec default local group MyRadiusServer if-authenticated

The RADIUS server will also need this vendor-specific attribute in the policy:

Vendor: Cisco
Name: Cisco-AV-Pair
Value: priv-lvl=15

If I SSH to the router using a RADIUS account, I should automatically see enable mode:

$ ssh billy@10.1.1.1
Password: 
Router#show privilege
Current privilege level is 15

I can now upload IOS images to a router with IP address 10.1.1.1 like this:

scp csr1000v-universalk9.16.06.06.SPA.bin billy@10.1.1.1:bootflash:/csr1000v-universalk9.16.06.06.SPA.bin

If copying images from the router where the egress interface is on a VRF, the source interface must be specified:

ip ssh source-interface GigabitEthernet0

And simply use the IOS copy command:

copy scp://billy@10.1.1.2:/csr1000v-universalk9.16.06.06.SPA.bin bootflash:

Note scp’s performance in IOS-XE 16.6.5, was very poor, but excellent in 16.6.7 and 16.9.4

IKEv2 VPNs to AWS on Cisco IOS devices

J5Leave a comment

I hadn’t worked with AWS VPNs since January and missed their announcement of supporting IKEv2.  The configuration is similar to GCP with the one exception of SA lifetime.  AWS appears to still use 8 hours (28800 seconds) as opposed to GCP’s 10 hours and the Cisco IOS default of 24 hours (86400 seconds)

Configuring an IKEv2 VPN to AWS

Create IKEv2 Proposal  and Policy, if not done already:

crypto ikev2 proposal IKEV2_PROPOSAL 
  encryption aes-cbc-256 aes-cbc-128    
  integrity sha512 sha384 sha256 sha1   
  group 16 14 2                       ! 16 = 4096, 14 = 2048, 2 = 1024 bit
crypto ikev2 policy IKEV2_POLICY 
  match fvrf any
  proposal IKEV2_PROPOSAL
!

Add entries to an existing keyring, or create a separate one

crypto ikev2 keyring AWS_IKEV2_KEYRING
 peer vpn-0d4fe4b8d9406518f
  address 13.52.37.68
  pre-shared-key XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Create IKEv2 Profile that links to said Keyring.  If behind NAT, specify public IP:

crypto ikev2 profile AWS_IKEV2_PROFILE
 match identity remote address 0.0.0.0 
 identity local address 203.0.113.222    ! Public IP, if behind NAT
 authentication local pre-share
 authentication remote pre-share
 keyring local AWS_IKEV2_KEYRING
 lifetime 28800
!

The IPSec parameters are same as IKEv1, except IKEv2 profile is added:

crypto ipsec transform-set ESP_AES128_SHA esp-aes esp-sha-hmac 
 mode tunnel
crypto ipsec profile AWS_IPSEC_PROFILE
 set security-association lifetime kilobytes disable
 set transform-set ESP_AES128_SHA 
 set pfs group2
 set ikev2-profile AWS_IKEV2_PROFILE

Finally, apply IPsec profile to the VTI with the internet-facing interface as source:

interface Tunnel1
 ip address 169.254.231.142 255.255.255.252
 ip virtual-reassembly in
 ip tcp adjust-mss 1379
 tunnel source GigabitEthernet0/0
 tunnel mode ipsec ipv4
 tunnel destination 13.52.37.68
 tunnel protection ipsec profile AWS_IPSEC_PROFILE

It was good to see them negotiate a nice strong AES-256 / SHA256 / Group16 (4096-bit) SA:

Router#show crypto ikev2 sa
Tunnel-id Local Remote fvrf/ivrf Status 
5 192.168.1.123/4500 35.52.37.68/4500 none/none READY 
Encr: AES-CBC, keysize: 256, PRF: SHA256, Hash: SHA256, DH Grp:16, Auth sign: PSK, Auth verify: PSK
Life/Active Time: 86400/1053 sec

A note about Cisco IOS software versions…

I’ve tested this configuration on a 1921 ISR G2 running IOS 15.5(3)M10

It seems Cisco introduced a slew of bugs relating to IKEv2 for both IOS and IOS-XE in mid-2019:

  • CSCvh66033 – IKEv2 – Crash with segmentation fault when debugs crypto ikev2 are enabled
  • CSCve08418 – IPsec/IKEv2 Installation Sometimes Fails With Simultaneous Negotiations
  • CSCvd69373 – IKEv2: Unable to initiate IKE session to a specific peer due to ‘in-neg’ SA Leak
  • CSCvg15158 – DMVPN session get stuck in NHRP and UP-NO-IKE state without active IKEv2 session until rekey
So, upgrading the latest software version is highly recommended.

 

VPNs to GCP using IKEv2 when your Cisco router is behind NAT

J5Leave a comment

I was able to follow this tutorial but had to make a few adjustments.  The main one is to configure the public IP address in the IKEv2 profile (see step 3 below). 

Remember of course that the router will need UDP ports 500 & 4500 forwarded by the firewall, which also must support ESP passthrough.

1) Configure a global IKEv2 proposal and policy.  

crypto ikev2 proposal MY_IKEV2_PROPOSAL 
 encryption aes-cbc-256 aes-cbc-192 aes-cbc-128
 integrity sha512 sha384 sha256
 group 24 16 14
!
crypto ikev2 policy MY_IKEV2_POLICY 
 proposal MY_IKEV2_PROPOSAL

2) And add the peer to the keyring:

crypto ikev2 keyring MY_KEYRING
 peer gcp1
  address 35.212.226.126
  pre-shared-key abcdef1234567890

3) Create a custom IKEv2 profile.  Note the highlighted public IP address and also the lifetime and DPD interval settings.

crypto ikev2 profile GCP_IKEV2_PROFILE
 match address local interface GigabitEthernet0
 match identity remote address 0.0.0.0
 ! If router is behind NAT, set this to the public IP
 identity local address 203.0.113.222
 authentication remote pre-share
 authentication local pre-share
 keyring local MY_KEYRING
 lifetime 36000                       ! 10 hour SA lifetime
 dpd 60 5 periodic                    ! 1 minute keepalives
!

4) Configure a custom IPSec transform set and profile.  This is 128-bit AES encryption with SHA-256 integrity:

! IPsec Settings
crypto ipsec transform-set ESP_AES128_SHA256 esp-aes esp-sha256-hmac
!
crypto ipsec profile GCP_IPSEC_PROFILE
 set security-association lifetime kilobytes disable
 set security-association lifetime seconds 10800
 set transform-set ESP_AES128_SHA256
 set pfs group14                          ! 2048-bit
 set ikev2-profile GCP_IKEV2_PROFILE
!

5) Finally, create the tunnel interface.  Unlike the IKEv2 profile, this simply references the External interface, not the public IP:

interface Tunnel1
 ip address 169.254.0.2 255.255.255.252
 ip mtu 1460
 ip virtual-reassembly in 
 ip tcp adjust-mss 1420
 tunnel source GigabitEthernet0
 tunnel mode ipsec ipv4
 tunnel destination 35.212.226.126
 tunnel protection ipsec profile GCP_IPSEC_PROFILE
!

Troubleshooting

The SAs should look like this:

Router#show crypto ikev2 sa
IPv4 Crypto IKEv2 SA

Tunnel-id Local Remote fvrf/ivrf Status 
2 192.168.1.123/4500 35.212.226.126/4500 none/none READY 
Encr: AES-CBC, keysize: 128, Hash: SHA256, DH Grp:14, Auth sign: PSK, Auth verify: PSK
Life/Active Time: 36000/1226 sec

Router#show crypto ipsec sa peer 35.212.226.126

interface: Tunnel1
    Crypto map tag: Tunnel1-head-0, local addr 192.168.1.123

   protected vrf: (none)
   local  ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
   remote ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
   current_peer 35.212.226.126 port 4500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 45, #pkts encrypt: 45, #pkts digest: 45
    #pkts decaps: 58, #pkts decrypt: 58, #pkts verify: 58
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 0, #recv errors 0

     local crypto endpt.: 192.168.1.123, remote crypto endpt.: 35.212.226.126
     path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0
     current outbound spi: 0x962EDB69(2519653225)
     PFS (Y/N): N, DH group: none

     inbound esp sas:
      spi: 0x10B829B(17531547)
        transform: esp-aes esp-sha-hmac ,
        in use settings ={Tunnel UDP-Encaps, }
        conn id: 5, flow_id: Onboard VPN:5, sibling_flags 80000040, crypto map: Tunnel1-head-0
        sa timing: remaining key lifetime (sec): (14259)
        Kilobyte Volume Rekey has been disabled
        IV size: 16 bytes
        replay detection support: Y  replay window size: 1024
        Status: ACTIVE(ACTIVE)

     inbound ah sas:

     inbound pcp sas:

     outbound esp sas:
      spi: 0x962EDB69(2519653225)
        transform: esp-aes esp-sha-hmac ,
        in use settings ={Tunnel UDP-Encaps, }
        conn id: 6, flow_id: Onboard VPN:6, sibling_flags 80000040, crypto map: Tunnel1-head-0
        sa timing: remaining key lifetime (sec): (14259)
        Kilobyte Volume Rekey has been disabled
        IV size: 16 bytes
        replay detection support: Y  replay window size: 1024
        Status: ACTIVE(ACTIVE)

     outbound ah sas:

     outbound pcp sas:

VPNs to GCP using IKEv1 when your Cisco router is behind NAT

J5Leave a comment

Tried my first VPN to GCP and didn’t have much luck with IKEv1.  While it did detect the remote router being behind NAT, Phase1 wouldn’t come up due to an ID mismatch:

received NAT-T (RFC 3947) vendor ID
remote host is behind NAT
IDir '192.168.1.123' does not match to '203.0.113.222'

Where 192.168.1.123 is the Real private IP of the router and 203.0.113.222 is the public NAT IP.

This is consistent with the GCP documentation on this topic, which states the following:

When using one-to-one NAT, your on-premises VPN gateway must identify itself using the same external IP address of the NAT device

See also the same configuration with IKEv2.

CheckPoint Initial Configuration via CLI

J5Leave a comment

The default credentials are admin/admin

Verify the management interface

show management interface

Set the management interface with IP address 192.168.1.222/255.255.255.0

set interface Mgmt ipv4-address 192.168.1.222 mask-length 24

Verify IP address for management interface

show interface Mgmt ipv4-address

Ping something

ping 192.168.1.1

Set the default route to 192.168.1.1

set static-route default nexthop gateway address 192.168.1.1 priority 1 on

Create internal route for 10.0.0.0/8 via gateway 10.10.10.10

set static-route 10.0.0.0/8 nexthop gateway address 10.10.10.1 on

Verify routing

show route

Set DNS servers

set dns primary 8.8.8.8
set dns secondary 9.9.9.9

Save the configuration

save config

Show all interface

show interfaces

Show interfaces with IP addresses configured

show security-gateway monitored-interfaces

Create an 802.3ad (LACP) bonded logical interface with eth1 & eth2 as physical members

add bonding group 1
set bonding group 1 mode 8023AD
set bonding group 1 lacp-rate fast
add bonding group 1 interface eth1
add bonding group 1 interface eth2

Create a VLAN sub-interface on bond1 with 802.1q tag 123

add interface bond1 vlan 123

Check software version

show version all

Get hardware information and serial number

show asset system

Change admin password

set user admin password

Set expert mode password

set expert-password

Check policy Status

fw stat

Clear the current local policy

fw unloadlocal

Check site-to-site VPN status

vpn tu tlist

Reset VPN tunnels (list/delete IKE/IPSec SAs)

vpn tu

Modify license, configure SNMP, reset SIC connection:

cpconfig

Verify number of CPUs

fw ctl multik stat

View CPU to connection distribution table

fw ctl affinity -l -r

Reboot the firewall

reboot

IPSec VPN Spoke Router using FQDN authentication

J5Leave a comment

In a previous post I’d covered doing site-to-site IPSec tunnels on Cisco routers when one or both devices are behind NAT.  But what if multiple spoke routers have dynamic IP addresses? Or how about many behind the same NAT address? 

The solution is to have the spoke routers authenticate using FQDN hostname rather than IP address.  I’ve seen lots of examples which overly complicate how to do this.  It’s actually pretty simple and only requires minor changes.  Let’s assume the spoke routers have dynamic IPs and the hub has a static IP of 203.0.113.222…

IKEv1

Spoke Router

On spoke routers with IKEv1, simply replace  the “crypto keyring” statement with “crypto isakmp peer” to use FQDN authentication and IKE aggressive mode, like this:

hostname spoke1
! 
no crypto keyring MyHub
crypto isakmp peer address 203.0.113.222 [vrf InternetVRFName]
 set aggressive-mode password XXXXXXXX
 set aggressive-mode client-endpoint fqdn spoke1.mydomain.com 
!

Note the fqdn hostname doesn’t necessarily need to match the router’s hostname

IKEv2

Spoke Router

Set the Hub’s IP address and pre-shared key in an IKEv2 keyring:

crypto ikev2 keyring MY_IKEV2_KEYRING
 peer MyHub
  address 203.0.113.222
  pre-shared-key MySecretKey1234    ! Must be 16 chars or longer

The identity (hostname) in the IKEv2 profile via the identity local line:

hostname spoke1
!
crypto ikev2 profile SPOKE_IKEV2_PROFILE
 match address local interface GigabitEthernet0/0
 match identity remote address 203.0.113.222 255.255.255.255
 identity local fqdn spoke1.spokedomain.com
 authentication remote pre-share
 authentication local pre-share
 keyring local MY_IKEV2_KEYRING
 dpd 20 2 periodic 
!

IPSec profile:

crypto ipsec profile SPOKE_IPSEC_PROFILE
 set security-association lifetime kilobytes disable
 set pfs group14
 set ikev2-profile SPOKE_IKEV2_PROFILE
!

Tunnel Interface and static route to 10.0.0.0/8:

interface Tunnel1
 ip address 169.254.1.100 255.255.255.0
 ip tcp adjust-mss 1379
 keepalive 10 3
 tunnel source GigabitEthernet0/0
 tunnel mode ipsec ipv4
 tunnel destination 203.0.113.222
 tunnel protection ipsec profile SPOKE_IPSEC_PROFILE
!
ip route 10.0.0.0 255.0.0.0 Tunnel1

Hub Router

Each Spoke will need an entry. Use identity in the entry, not hostname

crypto ikev2 keyring SPOKES_IKEV2_KEYRING
 peer spoke1
  identity fqdn spoke1.spokedomain.com
  pre-shared-key MySecretKey1234
 !

The IKEv2 profile will be a bit different. The domain is used to match multiple spokes:

crypto ikev2 profile HUB_IKEV2_PROFILE
 match address local interface GigabitEthernet0/0
 match identity remote fqdn domain spokedomain.com
 identity local fqdn myhub.hubdomain.com
 authentication remote pre-share
 authentication local pre-share
 keyring local SPOKES_IKEV2_KEYRING
 dpd 10 2 on-demand
 virtual-template 1

The IPSec profile is almost the same but is responder-only (passive):

crypto ipsec profile HUB_IPSEC_PROFILE
 set security-association lifetime kilobytes disable
 set pfs group14
 set ikev2-profile HUB_IKEV2_PROFILE
 responder-only

Rather than a regular tunnel interface, a virtual template one is used:

interface Virtual-Template1 type tunnel
 ip unnumbered Loopback0
 tunnel source GigabitEthernet0/0
 tunnel mode ipsec ipv4
 tunnel destination dynamic
 tunnel protection ipsec profile HUB_IPSEC_PROFILE