CheckPoint Dedicated Management Route

New feature (finally!) in R80.30 is the ability to enabled Management data plane Separation, in order to have a separate route table for the management interface and all management related functions (Policy installation, SSH, SNMP, syslog, GAIA portal, etc).

Let’s assume the interface “Mgmt” has already been set as the management interface with IP address 192.168.1.100 and wants default gateway 192.168.1.1, and “eth5” has been setup as the dedicated sync interface:

set mdps mgmt plane on
set mdps mgmt resource on
set mdps interface Mgmt management on
set mdps interface eth5 sync on
add mdps route 0.0.0.0/0 nexthop 192.168.1.1
save config
reboot

After the box comes up you can verify the management route has been set by going in to expert mode and the the “mplane” command to enter management space:

> expert
[Expert@MyCheckPoint:0]# mplane
Context set to Management Plane
[Expert@MyCheckPoint:1]# netstat -rn
Kernel IP routing table
Destination  Gateway       Genmask         Flags MSS Window irtt Iface
169.254.0.0  0.0.0.0       255.255.255.252 U     0   0      0    eth5
192.168.1.0  0.0.0.0       255.255.255.0   U     0   0      0    Mgmt
0.0.0.0      192.168.1.1   0.0.0.0         UGD   0   0      0    Mgmt

Routes from the main route table relating to management can then be deleted, which makes the data plane route table much cleaner:

[Expert@MyCheckpoint:1]# dplane
Context set to Data Plane

[Expert@MyCheckPoint:0]# netstat -rn
Kernel IP routing table
Destination   Gateway       Genmask         Flags MSS Window irtt Iface
203.0.113.32  0.0.0.0       255.255.255.224 U     0   0      0    bond1.11
192.168.222.0 0.0.0.0       255.255.255.0   U     0   0      0    bond1.22
0.0.0.0       203.0.113.33  0.0.0.0         UGD   0   0      0    bond1.11
192.168.0.0   192.168.222.1 255.255.0.0     UGD   0   0      0    bond1.22
Advertisement

Upping the IPv4 Unicast Route Limit on a Nexus 93180YC-EX

We attempted to load a partial route table from CenturyLink (aka Level3) on a Cisco Nexus 93180YC-EX and found the switch threw IPFIB-SLOT1-2-UFIB_ROUTE_CREATE error messages starting at around 200,000 routes:

IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 9, 202.153.28.0/24, flags:0x0, intf:0xd001a, Error: FIB TCAM FULL For IP Routes(1129381967)
IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 9, 202.153.27.0/24, flags:0x0, intf:0xd001a, Error: FIB TCAM FULL For IP Routes(1129381967)
IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 9, 202.153.26.0/24, flags:0x0, intf:0xd001a, Error: FIB TCAM FULL For IP Routes(1129381967)

This command shed some insight on the problem:

MySwitch# show vdc MySwitch resource

Resource                   Min       Max       Used      Unused    Avail   
--------                   ---       ---       ----      ------    -----   
vlan                       16        4094      45        0         4049    
vrf                        2         4096      9         0         4087    
port-channel               0         511       14        0         497     
u4route-mem                248       248       2         246       246     
u6route-mem                96        96        1         95        95      
m4route-mem                58        58        1         57        57      
m6route-mem                8         8         1         7         7

So by default,  only 248 MB of the switch’s TCAM is allocated to IPv4 unicast routes.  This means in a typical 2 ISP deployment, it won’t be able to handle more than a couple hundred thousand routes.

In cases where the desired IPv4 route table exceeds this, a different template such as internet-peering can be set

MySwitch(config)# system routing ?
template-dual-stack-host-scale Dual Stack Host Scale
template-internet-peering Internet Peering
template-lpm-heavy LPM Heavy
template-mpls-heavy MPLS Heavy Scale
template-multicast-heavy Multicast Heavy Scale

This requires a reboot and will show a scary message about disabling multicast routing:

MySwitch(config)# system routing template-internet-peering 
Warning: The command will take effect after next reload.
Multicast is not supported in this profile
Increase the LPM scale by resetting multicast LPM max-scale to 0 using below CLI
hardware profile multicast max-limit lpm-entries 0
Note: This requires copy running-config to startup-config before switch reload.

After the reboot, the memory can be carved out to a larger amount

MySwitch(config)#vdc MySwitch
MySwitch(config-vdc)#limit-resource u4route-mem minimum 256 maximum 512
MySwitch(config)#exit

And now we have more TCAM allocated to IPv4 unicast routes:

MySwitch# show vdc MySwitch resource 

Resource                   Min       Max       Used      Unused    Avail   
--------                   ---       ---       ----      ------    -----   
vlan                       16        4094      45        0         4049    
vrf                        2         4096      9         0         4087    
port-channel               0         511       14        0         497     
u4route-mem                512       512       2         510       510     
u6route-mem                96        96        1         95        95      
m4route-mem                58        58        1         57        57      
m6route-mem                8         8         1         7         7

And now we’re able to take about 286k routes from CenturyLink no problem:

Neighbor   V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
4.15.16.9  4  3356  122187     289   297236    0    0 00:04:55 286770

Now the part I still don’t understand is there’s still the same amount of resources allocated to IPv6 unicast and multicast routes.  It’s also not totally clear what the total TCAM memory amount is, but I would assume 1-2 GB.

 

 

Jumbo Frames on Nexus 93180YC-EX, 5672UP, and perhaps others

Cisco’s documentation implies that to enable jumbo frames on the 5K and 9K line, one must simply set mtu 9216 on the physical and logical L1/L2 interfaces:

Configure and Verify Maximum Transmission Unit on Cisco Nexus Platforms

However, currently working with the 93180YC-EX and previously worked with the 5672UP, I can tell you that both are actually based on the obscure 6K line.

And, per the 5672UP documentation, in order to get jumbo frames, you must do this additional step:

policy-map type network-qos jumbo
  class type network-qos class-default
          mtu 9216
system qos
  service-policy type network-qos jumbo

After applying this, also do a set mtu 9216 on the on the L3 SVIs:

Switch(config)#interface Vlan200
  no shutdown
  mtu 9216

Switch# show interface vl200 Vlan200 is up, line protocol is up, autostate enabled Hardware is EtherSVI, address is 70ea.1a44.d0a7 Internet Address is 192.168.200.1/24 MTU 9216 bytes, BW 1000000 Kbit, DLY 10 usec,

Switch# show interface et1/17
Ethernet1/17 is up
admin state is up, Dedicated Interface
Belongs to Po17
Hardware: 100/1000/10000/25000 Ethernet, address: 70ea.1a44.d0b8 (bia 70ea.1a44.d0b8)
Description: Storage Filer
MTU 9216 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
full-duplex, 10 Gb/s, media type is 10G

 

93180YC – Jumbo Frames