Upping the IPv4 Unicast Route Limit on a Nexus 93180YC-EX

We attempted to load a partial route table from CenturyLink (aka Level3) on a Cisco Nexus 93180YC-EX and found the switch threw IPFIB-SLOT1-2-UFIB_ROUTE_CREATE error messages starting at around 200,000 routes:

IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 9, 202.153.28.0/24, flags:0x0, intf:0xd001a, Error: FIB TCAM FULL For IP Routes(1129381967)
IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 9, 202.153.27.0/24, flags:0x0, intf:0xd001a, Error: FIB TCAM FULL For IP Routes(1129381967)
IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 9, 202.153.26.0/24, flags:0x0, intf:0xd001a, Error: FIB TCAM FULL For IP Routes(1129381967)

This command shed some insight on the problem:

MySwitch# show vdc MySwitch resource

Resource                   Min       Max       Used      Unused    Avail   
--------                   ---       ---       ----      ------    -----   
vlan                       16        4094      45        0         4049    
vrf                        2         4096      9         0         4087    
port-channel               0         511       14        0         497     
u4route-mem                248       248       2         246       246     
u6route-mem                96        96        1         95        95      
m4route-mem                58        58        1         57        57      
m6route-mem                8         8         1         7         7

So by default,  only 248 MB of the switch’s TCAM is allocated to IPv4 unicast routes.  This means in a typical 2 ISP deployment, it won’t be able to handle more than a couple hundred thousand routes.

In cases where the desired IPv4 route table exceeds this, a different template such as internet-peering can be set

MySwitch(config)# system routing ?
template-dual-stack-host-scale Dual Stack Host Scale
template-internet-peering Internet Peering
template-lpm-heavy LPM Heavy
template-mpls-heavy MPLS Heavy Scale
template-multicast-heavy Multicast Heavy Scale

This requires a reboot and will show a scary message about disabling multicast routing:

MySwitch(config)# system routing template-internet-peering 
Warning: The command will take effect after next reload.
Multicast is not supported in this profile
Increase the LPM scale by resetting multicast LPM max-scale to 0 using below CLI
hardware profile multicast max-limit lpm-entries 0
Note: This requires copy running-config to startup-config before switch reload.

After the reboot, the memory can be carved out to a larger amount

MySwitch(config)#vdc MySwitch
MySwitch(config-vdc)#limit-resource u4route-mem minimum 256 maximum 512
MySwitch(config)#exit

And now we have more TCAM allocated to IPv4 unicast routes:

MySwitch# show vdc MySwitch resource 

Resource                   Min       Max       Used      Unused    Avail   
--------                   ---       ---       ----      ------    -----   
vlan                       16        4094      45        0         4049    
vrf                        2         4096      9         0         4087    
port-channel               0         511       14        0         497     
u4route-mem                512       512       2         510       510     
u6route-mem                96        96        1         95        95      
m4route-mem                58        58        1         57        57      
m6route-mem                8         8         1         7         7

And now we’re able to take about 286k routes from CenturyLink no problem:

Neighbor   V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
4.15.16.9  4  3356  122187     289   297236    0    0 00:04:55 286770

Now the part I still don’t understand is there’s still the same amount of resources allocated to IPv6 unicast and multicast routes.  It’s also not totally clear what the total TCAM memory amount is, but I would assume 1-2 GB.

 

 

Advertisement

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s