We attempted to load a partial route table from CenturyLink (aka Level3) on a Cisco Nexus 93180YC-EX and found the switch threw IPFIB-SLOT1-2-UFIB_ROUTE_CREATE error messages starting at around 200,000 routes:
IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 9, 202.153.28.0/24, flags:0x0, intf:0xd001a, Error: FIB TCAM FULL For IP Routes(1129381967) IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 9, 202.153.27.0/24, flags:0x0, intf:0xd001a, Error: FIB TCAM FULL For IP Routes(1129381967) IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 9, 202.153.26.0/24, flags:0x0, intf:0xd001a, Error: FIB TCAM FULL For IP Routes(1129381967)
This command shed some insight on the problem:
MySwitch# show vdc MySwitch resource Resource Min Max Used Unused Avail -------- --- --- ---- ------ ----- vlan 16 4094 45 0 4049 vrf 2 4096 9 0 4087 port-channel 0 511 14 0 497 u4route-mem 248 248 2 246 246 u6route-mem 96 96 1 95 95 m4route-mem 58 58 1 57 57 m6route-mem 8 8 1 7 7
So by default, only 248 MB of the switch’s TCAM is allocated to IPv4 unicast routes. This means in a typical 2 ISP deployment, it won’t be able to handle more than a couple hundred thousand routes.
In cases where the desired IPv4 route table exceeds this, a different template such as internet-peering can be set
MySwitch(config)# system routing ? template-dual-stack-host-scale Dual Stack Host Scale template-internet-peering Internet Peering template-lpm-heavy LPM Heavy template-mpls-heavy MPLS Heavy Scale template-multicast-heavy Multicast Heavy Scale
This requires a reboot and will show a scary message about disabling multicast routing:
MySwitch(config)# system routing template-internet-peering
Warning: The command will take effect after next reload.
Multicast is not supported in this profile
Increase the LPM scale by resetting multicast LPM max-scale to 0 using below CLI
hardware profile multicast max-limit lpm-entries 0
Note: This requires copy running-config to startup-config before switch reload.
After the reboot, the memory can be carved out to a larger amount
MySwitch(config)#vdc MySwitch MySwitch(config-vdc)#limit-resource u4route-mem minimum 256 maximum 512 MySwitch(config)#exit
And now we have more TCAM allocated to IPv4 unicast routes:
MySwitch# show vdc MySwitch resource Resource Min Max Used Unused Avail -------- --- --- ---- ------ ----- vlan 16 4094 45 0 4049 vrf 2 4096 9 0 4087 port-channel 0 511 14 0 497 u4route-mem 512 512 2 510 510 u6route-mem 96 96 1 95 95 m4route-mem 58 58 1 57 57 m6route-mem 8 8 1 7 7
And now we’re able to take about 286k routes from CenturyLink no problem:
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 4.15.16.9 4 3356 122187 289 297236 0 0 00:04:55 286770
Now the part I still don’t understand is there’s still the same amount of resources allocated to IPv6 unicast and multicast routes. It’s also not totally clear what the total TCAM memory amount is, but I would assume 1-2 GB.