Uncategorized

Upgrading Cacti from 0.8.8 to 1.0.6

J5Leave a comment

Not too painful.  Cacti 1.x requires a few PHP modules that the legacy versions did not.  This was an easy fix in CentOS:

# yum install php-ldap
# yum install php-posix
# yum install php-mbstring
# apachectl restart

The next trick was on the database side.  I’d been using a MySQL / MariaDB 5.5 server hosted on a QNAP.  It looks like the installation script was reading the local /etc/my.cnf file to get tuning parameters, and complaining about several settings.  I ignored the warnings and proceeded.

The database upgrade took several seconds, but did complete.  However, upon login in via web, some pages were loading incomplete.  Fix was to reset the Server Base URL setting, since without it, the CSSes weren’t referenced properly.

cacti_server_base_url.png

That’s it.  Good to see Cacti finally on the 1.0 side of things

F5 LTM: Working with Data Groups via TMSH

J5Leave a comment

As my data group lists get larger and larger, I’ve needed to step away from the GUI to manage them.  API will be used eventually but for now TMSH will be the quick and dirty method of managing and verifying entries.

Note: Modifying data-groups via TMSH requires Administrator privileges.

modify ltm data-group internal fruits records add {“orange”}

To add multiple entries at once, just separate via comma:

modify ltm data-group internal fruits records add {“apple” “pear”}

To enter name value pairs, put the Value in another set of curly braces:

modify ltm data-group internal fruits records add { “banana” { data “yellow”} }

To add a value to a pre-existing key:

modify ltm data-group internal fruits records modify { "orange" { data "orange" }}

Delete a key:

modify ltm data-group internal fruits records delete {"pear"}

Now to list the values:

list ltm data-group internal fruits

BGP: remove-private-as command

J5Leave a comment

Take these two original paths, sent by router with ASN 111 to an eBGP peer:

10.0.0.0/16 169.254.1.2 0 100 111 65111 65222 7224
10.1.0.0/16 169.254.1.2 0 100 111 65111 65222 222 7224

Applying remove-private-as has no effect on the second, since the path contains public ASNs.  The paths receiveced by the eBGP peer will look like this:

10.0.0.0/16 169.254.1.2 0 100 7224
10.1.0.0/16 169.254.1.2 0 100 111 65111 65222 222 7224

But doing remove-private-as all will literally remove all private ASNs:

10.0.0.0/16 169.254.1.2 0 100 7224
10.1.0.0/16 169.254.1.2 0 100 111 222 7224

However with remove-private-as all replace-as the path will have the same length, but with each private substituted with public of the advertising peer:

10.0.0.0/16 169.254.1.2 0 100 111 111 7224
10.1.0.0/16 169.254.1.2 0 100 111 111 111 222 7224

 

How many 3750-E switches can an RPS 2300 backup?

J5Leave a comment

The Q&A sheet mentions that with dual 1150W power supplies, an RPS can backup 1 or 2 3750-E switches.  This is assuming the switches also have 1150W power supplies installed.

But what if they’re only using 750W?  The RPS would have a total of 2300W of power, while three of the switches would only require 2250W.    So it should be able to backup three switches, right?

Nope.  You can only backup two.  So, there’s actually really no point of installing 1150W power supplies in the RPS.

Switch#show env rps
DCOut State Connected Priority BackingUp WillBackup Portname SW#
----- ------- --------- -------- --------- ---------- --------------- ---
 1 Active Yes 6 Yes Yes FDO1525Y1T5 1
 2 Active No 6 No No <> -
 3 Active Yes 6 Yes Yes FDO1417R07E 3
 4 Active No 6 No No <> -
 5 Active Yes 6 No No FDO1406R1KU 5
 6 Active No 6 No No <> -

Yet another reason why the RPS sucks and StackPower on the 3750X and 3850 series is so much better.

DNS Resolution via VRF on Cisco IOS

J54 Comments

For several years I’ve been using VRFs for all management functions.  This greatly improves security since all management functions can be locked down to a certain interface, and also recover-ability in the even of routing problems.  The downside I keep finding is certain things either don’t work, or require special work-rounds. Case in point: DNS resolution.

Per Cisco, VRF-aware DNS functionality has been supported for quite a while.  However, I’m completely stumped on how to actually use it.  Sample config on an 2921 router running IOS 15.5(3)M4:

ip vrf Mgmt-intf
 rd 12345:123
!
ip domain-lookup 
ip domain list vrf Mgmt-intf mydomain.com
ip name-server vrf Mgmt-intf 10.20.30.40

!
interface Port-channel1.123
 encapsulation dot1Q 123
 ip vrf forwarding Mgmt-intf
 ip address 10.20.30.100 255.255.255.0
!
ip domain-lookup vrf Mgmt-intf source-interface po1.123

Still no joy.  Really seems there was a goof here in enabling this feature.  I’ll complain to Cisco and hopefully it will be fixed by the time I die.

Commands to Restore Cisco ACS from backup

J5Leave a comment

We’re still running ACS 5.4 patch 4, which was always buggy, but has gotten especially painful to manage via modern browsers.  Over the last few weeks I’ve realized this has now gone to catastrophic.  If editing a policy with say FireFox 49, trying to make a change will cause the entire policy to be deleted without being prompted.  It’s definitely time to patch, but in the meantime I needed to restore from backup.  So I SSH in to ACS, find last night’s backup file, and go to restore:

acs01/admin# restore acs01-backup-161004-0000.tar.gpg repository MyFTP  application acs 
Restore may require a restart of application services. Continue? (yes/no) [yes] ? yes
Initiating restore.  Please wait...
Backup file does not match installed application
% Application restore failed

Hmm….the application name is ‘acs’.  Maybe I have to put it in UPPER case?!?

acs01/admin# restore acs01-backup-161004-0000.tar.gpg repository MyFTP application ACS
Restore may require a restart of application services. Continue? (yes/no) [yes] ? yes
Initiating restore.  Please wait...
Calculating disk size for /opt/backup/restore-acs01-backup-161004-0000.tar.gpg-1475607189
Total size of restore files are 331 M.
Max Size defined for restore files are 105573 M.
% Backup file does not match installed application(s)

OK, now I’m concerned.  Wait – leave it to Cisco to throw a gotcha.  The “restore” command restores both ACS and the appliance OS.  To restore just ACS configuration, use the “acs restore command”:

acs01/admin# acs restore acs01-backup-161004-0000.tar.gpg repository MyFTP
Restore requires a restart of ACS services. Continue?  (yes/no) yes
Initiating restore.  Please wait...

Bingo!  And a few minutes later, everything is happy.  I logged in using IE8 and was able to make the policy changes without issue.

Ignoring Annoying Syslog messages in Cisco IOS

J5Leave a comment

asdlfkj

Sep 18 14:47:04 PDT: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec’d IPSEC packet has invalid spi for destaddr=198.18.99.123, prot=50, spi=0xA6517B6A(2790357866), srcaddr=82.221.105.6, input interface=Port-channel1.123

On the ASA, something like this is a simple fix.  Each log message has a 6 digit code, and the severity can be modified to say 6 (informational) or 7 (debug) to not trigger a call-home.

Under Cisco IOS of course, it’s not so easy.

 

 

Upgrading from Rancid 2.3.8 to 3.4.1

J53 Comments

I recently handed off management of our Palo Alto firewalls to a co-worker.  His task was to upgrade from 7.0 to 7.1 (which succeeded, but then broke our 8×8 phones…another topic) and asked if there was automated backups of the config files occurring.  My reply was “good question: no….but gee, I wonder if Rancid can do that”?

Turns out it can, beginning version 3.x.  Too bad we were still running 2.3.8.  So it was time to upgrade.  No biggie I thought, and began work on our CentOS 5.9 VM.  Then I remembered what a pain it is to upgrade apps in Linux when they use multiple languages and dependencies.

$ wget ftp://ftp.shrubbery.net/pub/rancid/rancid-3.4.1.tar.gz
$ tar -xzf rancid-3.4.1.tar.gz
$ cd rancid-3.4.1
$ ./configure --prefix=/usr/local --localstatedir=/home/rancid
checking Socket.pm version... Socket version 2.006 required--this is only version 1.78 at -e line 1.
BEGIN failed--compilation aborted at -e line 1.
configure: error: Socket.pm is older than 2.006; upgrade from http://metacpan.org/pod/Socket

Ok, so it needs a newer version of this Perl module.  After numerous Googles I find this is parr of Perl’s CPAN libray.  So I run this:

yum upgrade perl-CPAN

This upgraded from 1.78 to 1.82.  Still quite short of the 2.0006 version required.  So I begin realizing I have a bigger problem: CentOS 5.9 is really old and it’s time to switch to something newer.  Fortunately a co-worker had already built an Ubuntu VM to do some database monitoring, so that problem was solved.

This time I get a different error from configure: expect was not installed.  That was an easy fix, after I remembered how to install packages in Debian/Ubuntu:

$ apt-get install expect

The configure script now passed…well not quite, but we’ll get to that later.  After make & make install, I did a quick edit of the rancid.conf list to set LIST_OF_GROUPS, then next step was rancid-cvs.  This should create a CVS backend, but instead I got no output:

root@localhost:~# su - rancid
 $ rancid-cvs
 $

Uhhhhh weird?  Once I tried to actually run rancid the problem became more clear: cvs wasn’t installed.  How in the world did the configure script not detect that?  Anyhoo, not a big deal because I already know the fix:

apt-get install cvs

So I re-run rancid-cvs and once again, get no output.  More poking around Google shows the solution: blow away anything that was created by rancid-cvs and re-run it:

rm -Rf /home/rancid/*

rancid-cvs now shows output this time and finally looks like weren’t good to go.  But I try a rancid-run and it quits pretty quick, now showing this in the logs:

WARNING: Have you forgotten to update the FS in router.db?

Umm….what’s FS?  After poking around the Googles some more, it seems they changed the file syntax of router.db to use semicolons rather than full colons.  Uhh….ok?  colons aren’t valid in DNS hostnames so I don’t see where the conflict was, but whatever.  Easy fix:

myrouter.mydomain.com;cisco;up

I then found the other upgrade problem: device type changed for Dell PowerConnect 6348 and 8024-k switches.  Using ‘cisco’ doesn’t work now, and I had to switch to ‘smc’

So rancid 3.4.1 is now running happy as can be.  That only took 2 weeks of banging head on wall.  Bring on the containers, because I’m so sick of this crap.

 

 

Enabling Config Syncing on BigIP GTMs

J5Leave a comment

After two weeks of working with the consultant and getting nowhere, I booked a hotel for the weekend

We recently retired data centers, and in the process, transported its GTM to our home office.  Despite being gentle, its single hard drive failed in the process, and once the RMA unit arrive, I was once again reminded how confusing the configuration process is.  The F5 BigIP is loaded with nerd knobs, and the GTM is especially hairy as it has overlapping menus and settings different from the more popular LTM

1) Select the Self IP for Sync communication, and verify iQuery is allowed

This ideally should be an internal interface that has redundancy (for example, is two physical interfaces bonded via LACP).

Usually the internal Self IP is set to “allow all”.  Since I had deliberately set this Self IP as “allow none”, I had to change it to “Allow custom”, and add tcp port 4353.

sol13690

2) Set the config sync source Interface

This will be the step that anyone with F5 HA experience will be familiar with.  However, most will assume it to be under System -> High Availability -> Device Connectivity.

Instead, look under Device Management -> Devices -> and click the self GTM.  Then look under Device Connectivity -> ConfigSync

Note that this is the only step needed under the entire Device Management tree.

3) On the existing GTM, create the other GTM as a server

This is done under DNS -> GSLB -> Servers.  Enter the name and IP address of the other GTM, with the product as “BIG-IP System (Single)”.  Chose the same IP address as the Self IP selected in the previous step.  Set the health monitor to “bigip”

4) Run bigip_add on both units

This will use SSH to exchange SSL certificates.  In order for bigip_add to work, both sides must have SSH enabled, and the account must have advanced shell (bash) set.

admin@(f5bigip01)(cfg-sync In Sync)(Active)(/Common)(tmos)# bash
[admin@f5bigip01:Standby:In Sync] ~ # bigip_add admin@192.168.1.12
Retrieving remote and installing local BIG-IP's SSL certs ...
Password:
==> Done <==

admin@(f5bigip02)(cfg-sync In Sync)(Standby)(/Common)(tmos)# bash
[admin@f5bigip02:Active:In Sync] ~ # bigip_add admin@192.168.1.11
Retrieving remote and installing local BIG-IP's SSL certs ...
Password:
==> Done <==

If the account does not have SSH & bash enabled (which is the default setting), this error appears:

ERROR: Can’t read remote cert via /usr/bin/ssh

Since this is a 1-time exchange, I simply used the management port IP addresses, as it has SSH enabled.

Once this step has been completed, each GTM should have the other’s certificate installed under DNS -> GSLB -> Servers -> Trusted Server Certificates or /config/gtm/server.crt

F5 sol13823

5) On the device whos config you want to overwrite, run gtm_add

This will pull the gtm.conf configuration over iQuery (tcp port 4353). The IP address should match the one that was given in the first step.  In our case, this was the internal interface.

F5 sol13312

NAT Hairpinning on Cisco ISR

J51 Comment

I’ve never had a need to do NAT hairpinning on a Cisco ISR, as I’d typically have a fancy firewall like an ASA doing the work.  However, with this blog now hosted on a NAS inside my home network, I’ve found it necessary to support it.  Hairpinning essentially means the internal server is available via the public (global) IP address, even when coming from the private (local) network.  I didn’t want to forge DNS entries because it’s a pain to manage, and, well, it’s just wrong.

First, here’s my traditional NAT configuration.  Fa0/0 is the public interface connected to the ISP.  BVI is the Layer 3 private interface.

interface FastEthernet0/0
 ip address dhcp
 ip nat outside
!
interface Vlan1
 no ip address
 bridge-group 1
!
interface BVI1
 ip address 192.168.0.1 255.255.255.0
 ip nat inside
!
ip nat inside source list NATLIST interface FastEthernet0/0 overload
ip nat inside source static tcp 192.168.0.100 80 interface FastEthernet0/0 80
!
ip access-list extended NATLIST
 deny ip any 10.0.0.0 0.255.255.255
 deny ip any 172.16.0.0 0.15.255.255
 deny ip any 192.168.0.0 0.0.255.255
 permit ip any any
!
bridge 1 protocol ieee
bridge 1 route ip

Now the new config.  Pretty simple, but there’s a gotcha: the requirement for no ip redirects on both outside and inside interfaces.

interface FastEthernet0/0
 ip address dhcp
 no ip redirects
 ip nat enable
!
interface BVI1
 ip address 192.168.0.1 255.255.255.0
 no ip redirects
 ip nat enable
!
ip nat source list NATLIST interface FastEthernet0/0 overload
ip nat source static tcp 192.168.0.100 80 interface FastEthernet0/0 80

And here comes the gotcha: performance.  After switching to this configuration, my throughput over NAT went from about 90 Mbps to 15 Mbps.  Ouch.  Saw these numbers both on a 2811 and 1841.