I applied the latest Hotfix to our GTMs tonight and was checking logs just to verify there were no surprises. Unfortunately, there were: /var/log/gtm was showing SSL errors every 10 seconds complaining of being unable to verify the certificates. I check the self-sign certs, and sure enough, the had expired a few days ago.
The first step to fix is obvious – renew the cert via System -> Device Certificate -> Device Certificate. The only field that really matters here is hostname and duration; everything else can be left to defaults.
Now re-import each others new certificates System -> Device Certificate -> Trusted Device certificates.
Sync and failover between the BigIP devices is now fixed, buuuuuut the logs show that gtmd is still not happy:
Feb 21 18:07:20 bigip01 notice gtmd[13701]: 011ae020:5: Connection in progress to 192.168.1.2 Feb 21 18:07:20 bigip01 notice gtmd[13701]: 011ae01c:5: Connection complete to 192.168.1.2. Starting SSL handshake Feb 21 18:07:20 bigip01 iqmgmt_ssl_connect: SSL error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed Feb 21 18:07:20 bigip01 err gtmd[13701]: 011ae0fa:3: iqmgmt_ssl_connect: SSL error: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed (336134278)
This is happening on the secondary as well:
Feb 21 18:20:26 bigip02 iqmgmt_ssl_connect: SSL error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed Feb 21 18:20:26 bigip02 err gtmd[13788]: 011ae0fa:3: iqmgmt_ssl_connect: SSL error: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed (336134278)
As you may have guessed, GTM needs a separate step to pick up the cert changes. The solution is here is not all that obvious – run bigip_add on both systems to copy the certs via SSH and dump them in to /config/gtm/server.crt. Before doing it, keep in mind there’s two requirements for this command to succeed:
- The Self IP must be permit SSH. Since this is a one-time exchange, you can use management IPs even if the GTM server IP is something different. For non-management IPs, “Port Lockdown: Allow All” should be set.
- The user must be set to use Advanced Shell aka BASH (not tmsh). The simplest way to override this to login as ‘root’, since it’s a pre-defined user that is inherently set for Advanced Shell
[admin@bigip01:Active:In Sync] ~ # bigip_add root@10.1.1.2 Retrieving remote and installing local BIG-IP's SSL certs ... Enter root password for 10.1.1.2 if prompted ==> Done <== [admin@bigip02:Active:In Sync] ~ # bigip_add root@10.1.1.1 Retrieving remote and installing local BIG-IP's SSL certs ... Enter root password for 10.1.1.1 if prompted ==> Done <==
Boom! GTM is now happy now and I can go home.
Feb 21 18:25:11 bigip01 alert gtmd[13701]: 011a500b:1: SNMP_TRAP: Box 192.168.1.2 state change blue --> green Feb 21 18:26:16 bigip02 alert gtmd[13788]: 011a500b:1: SNMP_TRAP: Box 192.168.1.1 state change blue --> green
