I applied the latest Hotfix to our GTMs tonight and was checking logs just to verify there were no surprises.  Unfortunately, there were: /var/log/gtm was showing SSL errors every 10 seconds complaining of being unable to verify the certificates.  I check the self-sign certs, and sure enough, the had expired a few days ago.

The first step to fix is obvious – renew the cert via System -> Device Certificate -> Device Certificate.  The only field that really matters here is hostname and duration; everything else can be left to defaults.

bigip_device_certificate_renew

Now re-import each others new certificates System -> Device Certificate -> Trusted Device certificates.

bigip_trusted_device_certificate_import

Sync and failover between the BigIP devices is now fixed, buuuuuut the logs show that gtmd is still not happy:

Feb 21 18:07:20 bigip01 notice gtmd[13701]: 011ae020:5: Connection in progress to 192.168.1.2 
Feb 21 18:07:20 bigip01 notice gtmd[13701]: 011ae01c:5: Connection complete to 192.168.1.2. Starting SSL handshake
Feb 21 18:07:20 bigip01 iqmgmt_ssl_connect: SSL error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
Feb 21 18:07:20 bigip01 err gtmd[13701]: 011ae0fa:3: iqmgmt_ssl_connect: SSL error: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed (336134278)

This is happening on the secondary as well:

Feb 21 18:20:26 bigip02 iqmgmt_ssl_connect: SSL error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
Feb 21 18:20:26 bigip02 err gtmd[13788]: 011ae0fa:3: iqmgmt_ssl_connect: SSL error: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed (336134278)

As you may have guessed, GTM needs a separate step to pick up the cert changes.  The solution is here is not all that obvious – run bigip_add on both systems to copy the certs via SSH and dump them in to /config/gtm/server.crt.  Before doing it, keep in mind there’s two requirements for this command to succeed:

  1. The Self IP must be permit SSH.  Since this is a one-time exchange, you can use management IPs even if the GTM server IP is something different.   For non-management IPs, “Port Lockdown: Allow All” should be set.
  2. The user must be set to use Advanced Shell aka BASH (not tmsh).  The simplest way to override this to login as ‘root’, since it’s a pre-defined user that is inherently set for Advanced Shell
[admin@bigip01:Active:In Sync] ~ # bigip_add root@10.1.1.2
Retrieving remote and installing local BIG-IP's SSL certs ...
Enter root password for 10.1.1.2 if prompted
==> Done <==

[admin@bigip02:Active:In Sync] ~ # bigip_add root@10.1.1.1
Retrieving remote and installing local BIG-IP's SSL certs ...
Enter root password for 10.1.1.1 if prompted
==> Done <==

Boom!  GTM is now happy now and I can go home.

Feb 21 18:25:11 bigip01 alert gtmd[13701]: 011a500b:1: SNMP_TRAP: Box 192.168.1.2 state change blue --> green
Feb 21 18:26:16 bigip02 alert gtmd[13788]: 011a500b:1: SNMP_TRAP: Box 192.168.1.1 state change blue --> green