Browser to ADFS server works fine, but dies when going through the F5 LTM.  Packet capture showed the F5 would send a client hello SSL handshake message as expected, with the ADFS server responding with a TCP RST.

Upon doing some more digging, found this the ADFS 2016 guide:

The load balancer MUST NOT terminate SSL. AD FS supports multiple use cases with certificate authentication which will break when terminating SSL. Terminating SSL at the load balancer is not supported for any use case.

So, the F5 Virtual server should be configured as Layer 4.

The unsupported work-around is set a custom ServerSSL profile with the server name field:

ltm profile server-ssl /Common/serverssl-myserver {
 app-service none
 defaults-from /Common/serverssl
 server-name adfs.mydomain.com
}