Sorry for the long subject line, but I wanted to be clear on the issue.
I have deployed vCenter 6.7, and I have two HP DL360 Gen 10 boxes that I'm installing vSphere 6.7 on. These 2 boxes have hardware that is all on the HCL, and I'm using 4 SSDs in each of them for vSAN. This includes the 2 port 10Gb adapter that is direct connected between the two.
I have logged into each of the hosts and set the management interface to also carry the vSAN witness traffic. The vSan witness box is in a different VLAN and in a different Site, but the connection is definitely sufficient according to the documentation, it's got 100Mbps with rtt averaging around 10ms.
I just setup my vSAN cluster, and everything on the cluster in health is passing checks, however shortly after the build the witness server throws two warnings:
Host cannot communicate with one or more other nodes in the vSAN enabled cluster
Host with vSAN service enabled is not in the vCenter cluster
I can ping the witness server from either of the two nodes in the cluster, and from the same vlan I can ping back to the two servers, so I'm not sure why these warnings are showing up.
Also I can't put the hosts in the cluster in maintenance mode since the witness server can't communicate with them.
I thought it might be to do with routing, even though the default routes in my network should handle it. I created routes on the two esxi servers to direct their traffic to the witness host through the default gateway, but this had no effects.