Hi all,
I post here due to the slow response time of our case with VMWare. I had to roll back (power back on and rejoin to AD) our Windows vCenter server 6 since this migration did not work. I thought I had a smooth upgrade going after I found out I had to add our vmware service account the Replace a Process Level Token privelage in windows. Once I fixed that minor thing the migration assistant worked well.
The VCSA deployed, powered off the Windows vCenter machine, the new one joined to AD and then eventually after some time it gave an error.
Analytics Service registration with Component Manager failed. I downloaded a log file and inside the log package more details of the error is found in this file /var/log/firstboot/analytics_firstboot.py_14589_stderr.log
2020-04-13T16:28:06.223Z Failed to register Analytics Service with Component Manager: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:719)
2020-04-13T16:28:06.231Z Traceback (most recent call last):
File "/usr/lib/vmware-analytics/firstboot/analytics_firstboot.py", line 214, in register_with_cm
cloudvm_sso_cm_register(keystore, cisreg_spec, key_alias, dyn_vars, isPatch=is_patch)
File "/usr/lib/vmware-cm/bin/cloudvmcisreg.py", line 706, in cloudvm_sso_cm_register
serviceId = do_lsauthz_operation(cisreg_opts_dict)
File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 997, in do_lsauthz_operation
ls_obj = LookupServiceClient(ls_url, retry_count=60)
File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 307, in __init__
self._init_service_content()
File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 287, in do_retry
return req_method(self, *args, **kargs)
File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 297, in _init_service_content
self.service_content = si.RetrieveServiceContent()
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 557, in <lambda>
self.f(*(self.args + (obj,) + args), **kwargs)
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 363, in _InvokeMethod
return self._stub.InvokeMethod(self, info, args)
File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1385, in InvokeMethod
conn.request('POST', self.path, req, headers)
File "/usr/lib/python3.5/http/client.py", line 1123, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python3.5/http/client.py", line 1168, in _send_request
self.endheaders(body)
File "/usr/lib/python3.5/http/client.py", line 1119, in endheaders
self._send_output(message_body)
File "/usr/lib/python3.5/http/client.py", line 944, in _send_output
self.send(msg)
File "/usr/lib/python3.5/http/client.py", line 887, in send
self.connect()
File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1032, in connect
six.moves.http_client.HTTPSConnection.connect(self)
File "/usr/lib/python3.5/http/client.py", line 1277, in connect
server_hostname=server_hostname)
File "/usr/lib/python3.5/ssl.py", line 385, in wrap_socket
_context=self)
File "/usr/lib/python3.5/ssl.py", line 760, in __init__
self.do_handshake()
File "/usr/lib/python3.5/ssl.py", line 996, in do_handshake
self._sslobj.do_handshake()
File "/usr/lib/python3.5/ssl.py", line 641, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:719)
2020-04-13T16:28:06.233Z Exception: Traceback (most recent call last):
File "/usr/lib/vmware-analytics/firstboot/analytics_firstboot.py", line 214, in register_with_cm
cloudvm_sso_cm_register(keystore, cisreg_spec, key_alias, dyn_vars, isPatch=is_patch)
File "/usr/lib/vmware-cm/bin/cloudvmcisreg.py", line 706, in cloudvm_sso_cm_register
serviceId = do_lsauthz_operation(cisreg_opts_dict)
File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 997, in do_lsauthz_operation
ls_obj = LookupServiceClient(ls_url, retry_count=60)
File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 307, in __init__
self._init_service_content()
File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 287, in do_retry
return req_method(self, *args, **kargs)
File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 297, in _init_service_content
self.service_content = si.RetrieveServiceContent()
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 557, in <lambda>
self.f(*(self.args + (obj,) + args), **kwargs)
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 363, in _InvokeMethod
return self._stub.InvokeMethod(self, info, args)
File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1385, in InvokeMethod
conn.request('POST', self.path, req, headers)
File "/usr/lib/python3.5/http/client.py", line 1123, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python3.5/http/client.py", line 1168, in _send_request
self.endheaders(body)
File "/usr/lib/python3.5/http/client.py", line 1119, in endheaders
self._send_output(message_body)
File "/usr/lib/python3.5/http/client.py", line 944, in _send_output
self.send(msg)
File "/usr/lib/python3.5/http/client.py", line 887, in send
self.connect()
File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1032, in connect
six.moves.http_client.HTTPSConnection.connect(self)
File "/usr/lib/python3.5/http/client.py", line 1277, in connect
server_hostname=server_hostname)
File "/usr/lib/python3.5/ssl.py", line 385, in wrap_socket
_context=self)
File "/usr/lib/python3.5/ssl.py", line 760, in __init__
self.do_handshake()
File "/usr/lib/python3.5/ssl.py", line 996, in do_handshake
self._sslobj.do_handshake()
File "/usr/lib/python3.5/ssl.py", line 641, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:719)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/vmware-analytics/firstboot/analytics_firstboot.py", line 314, in main
fb.register_with_cm(analytics_int_http, is_patch)
File "/usr/lib/vmware-analytics/firstboot/analytics_firstboot.py", line 225, in register_with_cm
problem_id='install.analytics.cmregistration.failed')
cis.baseCISException.BaseInstallException: {
"componentKey": "analytics",
"detail": [
{
"id": "install.analytics.cmregistration.failed",
"localized": "Analytics Service registration with Component Manager failed.",
"translatable": "Analytics Service registration with Component Manager failed."
}
],
"problemId": "install.analytics.cmregistration.failed",
"resolution": {
"id": "install.analytics.cmregistration.failed.res",
"localized": "Please search for these symptoms in the VMware Knowledge Base for any known issues and possible resolutions. If none can be found, collect a support bundle and open a support request.",
"translatable": "Please search for these symptoms in the VMware Knowledge Base for any known issues and possible resolutions. If none can be found, collect a support bundle and open a support request."
}
}
2020-04-13T16:28:06.233Z VMware Analytics Service firstboot failed
I google search this and found this vmware kb
https://kb.vmware.com/s/article/67198
This lead me down the rabbit hole of enabling ssh and bash so I could winscp our root cert ca pem file onto the box. I did this and copied our root ca certificate which is a windows 2012 R2 certificate authority to /etc/ssl/certs.
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert /etc/ssl/certs/Root-CA.cert.pem
It imported successfully (or so it told me)
I hit retry and it was the same error. I then rebooted the new VCSA thinking maybe it needs to be rebooted to take affect. It rebooted but changed its IP address to the final IP that the old windows vcenter was. So I changed it back to the temporary ip address using DCUI so the migration wizard could contact it. I still get the same error when retrying the migration wizard that is running on my windows box "Analytics Service registration with Component Manager failed".
I then followed this guide to try to replace the vSPhere 6.0 Machine SSL certificate with a VMCA issued certificate
https://kb.vmware.com/s/article/2112279
However I get an error and tells me to check the and in the certificate-manager.log. An excerpt from that log right around the ERROR lines:
2020-04-13T16:52:47.565Z INFO certificate-manager MACHINE_SSL_CERT certificate replaced successfully. SerialNumber and Thumbprint changed.
2020-04-13T16:52:47.662Z INFO certificate-manager lstool command currently being executed is : ['/usr/java/jre-vmware/bin/java', '-Djava.security.properties=/etc/vmware/java/vmware-override-java.security', '-cp', '/usr/lib/vmidentity/tools/lib/lookup-client.jar:/usr/lib/vmidentity/tools/lib/*', '-Dlog4j.configuration=tool-log4j.properties', 'com.vmware.vim.lookup.client.tool.LsTool', 'get-site-id', '--no-check-cert', '--url', 'https://drvcenter.diamondcu.com:443/lookupservice/sdk']
2020-04-13T16:52:49.487Z ERROR certificate-manager 'lstool get-site-id' failed: 1
2020-04-13T16:52:49.490Z ERROR certificate-manager Error while replacing Machine SSL Cert, please see /var/log/vmware/vmcad/certificate-manager.log for more information.
2020-04-13T16:52:49.490Z ERROR certificate-manager 'lstool get-site-id' failed: 1
2020-04-13T16:52:49.492Z INFO certificate-manager Performing rollback of Machine SSL Cert...
Now, not sure what to do, and just crickets with vmware support... I powered off the VCSA, powered back on the windows vcenter, rejoined to to AD and rebooted and now were back to vcenter 6.0 update 3.
Any ideas? The vmware certificates always put us through utter hell on the windows environment and it seems its going to continue to be that way on the VCSA.
We really would like to migrate off of the Windows Server 2008 R2 vms running vcenter 6.0. This is our DR site. I haven't even touched production yet. I wouldn't be surprised if our production site will give us difficulties. I can't install the latest windows vcenter 6.0 patch... error 1603 starting some service... and none of the kb articles helped (removing some vmware java stuff, etc..). So I can't wait to get off of the Windows platform and upgrade.
DR Vcenter 6.0.0 build 14510545 - attempting first. - Running on fully patched Windows Server 2008 R2
SRM virtual appliance 8.2
vSphere replicaiton virtual appliance 8.2 (its receiving inbound from hq)
2 ESXi 6.0.0, 15169789 hosts - HP 380g8's, eventually will take them to latest ESXi 6.5 build using HP's custom image. Hosts are not officially supported past 6.5.
HQ Vcenter 6.0.0 build 9313458 - will do second. - Running on fully patched Windows Server 2008 R2
SRM virtual appliance 8.2
vSphere replicaiton virtual appliance 8.2 (Its replicating 28 vms to the DR site).
8 ESXi 6.0.0, 15169789 hosts. Dell FC640's, eventually will take them to latest ESXi 6.7 build using Dells custom image.