Hello,
I have 2 vcenter 6.5 servers in one sso domain.
Some time ago I have seen alarm about certificate. So i renewed all from gui, but alarm persisted.
Today I could not log in to vspere web ui, had error "User name and password are required" on the first server, and blank screen on the second one. So i restarted both. After reboot
503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00005567895eb3d0] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)
error is displayed.
In /var/log/vmware/messages there was error:
faultstring: The token authority rejected an issue request for TimePeriod [startTime=Fri May 08 06:42:07 UTC 2020, endTime=Fri May 08 06:52:07 UTC 2020] :: Signing certificate is not valid at Fri May 08 06:42:07 UTC 2020, cert validity: TimePeriod [startTime=Tue May 08 20:01:11 UTC 2018, endTime=Thu May 07 20:01:11 UTC 2020]
So I have decided to generate new certificates from cli with /usr/lib/vmware-vmca/bin/certificate-manager. Tried regenerating and resetting, but both operations failed when manager tried to start services. Anyway, using cmd /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text showed newly generated certificates.
# service-control --status
Running:
applmgmt lwsmd vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-eam vmware-psc-client vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vmon vmware-vpostgres vsphere-client vsphere-ui
Stopped:
pschealth vmcam vmware-cm vmware-content-library vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-rbd-watchdog vmware-sps vmware-updatemgr vmware-vapi-endpoint vmware-vcha vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm
Don't know where to look at, but below are some logs. Seems like something with SSO.
grep -i error /var/log/vmware/vpxd/vpxd.log
2020-05-08T12:24:43.997Z error vpxd[7F6D42D58800] [Originator@6876 sub=[SSO][SsoFactory_CreateFacade]] Unable to create SSO facade: N5Vmomi5Fault11SystemError9ExceptionE(vmodl.fault.SystemError)
2020-05-08T12:24:43.999Z error vpxd[7F6D42D58800] [Originator@6876 sub=Main] Init failed. SystemError: N5Vmomi5Fault11SystemError9ExceptionE(vmodl.fault.SystemError)
2020-05-08T12:24:43.999Z error vpxd[7F6D42D58800] [Originator@6876 sub=Default] Failed to intialize VMware VirtualCenter. Shutting down
vapi/endpoint/endpoint.log
2020-05-08T09:35:46.938Z | ERROR | state-manager1 | ComponentManagerClientWrapper | SSO lookup failed.
java.util.concurrent.ExecutionException: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
at com.vmware.vim.vmomi.core.impl.BlockingFuture.get(BlockingFuture.java:81)
at com.vmware.cis.cm.client.ComponentManagerClient.lookup(ComponentManagerClient.java:876)
at com.vmware.cis.cm.client.ComponentManagerClient$3.call(ComponentManagerClient.java:939)
at com.vmware.cis.cm.client.ComponentManagerClient$3.call(ComponentManagerClient.java:933)
at com.vmware.cis.cm.client.ComponentManagerClient.retry(ComponentManagerClient.java:548)
at com.vmware.cis.cm.client.ComponentManagerClient.cachedLookup(ComponentManagerClient.java:929)
at com.vmware.cis.cm.client.ComponentManagerClient.cachedLookup(ComponentManagerClient.java:908)
at com.vmware.cis.cm.client.ComponentManagerClient.lookupSso(ComponentManagerClient.java:993)
at com.vmware.vapi.endpoint.cis.ComponentManagerClientWrapper.lookupSso(ComponentManagerClientWrapper.java:171)
at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.ssoSettings(SsoSettingsBuilder.java:171)
at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:56)
at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:354)
at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:168)
at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:151)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:256)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51)
... 3 more
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:140)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:45)
... 3 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:72)
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:117)
... 13 more
2020-05-08T09:35:46.941Z | ERROR | state-manager1 | SsoSettingsBuilder | Failded to retrieve SSO settings.
com.vmware.vapi.endpoint.config.ConfigurationException: SSO lookup failed.
at com.vmware.vapi.endpoint.cis.ComponentManagerClientWrapper.lookupSso(ComponentManagerClientWrapper.java:174)
at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.ssoSettings(SsoSettingsBuilder.java:171)
at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:56)
at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:354)
at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:168)
at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:151)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:256)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51)
... 3 more
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:140)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:45)
... 3 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:72)
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:117)
... 13 more
2020-05-08T09:35:46.941Z | INFO | state-manager1 | HealthStatusCollectorImpl | HEALTH ORANGE Failed to retrieve SSO settings from component manager.
2020-05-08T09:35:46.941Z | ERROR | state-manager1 | DefaultStateManager | Could not initialize endpoint runtime state.
com.vmware.vapi.endpoint.config.ConfigurationException: Failed to retrieve SSO settings.
at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:63)
at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:354)
at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:168)
at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:151)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.vmware.vapi.endpoint.config.ConfigurationException: SSO lookup failed.
at com.vmware.vapi.endpoint.cis.ComponentManagerClientWrapper.lookupSso(ComponentManagerClientWrapper.java:174)
at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.ssoSettings(SsoSettingsBuilder.java:171)
at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:56)
... 10 more
Caused by: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:256)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51)
... 3 more
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:140)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:45)
... 3 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:72)
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:117)
... 13 more
2020-05-08T09:35:46.941Z | INFO | state-manager1 | HealthStatusCollectorImpl | HEALTH ORANGE Application error has occurred. Please check log files for more information.
2020-05-08T09:35:46.941Z | INFO | state-manager1 | HealthStatusCollectorImpl | HEALTH GREEN Current vApi Endpoint health status is created between 2020-05-08T09:35:46UTC and 2020-05-08T09:35:46UTC.
2020-05-08T09:35:46.941Z | INFO | state-manager1 | HealthConfigurationEventListener | Computed health status is = ORANGE
2020-05-08T09:35:46.941Z | INFO | state-manager1 | HealthConfigurationEventListener | HEALTH Failed to retrieve SSO settings from component manager.
2020-05-08T09:35:46.941Z | INFO | state-manager1 | HealthConfigurationEventListener | HEALTH Application error has occurred. Please check log files for more information.
2020-05-08T09:35:46.941Z | INFO | state-manager1 | HealthConfigurationEventListener | HEALTH Current vApi Endpoint health status is created between 2020-05-08T09:35:46UTC and 2020-05-08T09:35:46UTC.
2020-05-08T09:35:46.941Z | INFO | state-manager1 | DefaultStateManager | lock
2020-05-08T09:35:46.942Z | INFO | state-manager1 | DefaultStateManager | Initial state build failed. Will retry after 5 seconds.
2020-05-08T09:35:46.942Z | INFO | state-manager1 | DefaultStateManager | unlock
2020-05-08T09:35:50.607Z | INFO | shutdown-hook | ApiEndpointServer | Start shutting down...
2020-05-08T09:35:50.607Z | INFO | shutdown-hook | DefaultStateManager | shutdown
2020-05-08T09:35:50.614Z | INFO | shutdown-hook | ApiEndpointServer | Shutdown.
But couldn't find nothing intresting in sso
sso/ssoAdminServer.log
[2020-05-08T12:24:43.988Z pool-6-thread-5 opId=21190014-abf6-4825-8234-ea672d2cdbb0 ERROR com.vmware.vim.vmomi.server.http.impl.CompletionContinuerTask] Failed to serialize response
com.vmware.vim.binding.vmodl.fault.SystemError: Failed to serialize response
at com.vmware.vim.vmomi.server.exception.ExceptionUtil.buildFaultForInternalException(ExceptionUtil.java:22) ~[vlsi-server.jar:?]
at com.vmware.vim.vmomi.server.http.impl.CompletionContinuerTask.complete(CompletionContinuerTask.java:95) [vlsi-server.jar:?]
at com.vmware.vim.vmomi.server.http.impl.CompletionContinuerTask.complete(CompletionContinuerTask.java:63) [vlsi-server.jar:?]
at com.vmware.vim.vmomi.server.http.impl.CompletionContinuerTask.run(CompletionContinuerTask.java:53) [vlsi-server.jar:?]
at com.vmware.vim.vmomi.server.common.impl.RunnableWrapper$1.run(RunnableWrapper.java:47) [vlsi-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_221]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_221]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_221]
Caused by: com.vmware.vim.vmomi.core.exception.MarshallException: Missing non-optional return value
at com.vmware.vim.vmomi.server.impl.SoapBindingImpl.serializeResponse(SoapBindingImpl.java:168) ~[vlsi-server.jar:?]
at com.vmware.vim.vmomi.server.http.impl.CompletionContinuerTask.complete(CompletionContinuerTask.java:84) ~[vlsi-server.jar:?]
... 6 more
sso/vmware-identity-sts.log //still some problem with certificate?
[2020-05-08T12:58:34.733Z tomcat-http--39 vsphere.local 97b81203-5e98-4499-b11f-1f5b6ade0789 INFO com.vmware.identity.sts.ws.SOAPFaultHandler] Returning a SOAP Fault with code: ns0:InvalidTimeRange and description: The token authority rejected an issue request for TimePeriod [startTime=Fri May 08 12:58:34 UTC 2020, endTime=Fri May 08 13:08:34 UTC 2020] :: Signing certificate is not valid at Fri May 08 12:58:34 UTC 2020, cert validity: TimePeriod [startTime=Tue May 08 20:01:11 UTC 2018, endTime=Thu May 07 20:01:11 UTC 2020]
sso/lookupServer.log
[2020-05-08T08:09:40.314Z ERROR] [OpenLdapClientLibrary] Exception when calling ldap_search_s: base=cn=4cea3f17-670c-4ee6-938c-c7e1aaec7cfe,cn=ServiceRegistrations,cn=LookupService,cn=silp,cn=sites,cn=configuration,dc=vsphere,dc=local, scope=2, filter=(objectclass=*), attrs=null, attrsonly=0
com.vmware.identity.interop.ldap.NoSuchObjectLdapException: No such object
Any help would be appreciated