Hello, we ran into the issue of the root volume running out of disk space due to very large log files the other day. I manged to clear up the space but I had to hard crash and reboot the vCenter virtual appliance in the process.
The system is back up and working, except vCenter is not starting up.
The issue I am running into is the vmware-invsvc and vmware-vpxd service will not start.
Starting VMware Inventory Service...
Waiting for VMware Inventory Service................................................
WARNING: VMware Inventory Service may have failed to start.
error logs from /var/log/vmware/invsvc/wrapper.log
STATUS | wrapper | 2017/09/25 19:30:51 |
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'wrapper_start': Command line[0] : /usr/sbin/cloudvm-ram-size
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'wrapper_start': Command line[1] : -J
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'wrapper_start': Command line[2] : vmware-invsvc
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'wrapper_start': Command line[3] : -O
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'wrapper_start': Command line[4] : /etc/vmware/java/vmware-invsvc_jvm.conf
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'wrapper_start': Command launched (pid: 4360), blocking for up to 3600 seconds...
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'wrapper_start': Command completed with exit code: 0
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'wrapper_start': Command completed with exit code: 0 Continuing.
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'jvm_prelaunch': Command line[0] : ln
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'jvm_prelaunch': Command line[1] : -f
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'jvm_prelaunch': Command line[2] : /usr/java/jre-vmware/bin/java
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'jvm_prelaunch': Command line[3] : /usr/java/jre-vmware/bin/vmware-invsvc
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'jvm_prelaunch': Command launched (pid: 4363), blocking for up to 15 seconds...
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'jvm_prelaunch': Command completed with exit code: 0
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'jvm_prelaunch': Command completed with exit code: 0 Continuing.
STATUS | wrapper | 2017/09/25 19:30:51 | Launching a JVM...
INFO | jvm 1 | 2017/09/25 19:30:52 | Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M
INFO | jvm 1 | 2017/09/25 19:30:52 | WrapperManager: Initializing...
STATUS | wrapper | 2017/09/25 19:31:44 | <-- Wrapper Stopped
STATUS | wrapper | 2017/09/25 20:39:14 | --> Wrapper Started as Daemon
STATUS | wrapper | 2017/09/25 20:39:14 | Java Service Wrapper Professional Edition 64-bit 3.5.24
STATUS | wrapper | 2017/09/25 20:39:14 | Copyright (C) 1999-2014 Tanuki Software, Ltd. All Rights Reserved.
STATUS | wrapper | 2017/09/25 20:39:14 | http://wrapper.tanukisoftware.com
STATUS | wrapper | 2017/09/25 20:39:14 | Licensed to VMware Global, Inc. for VMware vCenter Inventory Service
STATUS | wrapper | 2017/09/25 20:39:14 |
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'wrapper_start': Command line[0] : /usr/sbin/cloudvm-ram-size
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'wrapper_start': Command line[1] : -J
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'wrapper_start': Command line[2] : vmware-invsvc
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'wrapper_start': Command line[3] : -O
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'wrapper_start': Command line[4] : /etc/vmware/java/vmware-invsvc_jvm.conf
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'wrapper_start': Command launched (pid: 26295), blocking for up to 3600 seconds...
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'wrapper_start': Command completed with exit code: 0
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'wrapper_start': Command completed with exit code: 0 Continuing.
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'jvm_prelaunch': Command line[0] : ln
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'jvm_prelaunch': Command line[1] : -f
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'jvm_prelaunch': Command line[2] : /usr/java/jre-vmware/bin/java
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'jvm_prelaunch': Command line[3] : /usr/java/jre-vmware/bin/vmware-invsvc
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'jvm_prelaunch': Command launched (pid: 26298), blocking for up to 15 seconds...
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'jvm_prelaunch': Command completed with exit code: 0
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'jvm_prelaunch': Command completed with exit code: 0 Continuing.
STATUS | wrapper | 2017/09/25 20:39:14 | Launching a JVM...
INFO | jvm 1 | 2017/09/25 20:39:14 | Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M
INFO | jvm 1 | 2017/09/25 20:39:15 | WrapperManager: Initializing...
STATUS | wrapper | 2017/09/25 20:39:59 | <-- Wrapper Stopped
vcenter:/var/log/vmware/invsvc # cat vmware-invsvc-gc.log.0
2017-09-25T20:39:17.314+0000: 2.794: [GC2017-09-25T20:39:17.330+0000: 2.810: [SoftReference, 0 refs, 0.0003020 secs]2017-09-25T20:39:17.330+0000: 2.810: [WeakReference, 1097 refs, 0.0002410 secs]2017-09-25T20:39:17.330+0000: 2.810: [FinalReference, 3318 refs, 0.0055650 secs]2017-09-25T20:39:17.336+0000: 2.816: [PhantomReference, 1 refs, 0.0001440 secs]2017-09-25T20:39:17.336+0000: 2.816: [JNI Weak Reference, 0.0000230 secs] [PSYoungGen: 132096K->11180K(153600K)] 132096K->11188K(503296K), 0.0236970 secs] [Times: user=0.04 sys=0.02, real=0.02 secs]
2017-09-25T20:39:18.798+0000: 4.278: [GC2017-09-25T20:39:18.815+0000: 4.295: [SoftReference, 0 refs, 0.0004490 secs]2017-09-25T20:39:18.815+0000: 4.295: [WeakReference, 1055 refs, 0.0003090 secs]2017-09-25T20:39:18.816+0000: 4.296: [FinalReference, 4585 refs, 0.0044610 secs]2017-09-25T20:39:18.820+0000: 4.300: [PhantomReference, 0 refs, 0.0001230 secs]2017-09-25T20:39:18.820+0000: 4.300: [JNI Weak Reference, 0.0000170 secs] [PSYoungGen: 143276K->12052K(153600K)] 143284K->12132K(503296K), 0.0246030 secs] [Times: user=0.05 sys=0.02, real=0.03 secs]
2017-09-25T20:39:20.235+0000: 5.715: [GC2017-09-25T20:39:20.259+0000: 5.739: [SoftReference, 0 refs, 0.0002660 secs]2017-09-25T20:39:20.259+0000: 5.739: [WeakReference, 1058 refs, 0.0002230 secs]2017-09-25T20:39:20.259+0000: 5.739: [FinalReference, 3120 refs, 0.0037950 secs]2017-09-25T20:39:20.263+0000: 5.743: [PhantomReference, 0 refs, 0.0002100 secs]2017-09-25T20:39:20.263+0000: 5.743: [JNI Weak Reference, 0.0000430 secs] [PSYoungGen: 144148K->17825K(153600K)] 144228K->17913K(503296K), 0.0303560 secs] [Times: user=0.07 sys=0.01, real=0.03 secs]
2017-09-25T20:39:21.371+0000: 6.851: [GC2017-09-25T20:39:21.426+0000: 6.906: [SoftReference, 0 refs, 0.0006670 secs]2017-09-25T20:39:21.427+0000: 6.907: [WeakReference, 1061 refs, 0.0004830 secs]2017-09-25T20:39:21.427+0000: 6.907: [FinalReference, 3052 refs, 0.0066890 secs]2017-09-25T20:39:21.434+0000: 6.914: [PhantomReference, 0 refs, 0.0001320 secs]2017-09-25T20:39:21.434+0000: 6.914: [JNI Weak Reference, 0.0000100 secs] [PSYoungGen: 149921K->21495K(285696K)] 150009K->30004K(635392K), 0.0658520 secs] [Times: user=0.16 sys=0.02, real=0.07 secs]
2017-09-25T20:39:23.395+0000: 8.875: [GC2017-09-25T20:39:23.498+0000: 8.978: [SoftReference, 0 refs, 0.0004150 secs]2017-09-25T20:39:23.499+0000: 8.979: [WeakReference, 0 refs, 0.0001070 secs]2017-09-25T20:39:23.499+0000: 8.979: [FinalReference, 0 refs, 0.0001230 secs]2017-09-25T20:39:23.499+0000: 8.979: [PhantomReference, 0 refs, 0.0001010 secs]2017-09-25T20:39:23.499+0000: 8.979: [JNI Weak Reference, 0.0000170 secs] [PSYoungGen: 285687K->21504K(285696K)] 294196K->53238K(635392K), 0.1084870 secs] [Times: user=0.21 sys=0.10, real=0.11 secs]
Heap
PSYoungGen total 285696K, used 171249K [0x00000007b8800000, 0x00000007de700000, 0x0000000800000000)
eden space 264192K, 56% used [0x00000007b8800000,0x00000007c1a3c790,0x00000007c8a00000)
from space 21504K, 100% used [0x00000007c8a00000,0x00000007c9f00000,0x00000007c9f00000)
to space 46592K, 0% used [0x00000007db980000,0x00000007db980000,0x00000007de700000)
ParOldGen total 349696K, used 31734K [0x0000000729800000, 0x000000073ed80000, 0x00000007b8800000)
object space 349696K, 9% used [0x0000000729800000,0x000000072b6fdad8,0x000000073ed80000)
PSPermGen total 58368K, used 58223K [0x0000000719800000, 0x000000071d100000, 0x0000000729800000)
object space 58368K, 99% used [0x0000000719800000,0x000000071d0dbc08,0x000000071d100000)
error log from inv-svc.log indicated LDAP error.
; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.vmware.vim.query.server.accesscontrol.impl.Lo tusInitializer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials
LDAP error [code: 49]
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:106)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1317)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1076)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322)
... 65 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'memCache' defined in class path resource [server/config/secu rity-config.xml]: Cannot resolve reference to bean 'globalAclLotusCache' while setting bean property 'globalAclLotusCache'; nested exception is org.springframe work.beans.factory.BeanCreationException: Error creating bean with name 'aclLotusInitializer' defined in class path resource [server/config/authorization-confi g.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.vmware.vi m.query.server.accesscontrol.impl.LotusInitializer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com.vmware.identity.interop.l dap.InvalidCredentialsLdapException: Invalid credentials
LDAP error [code: 49]
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:106)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1317)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1076)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322)
... 75 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'aclLotusInitializer' defined in class path resource [server/ config/authorization-config.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.vmware.vim.query.server.accesscontrol.impl.LotusInitializer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com .vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials
LDAP error [code: 49]
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:286)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:993)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:897)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:281)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322)
... 85 more
Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.vmware.vim.query.server.accesscontrol.impl.LotusInitiali zer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid cr edentials
LDAP error [code: 49]
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:141)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:105)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:278)
... 96 more
Caused by: java.lang.RuntimeException: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials
LDAP error [code: 49]
at com.vmware.cis.lotus.LdapConnectionFactory.getConnection(LdapConnectionFactory.java:142)
at com.vmware.vim.query.server.accesscontrol.impl.LotusInitializer.<init>(LotusInitializer.java:26)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:126)
... 98 more
I tried resetting the the password using /usr/lib/vmware-vmdir/bin/vdcadmintool
and that did not fix it.
any help or advise appreciated.