Hi All,
I have two VDP 6 instances. One week ago a saw that CPU usage on one of them is almost always 100%. The backup works, Second instance have about 20% CPU utilization.
The diffrence I find: (second instance does not have this kind of errors)
root@clvdp01:/space/avamar/var/mc/server_log/#: tail -f axis2.log
2015-05-18 10:04:04,273 [218184632@qtp-1305303392-4784] ERROR org.apache.axis2.transport.http.AxisServlet - Java heap space
java.lang.OutOfMemoryError: Java heap space
2015-05-18 10:04:37,105 [1384024202@qtp-1305303392-4789] ERROR org.apache.axis2.transport.http.AxisServlet - Java heap space
java.lang.OutOfMemoryError: Java heap space
2015-05-18 10:05:13,130 [1795787615@qtp-1305303392-4779] ERROR org.apache.axis2.transport.http.AxisServlet - Java heap space
java.lang.OutOfMemoryError: Java heap space
2015-05-18 10:05:49,640 [1593678748@qtp-1305303392-4821] ERROR org.apache.axis2.transport.http.AxisServlet - Java heap space
java.lang.OutOfMemoryError: Java heap space
2015-05-18 10:09:11,437 [1107337480@qtp-1305303392-4777] ERROR org.apache.axis2.transport.http.AxisServlet - Java heap space
java.lang.OutOfMemoryError: Java heap space
Below more information:
status.dpn
Mon May 18 10:05:20 CEST 2015 [clvdp01] Mon May 18 08:05:20 2015 UTC (Initialized Thu Oct 2 10:31:18 2014 UTC)
Node IP Address Version State Runlevel Srvr+Root+User Dis Suspend Load UsedMB Errlen %Full Percent Full and Stripe Status by Disk
0.0 10.190.4.61 7.1.81-107 ONLINE fullaccess mhpu+0hpu+0hpu 1 false 6.05 7460 6866739 43.0% 43%(onl:1792) 43%(onl:1795) 43%(onl:1792)
Srvr+Root+User Modes = migrate + hfswriteable + persistwriteable + useraccntwriteable
System ID: 1412245878@00:50:56:A2:0D:56
All reported states=(ONLINE), runlevels=(fullaccess), modes=(mhpu+0hpu+0hpu)
System-Status: ok
Access-Status: full
Last checkpoint: cp.20150517093236 finished Sun May 17 11:33:38 2015 after 01m 01s (OK)
Last GC: finished Mon May 18 09:34:53 2015 after 01h 27m >> recovered 11.24 GB (OK)
Last hfscheck: finished Sun May 17 11:30:06 2015 after 01h 18m >> checked 2832 of 2832 stripes (OK)
Maintenance windows scheduler capacity profile is active.
The maintenance window is currently running.
Currently running task(s): crunchwait
Next backup window start time: Tue May 19 00:00:00 2015 CEST
Next maintenance window start time: Tue May 19 08:00:00 2015 CEST
dpnctl status
Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)
dpnctl: INFO: gsan status: up
dpnctl: INFO: MCS status: up.
dpnctl: INFO: Backup scheduler status: up.
dpnctl: INFO: axionfs status: down.
dpnctl: INFO: Maintenance windows scheduler status: enabled.
dpnctl: INFO: Unattended startup status: enabled.
dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]
top
top - 10:11:05 up 3 days, 11:21, 1 user, load average: 6.19, 5.49, 5.22
Tasks: 232 total, 1 running, 230 sleeping, 0 stopped, 1 zombie
Cpu(s): 52.5%us, 0.7%sy, 0.9%ni, 42.4%id, 3.6%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 7926796k total, 7515232k used, 411564k free, 9240k buffers
Swap: 4200988k total, 0k used, 4200988k free, 2833152k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8547 admin 20 0 2030m 1.3g 5580 S 596 16.9 19371:50 java
root@clvdp01:/space/avamar/var/mc/server_log/#: ps aux | grep 8547
admin 8547 387 16.8 2077128 1335904 ? Sl May14 19374:25 /usr/java/latest/bin/java -Xmx1G -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -server -ea -cp /usr/local/avamar/lib/avamar-mcsecurity.jar:/usr/local/avamar/lib/commons-cli-1.2.jar:/usr/local/avamar/lib/commons-lang-2.6.jar:/usr/local/avamar/lib/commons-logging-adapters-1.1.jar:/usr/local/avamar/lib/rsa-jsafe-cryptojFIPS-5.0.jar:/usr/local/avamar/lib/rsa-jsafe-util-5.0.jar:/usr/local/avamar/lib/spring-jdbc-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-tx-3.1.1.RELEASE.jar:/usr/local/avamar/lib/mcserver.jar:/usr/local/avamar/var/mc/server_data/prefs:lib:/usr/local/avamar/lib/asn_server.jar:/usr/local/avamar/lib/mail.jar:/usr/local/avamar/lib/activation.jar:/usr/local/avamar/lib/xercesImpl.jar:/usr/local/avamar/lib/xmlsec.jar:/usr/local/avamar/lib/xmldsig.jar:lib/postgresql-8.4-702.jdbc4.jar:/usr/local/avamar/lib/jaxrpc.jar:/usr/local/avamar/lib/commons-logging-1.0.4.jar:/usr/local/avamar/lib/commons-validator.jar:/usr/local/avamar/lib/commons-discovery-0.2.jar:/usr/local/avamar/lib/commons-collections-3.2.1.jar:/usr/local/avamar/lib/saaj.jar:/usr/local/avamar/lib/wbem.jar:/usr/local/avamar/lib/wsdl4j-1.5.1.jar:/usr/local/avamar/lib/log4j-1.2.14.jar:/usr/local/avamar/lib/axis.jar:/usr/local/avamar/lib/vim25.jar:/usr/local/avamar/lib/commons-codec-1.3.jar:/usr/local/avamar/lib/jema_api.jar:/usr/local/avamar/lib/jetty-6.1.23.jar:/usr/local/avamar/lib/jetty-util-6.1.23.jar:/usr/local/avamar/lib/servlet-api-3.0.jar:/usr/local/avamar/lib/jsp-api-2.1.jar:/usr/local/avamar/lib/jsp-2.1.jar:/usr/local/avamar/lib/core-3.1.1.jar:/usr/local/avamar/lib/ant-1.6.5.jar:/usr/local/avamar/lib/apputils25.jar:/usr/local/avamar/lib/j2ssh-core-0.2.9.jar:/usr/local/avamar/lib/snmp4j-1.11.jar:/usr/local/avamar/lib/mccommons.jar:/usr/local/avamar/lib/user-authentication.jar:/usr/local/avamar/lib/nis-1.2.1.jar:/usr/local/avamar/lib/providerutil-1.2.1.jar:/usr/local/avamar/lib/commons-lang-2.6.jar:/usr/local/avamar/lib/commons-io-1.4.jar:/usr/local/avamar/lib/jutf7-1.0.0.jar:/usr/local/avamar/lib/i18n:/usr/local/avamar/lib/certj.jar:/usr/local/avamar/lib/cryptojFIPS.jar:/usr/local/avamar/lib/sslj.jar:/usr/local/avamar/lib/amqp-client-3.1.3.jar:/usr/local/avamar/lib/events-dispatcher-api-1.0.0.jar:/usr/local/avamar/lib/events-dispatcher-amqp-1.0.0.jar:/usr/local/avamar/lib/jackson-core-asl-1.8.3.jar:/usr/local/avamar/lib/jackson-mapper-asl-1.8.3.jar:/usr/local/avamar/lib/aopalliance-1.0.jar:/usr/local/avamar/lib/aspectjweaver-1.6.12.jar:/usr/local/avamar/lib/spring-aop-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-asm-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-beans-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-context-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-core-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-expression-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-amqp-1.2.0.RELEASE.jar:/usr/local/avamar/lib/spring-rabbit-1.2.0.RELEASE.jar:/usr/local/avamar/lib/spring-web-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-jdbc-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-tx-3.1.1.RELEASE.jar::/usr/local/avamar/lib/spring-security-config-3.1.0.RELEASE.jar:/usr/local/avamar/lib/spring-security-core-3.1.0.RELEASE.jar:/usr/local/avamar/lib/spring-security-crypto-3.1.0.RELEASE.jar:/usr/local/avamar/lib/spring-security-web-3.1.0.RELEASE.jar -Djava.rmi.server.hostname=clvdp01 -Djava.util.logging.config.file=/usr/local/avamar/lib/mcserver_logging.properties -Djava.security.egd=file:/dev/./urandom -Djava.io.tmpdir=/usr/local/avamar/var/mc/server_tmp -Djava.util.prefs.PreferencesFactory=com.avamar.mc.util.MCServerPreferencesFactory -Djavax.xml.parsers.DocumentBuilderFactory=org.apache.xerces.jaxp.DocumentBuilderFactoryImpl -Djavax.net.ssl.keyStore=lib/rmi_ssl_keystore -Djavax.net.ssl.trustStore=lib/rmi_ssl_keystore -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/local/avamar/lib/log4j.properties com.avamar.mc.CLI --start
Is Anyone is able to help?
Best Regards,
Jacek