VMware

vCenter server and Inventory service slow performance / Migration failing with pbmfault error

Hi All ,

I came across this issue today, vCenter server started to perform very slowly and the Inventory service was taking a very long time to start.

There was no specific issues , and it all started so suddenly to my surpise.

It’s a vCenter 5.5 , were vCenter and IS are installed on the same machine, with vCenter connecting to external DB.

After further digging on logs , got below messages

ds.log

RROR com.vmware.vim.query.server.provider.impl.ProviderManagerServiceImpl] Store eror processing atom feed:
com.vmware.vim.query.server.store.exception.StoreException: com.xhive.error.XhiveException: SERVER_TERMINATED: The server has terminated, Original message: Not enough storage is available to process this command
at com.vmware.vim.query.server.store.impl.SessionPoolImpl.getSession(SessionPoolImpl.java:165)
at com.vmware.vim.query.server.store.impl.StoreImpl.updateVmomiPullAtomFeed(StoreImpl.java:2105)

Caused by: com.xhive.error.XhiveException: SERVER_TERMINATED: The server has terminated, Original message: Not enough storage is available to process this command
at xDB_9_0_11.uB.zd(xdb:366)

vpxd.log

2015-04-14T21:43:36.221-04:00 [10176 error ‘VmProv’ opID=task-internal-1-69b2e9b1-c-1d-3-af] [CallbackManager] Got exception while invoking precheck on PbmCallBack: Invalid response code: 503 Service Unavailable
–> backtrace[00] rip 000000018018b6aa
–> backtrace[01] rip 0000000180105288
–> backtrace[02] rip 000000018010668e
–> backtrace[03] rip 00000001800900f8
–> backtrace[04] rip 00000000005910d5

Also during the same time the vmotions were failing with pbmfault errors

There were no issues with drive space or DB at all.

Checked the Heap memory (wrapper.conf) for the services and it was looking fine as well.

Thus I tried the below steps , as it was failing with pbmfault and I though it must be Profile driven storage space.

Did the below steps :

1. Stop Profile driven storage service.
2. Goto C:\Program Files\VMware\Infrastructure\Profile-Driven Storage\conf\log4j.properties
change the values of MaxFileSize and MaxBackupIndex as below.

log4j.appender.file.MaxFileSize=10MB
log4j.appender.file.MaxBackupIndex=50

3. Delete all the logs for PRofile Driven Storage, or take a backup (we took a backup on destop)
4. Restart the Profile-Driven Storage service.
5. Restart the Inventory service

Wow , everything started looking normal and it started working fine.

It seems like somehow the log file disk only for Profile driven storage was getting full.

Happy Troubleshooting 🙂

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s