Hi All ,
I came across this issue today, vCenter server started to perform very slowly and the Inventory service was taking a very long time to start.
There was no specific issues , and it all started so suddenly to my surpise.
It’s a vCenter 5.5 , were vCenter and IS are installed on the same machine, with vCenter connecting to external DB.
After further digging on logs , got below messages
RROR com.vmware.vim.query.server.provider.impl.ProviderManagerServiceImpl] Store eror processing atom feed:
com.vmware.vim.query.server.store.exception.StoreException: com.xhive.error.XhiveException: SERVER_TERMINATED: The server has terminated, Original message: Not enough storage is available to process this command
Caused by: com.xhive.error.XhiveException: SERVER_TERMINATED: The server has terminated, Original message: Not enough storage is available to process this command
2015-04-14T21:43:36.221-04:00 [10176 error ‘VmProv’ opID=task-internal-1-69b2e9b1-c-1d-3-af] [CallbackManager] Got exception while invoking precheck on PbmCallBack: Invalid response code: 503 Service Unavailable
–> backtrace rip 000000018018b6aa
–> backtrace rip 0000000180105288
–> backtrace rip 000000018010668e
–> backtrace rip 00000001800900f8
–> backtrace rip 00000000005910d5
Also during the same time the vmotions were failing with pbmfault errors
There were no issues with drive space or DB at all.
Checked the Heap memory (wrapper.conf) for the services and it was looking fine as well.
Thus I tried the below steps , as it was failing with pbmfault and I though it must be Profile driven storage space.
Did the below steps :
1. Stop Profile driven storage service.
2. Goto C:\Program Files\VMware\Infrastructure\Profile-Driven Storage\conf\log4j.properties
change the values of MaxFileSize and MaxBackupIndex as below.
3. Delete all the logs for PRofile Driven Storage, or take a backup (we took a backup on destop)
4. Restart the Profile-Driven Storage service.
5. Restart the Inventory service
Wow , everything started looking normal and it started working fine.
It seems like somehow the log file disk only for Profile driven storage was getting full.
Happy Troubleshooting 🙂