How to remove an orphaned Nexus 1000v DVS in vCenter

What if you Accidentally deleted the VSM, and now are left with the DVS still showing in vCenter

Nexus 1000v & Any version of vCenter

Conditions / Environment

The DVS must be gracefully removed from the VSM before deleting it. w.

Solution:

Deploy a temporary VSM
Restore the startup config (or at least restore the previous switchname)
You can use the command “vmware vc extension-key <extension-id>” to have a new CP connect to the existing DVS.
The extension id here in this command should be the same one that is tied down to the DVS. You can find the key using the one of two ways:
In vCenter navigate to the Networking View
Select the DVS in the Left navigation pane.
Click on the Summary tab on the right.
The Extension key is listed under Annotations

or

Go to the VC’s mob by pointing your browser to it [ https://<VC_IP_ADDR>/mob%5D
Go to rootFolder ‘group-d1’
Find your datacenter from ‘childEntity’ containing Datacenter-IDs [When you click on a datacenter, you will find a name associated with it]
From your datacenter, go to networkFolder [Eg: group-n6]
From the network folder select the child entity [Eg:group-n373]
In the ‘childEntity’ click on your dvs [eg: dvs-7]
Under the DVS “config” attribute, you can find the extension key in use
Assign the extension key id to the VSM using “vmware vc extension-key <extension-id>”.
Once you key in this extension key, verify the new extension key on the CP using “show vmware vc extension-key”.
Save and reboot the VSM.
Delete the extension key present on the VC using MOB. (Unregister extension API):

Goto the extension manager [https://<VC_IP_ADDR>/mob/?moid=ExtensionManager]

Click on Unregister extension [https://<VC_IP_ADDR>/mob/?moid=ExtensionManager&method=unregisterExtension]

Paste “Cisco_Nexus_1000V_<DVS TO RECOVER’s KEY>” (your extension key attached to the DVS) and click on “Invoke Method”

8. Now you are ready to re-register the extension. If you are getting the .xml file through the browser, make sure you refresh the browser before downloading the XML file
9. Re-register the extension plugin
10. Setup the SVS connection properties (VC IP, Port, Datacenter name etc)
11. ‘Connect’ for your svs connection
12. Last but not least, gracefully remove the DVS using “no vmware dvs” from the SVS context on the VSM.

Once you verify the DVS removes from vCenter, you can delete the temporary VSM safely.

Happy Learning 😉

Advertisements

Re-register a vCloud VM in vCenter

Consider during troubleshooting a VM issue , you would have to unregister the VM which is being managed by vCloud.

Let’s call the VM “chosenone(0e68e734-ce66-3567-15ca98dfc228)”

If we search the vCloud database, we can see the moref of this VM is vm-xxxx.  You will find this moref under the following tables:

computevm
licensing_vm_data
networked_vm
property_map
vapp_vm
vc_task_mapping
vm
vm_inv

select id, name,vc_id ,moref, cloud_uuid,instance_uuid from vm where name like ‘%chosenone%’
select vm_inv_id, name,moref ,vc_id, cloud_uuid,instance_uuid from vm_inv where name like ‘%chosenone%’

In the VM’s vmx file, we find the following UUIDs

uuid.bios = “42 2a f4 r5 a8 77 01 bd-52 17 99 78 64 8f f3 9d”
vc.uuid = “50 2a 53 a7 38 91 b1 06-a2 35 34 76 89 c7 a4 09″
cloud.uuid = “a8108db8-9845-43a7-bd4c-98ca588afa0d”

Un-registering  the VM and re-registering the VM, the VM will keep the vc.uuid (instance_uuid in the database) and the cloud.uuid.  However, it will get a new moref.

Moref ?  What is it ?

The MoRef ID is a VMware internal identifier that is generated by vSphere when new objects are created in vCenter (e.g. adding ESXi host or creating a VM). This MoRef ID is used by all VMware solutions to reference objects within vCenter

With vCloud Director, to uniquely identify a virtual machine, you would use the vCloud Director Object ID that is automatically generated by vCloud Director when a new virtual machine is created.

If you need to uniquely identify a virtual machine in both vSphere and vCloud, you can still use the identifiers listed above in their respective stack but how do you go about correlating the two? As mentioned earlier, all VMware solutions leverage the MoRef ID to reference objects within vCenter. Since vCloud Director sits on top of vSphere, it too also uses the MoRef ID to map entities between the two environments. As part of the vCloud API, there is a vCloud Extension API that provides information about the underlying vSphere infrastructure including the MoRef ID of a VM and vCenter Server instanceUuid.

Coming back , After you re-register the VM, you may see two entries in the vCloud databases’s vm table.  Once the vCloud VC inventory services runs or you re-connect vCenter through vCloud, the old entry (moref vm-xxxx) will disappear.

If you try to start the VM in vCloud, you will receive the following error
Error: Could not find object with moref “vm-xxxx″ and VC ID in inventory category “VirtualMachine”.If you search the vCloud database for this moref, you will find it under the following tables.

select * from computevm where vmmoref = ‘vm-xxxx
select * from networked_vm  where moref = ‘vm-xxxx’

We need to fix this by replacing the old moref in these tables with the new moref.

The following query helps us to find the moref id

select moref from vm where name = chosenone (0e68e734-ce66-3567-15ca98dfc228)’ > vm-xxxx

Update the tables with the new moref

update computevm set vmmoref = ‘vm-xxxx where vmmoref = ‘vm-xxxx’
update networked_vm set moref = ‘vm-xxxx where moref = ‘vm-xxxx’

You should be able to power on the vm now .

We would may be need  to refresh vCenter through vCloud so that vCloud will mark the VMs that is it manages.

The vm should show as being managed by vcloud solution now and should be mapped fine.

Happy Learning 🙂

vCenter server and Inventory service slow performance / Migration failing with pbmfault error

Hi All ,

I came across this issue today, vCenter server started to perform very slowly and the Inventory service was taking a very long time to start.

There was no specific issues , and it all started so suddenly to my surpise.

It’s a vCenter 5.5 , were vCenter and IS are installed on the same machine, with vCenter connecting to external DB.

After further digging on logs , got below messages

ds.log

RROR com.vmware.vim.query.server.provider.impl.ProviderManagerServiceImpl] Store eror processing atom feed:
com.vmware.vim.query.server.store.exception.StoreException: com.xhive.error.XhiveException: SERVER_TERMINATED: The server has terminated, Original message: Not enough storage is available to process this command
at com.vmware.vim.query.server.store.impl.SessionPoolImpl.getSession(SessionPoolImpl.java:165)
at com.vmware.vim.query.server.store.impl.StoreImpl.updateVmomiPullAtomFeed(StoreImpl.java:2105)

Caused by: com.xhive.error.XhiveException: SERVER_TERMINATED: The server has terminated, Original message: Not enough storage is available to process this command
at xDB_9_0_11.uB.zd(xdb:366)

vpxd.log

2015-04-14T21:43:36.221-04:00 [10176 error ‘VmProv’ opID=task-internal-1-69b2e9b1-c-1d-3-af] [CallbackManager] Got exception while invoking precheck on PbmCallBack: Invalid response code: 503 Service Unavailable
–> backtrace[00] rip 000000018018b6aa
–> backtrace[01] rip 0000000180105288
–> backtrace[02] rip 000000018010668e
–> backtrace[03] rip 00000001800900f8
–> backtrace[04] rip 00000000005910d5

Also during the same time the vmotions were failing with pbmfault errors

There were no issues with drive space or DB at all.

Checked the Heap memory (wrapper.conf) for the services and it was looking fine as well.

Thus I tried the below steps , as it was failing with pbmfault and I though it must be Profile driven storage space.

Did the below steps :

1. Stop Profile driven storage service.
2. Goto C:\Program Files\VMware\Infrastructure\Profile-Driven Storage\conf\log4j.properties
change the values of MaxFileSize and MaxBackupIndex as below.

log4j.appender.file.MaxFileSize=10MB
log4j.appender.file.MaxBackupIndex=50

3. Delete all the logs for PRofile Driven Storage, or take a backup (we took a backup on destop)
4. Restart the Profile-Driven Storage service.
5. Restart the Inventory service

Wow , everything started looking normal and it started working fine.

It seems like somehow the log file disk only for Profile driven storage was getting full.

Happy Troubleshooting 🙂

vSphere 6.0 Platform Services Controller – How to decommission a PSC and vCenter

Stop the Platform Services Controller that you no longer need.
Manually repoint all vCenter Servers and all vCenter Server instances registered with the Platform Services Controller to another Platform Services Controller within the domain.
Platform Services Controller appliance:

Log in as root to the appliance shell of one of the Platform Services Controller
To enable the Bash shell, run the shell.set –enabled true command.
Run the shell command to start the Bash shell and log in.
Run the cmsso-util unregister command to unregister the stopped Platform Services Controller
cmsso-util unregister –node-pnid Platform-Services-Controller-System-Name –username administrator@your_domain_name –passwd vCenter-Single-Sign-On-password

Platform Services Controller Windows

On the Platform Service Controller, click Start > Run, type cmd, and click OK. The Command Prompt window opens.
Navigate to C:\Program Files\VMware\vCenter Server\bin\
Run the cmsso-util unregister command to unregister the stopped Platform Services Controller:
cmsso-util unregister –node-pnid Platform-Services-Controller-System-Name –username administrator@your_domain_name –passwd vCenter-Single-Sign-On-password
You must run this command only on one of the Platform Services Controller replication partners, as the synchronization removes the entries from all other Platform Services Controller replication partners.
To decommission a vCenter Server

vCenter Server Appliance:

Power off the vCenter Server Appliance that you no longer need.
Log in as root to the appliance shell of the Platform Services Controller appliance with which the vCenter Server Appliance is registered.
To enable the Bash shell, run the shell.set –enabled true command.
Run the shell command to start the Bash shell and log in.
Run the cmsso-util unregister command to unregister the powered off vCenter Server Appliance.
cmsso-util unregister –node-pnid vCenter-Server-Appliance-System-Name –username administrator@your_domain_name –passwd vCenter-Single-Sign-On-password
On the Platform Service Controller, click Start > Run, type cmd, and click OK. The Command Prompt window opens.
Navigate to C:\Program Files\VMware\vCenter Server\bin\
Run the cmsso-util unregister command to unregister the powered off vCenter Server Appliance.
cmsso-util unregister –node-pnid vCenter-Server-Appliance-System-Name –username administrator@your_domain_name –passwd vCenter-Single-Sign-On-password

Here, vCenter-Server-Appliance-System-Name is the FQDN or IP address of the vCenter Server Appliance that you want to decommission. You must run this command only on the Platform Services Controller appliance with which your vCenter Server Appliance is registered.