XenServer, it’s a nice product, but for some reason I AL-WAYS get the creeps when I need to update the server software with a new service pack or hot-fix. Checking the Citrix forums, there are tons of threads about people loosing their “Management interface” after an upgrade. I’ve read almost all of the posts on the forums related to this issue and seriously, none of them worked, so I started to play around with the available options in the “xe” toolkit. My goal was to get the server back in the pool, without re-installing the server and I succeeded. (Yay!)
My setup is a single pool with 4x Advanced Enterprise Edition servers (this will work with all license types). So this morning I updated the pool master, all 4 hot-fixes (XS60E001 to XS60E004), flawless! Next server! After 15mins the machine wasn’t back up yet. So I logged into the console (via ssh), “No Interfaces configured” *sigh*. And the hunt began looking for a solution. Nothing found, only option was to reinstall, no deal!
Before continuing: Your local storage will be GONE after this, you can create new one afterwards, so before continuing, backup your data!!!!
Back to the “dead” server. I removed it from the pool by logging into the current pool master and typed:
xe host-list
I found my “dead” server:
uuid ( RO) : 328dbe1c-4b63-4843-b8b3-40ad30acf3db name-label ( RW): rischa-xen02 name-description ( RW): Default install of XenServer
Then I made the pool master forget the “dead” server with:
xe host-forget uuid=328dbe1c-4b63-4843-b8b3-40ad30acf3db
This might take some time, for me it took so long I pressed CTRL-C to stop, but when I pressed it, I noticed the server disappearing in XenCenter. I only had the broken Storage Repositories from the “dead” server left in the list, NICE! Let’s clean those up..
I removed the SR’s by finding the “dead” server SR’s first:
xe sr-list
Gave me:
..snip.. uuid ( RO) : 28006370-23ab-9975-e33c-7f06a1b88516 name-label ( RW): Removable storage name-description ( RW): host ( RO): <not in database> type ( RO): udev content-type ( RO): disk uuid ( RO) : f6bf51a8-26b7-4c82-b4f2-8be70ca7263b name-label ( RW): Local storage name-description ( RW): host ( RO): <not in database> type ( RO): lvm content-type ( RO): user uuid ( RO) : b4adcc4f-86c3-8cba-2e76-44ffe4f24805 name-label ( RW): DVD drives name-description ( RW): Physical DVD drives host ( RO): <not in database> type ( RO): udev content-type ( RO): iso ..snip..
(It’s always the SR’s with host “<not in database>”, so remove those with:
xe sr-forget uuid=28006370-23ab-9975-e33c-7f06a1b88516
xe sr-forget uuid=f6bf51a8-26b7-4c82-b4f2-8be70ca7263b
xe sr-forget uuid=b4adcc4f-86c3-8cba-2e76-44ffe4f24805
Now, our pool doesn’t have a clue about the “dead” server, it’s removed from it, part 1 done. Now we still have our “dead” server that THINKS he’s part of the pool, but shouldn’t. So go to the console of your “dead” server and follow the next few steps.
Let’s reset the state.db first with:
mv /var/xapi/state.db /var/xapi/state.db_old
Now tell the server to transition into a master with:
xe pool-emergency-transition-to-master
Let it scan it’s local pif’s:
xe pif-scan host-uuid=328dbe1c-4b63-4843-b8b3-40ad30acf3db
And check the pif list:
xe pif-list
It should give you something like this:
uuid ( RO) : 3cdafc3d-438b-0c42-e220-767f373cc0e6 device ( RO): eth1 currently-attached ( RO): true VLAN ( RO): -1 network-uuid ( RO): e7797467-a284-3d2c-e1b4-bb5697d2511c uuid ( RO) : 6e025c8f-109c-4603-80db-0cdcc28e5219 device ( RO): eth0 currently-attached ( RO): false VLAN ( RO): -1 network-uuid ( RO): a662c677-2d6b-808a-666f-b5fb0a5b436b
Now run “xsconsole” and you should be able to configure a network interface as management interface.
Return to your XenCenter and add the server to it.
Also good practice is to remove your server from the pool before applying any update or hot-fix!
I hope this solved your problem as it did for me. Leave a comment if you’re having problems.
Note: After adding the server back to the pool and you make use of iSCSI, don’t forget to set a new IQN for the server.
you saved my day!
thanks for sharing ….
Thanks alot for this! Worked like a charm!
You just saved me at least 4 hours on a Friday night! Awesome.
Thanx mate! works like a charm, but its fuckin up the disks/optical drives for the host, but still, much faster than a reinstall :)
In my case (after update 6.2 to 6.2 SP1) I get “This operation cannot be completed as the host is still live. host: ()” if I run the command xe host-forget uuid= on the pool master.
Any ideas?
edit: After shutting down of the “bad” server I could run the command xe host-forget uuid=… on the pool master successfully. Thanks a lot
Nice. It worked for me.
Thank you.
I can’t rescue SR after execute “xe sr-forget uuid=”. I can put the server in a Pool and I can see you normally. But its local storage not appear on XenCenter.
I did executed any commands, eg. lvstatus. With this command I can saw that LV is NOT ENABLE. Then I did executed vgchange -ay, it stay enable. But not appear in XenCenter. I did reboot and one more time the LV is NOT ENABLE.
What can I do?
Tks
Amazing !!! saved me !!!
Thanks !!
Thanks man!! you saved my day! :D