Horrors of Converting an Entire Network From Physical to Virtual

Part One – The Trial Run (or Tests Always Go So Well)

This blog will be in several parts as I go through the process of, as the title suggests, converting an entire network from physical servers to virtual servers.  First and foremost, I have to get it off my chest, but the term “cloud” has got to be one of the most annoying overused and widely defined (or undefined as pretty much everything internet related is being called part of the cloud) words of the year.

Strangely, people hate the word virtualization.   They hate the fact that their server may, gasp, be a virtual server as opposed to a “real” server.  Nevermind that there are many factors that make virtualization a better positive than that of a physical device.  Things like better redundancy should there be a failure, lower administrative costs, in many cases better technology, quicker and easier upgrades in terms of memory… the list goes on.

Enter “The Cloud.”  With the Cloud, it must not be virtualization.  In fact, it must be really good since everyone uses the Cloud.  There are even commercials on TV that make the Cloud look really useful!  Nevermind that somebody or group of sombodies decided that they would esentially take the entire internet, anything that doesn’t quite fit in the defines of current consumer based understanding of technology, or anything that has been shunned due to whatever misconceptions or simple dislikes, and name it after a Microsoft Visio shape.

Throughout these articles I will be using both virtualization and cloud interchangeably.

My cloud rant aside, I have been tasked with converting approximately 700 physical servers into virtualized servers.  Easy, right?

All I really need to do is decide on a platform, pay some cash for the P2V software, get some hardware that I can use to condense the new servers into, and hit go.

Not so fast.

The platform was easy enough.  We have been using Hyper-V on Microsoft Server 2008 and 2008R2 for the last year for the hundred plus servers that we already were running as Cloud servers.  All I had to do was grab Microsoft’s System Center to centralize administration since up to this point we managed them on a server by server basis.

System Center also offered the feature of P2V (physical to virtual) so that base was covered as well.  Awesome! Hardware? Well I sat down and weighed the costs of different types of devices, setups, and configurations.  Long story short, We decided to go with a dual SAN solution with a number of customized Dell c6100 servers BUT, and I underscore BUT, I needed to make sure that this solution would work before finalizing the purchase.

For testing I set up a Dell PowerEdge r410 with similar specs to one of the blades of the c6100 as the host machine, configured a Dell PowerEdge 2950 with an iSCSI connection to represent the SAN, and a number of servers running various versions of Windows Server 2003, 2008, and 2008r2 with a wide range of hardware configurations.  All these I placed inside the office’s internal domain and began my adventure.

So clicking convert a physical to virtual would be the next logical step correct? (Besides setting up the host and storage servers in System Center which, at least for now, I won’t go into as they went smoothly – at least in the test environment.)

Wrong.  Due to some funkiness that happened over a year ago on our live network and some modifications to the production domain policies that essentially removed the admin$ share (and set it up so that every time the server rebooted it would remove it), it took some time to go through and correct the basic errors preventing communication between System Center and the test conversion servers.

After a few modifications in the registry, the admin$ is now back and useable and not going away anytime soon. So I launch the conversion process.  Being that I want to make sure that the conversion is completely up to date, downtime is minimal, and get a better chance of success for completion, I decide to go ahead and do the conversion in offline mode.

Easy enough to choose and click on, however not so easy when it came to needing network drivers for the initial test subject.  In this case it was a Dell PowerEdge r200 with Broadcom Nic adapters.

To backup a little bit, for those who do not know how the offline conversion process works, it basically utilizes a custom-built WinPE image for the specific server created by System Server which includes the currently assigned IP address of the server which makes it nice so I don’t have to keep track of additional IP address usage during the process and prevent conflicts, etc.  It also allows you to set the physical to be completely shut down once the transfer is complete and automatically launch the newly created virtual immediately upon completion.  No need to go back and power stuff off and then launch the virtual manually – less touch time = great in my book.

Back to the story at hand, the drivers that were available for those Broadcom adapters apparently were not good enough for System Center to use and it kept refusing to build the WinPE environment.  A visit to Dell’s website didn’t provide any resolution either as the 50,000 versions available there were also rejected (I know, I’m exaggerating a little.)

Fast forward 2 hours later and hundreds of sites visited, versions of drivers downloaded and tried, a bit of cursing, and at least one pot of coffee to help keep my sanity and I finally found one that actually worked.  It doesn’t matter to me that the driver found wasn’t really for the adapters I have on hand – this is a temporary WinPE build so once done always done.  It’s amazing what you can find in the remote corners of the internet, you know, on some remote oddball response buried deep in a semi-tech, semi-gaming forum about a subject completely different from what you are searching for found on the 8th result on page 26,345 in your Google results?

I probably should have taken this as a sign that the overall project was going to give me a bigger headache, but hey I was pumped.  I documented all the fixes I had to do, found all the drivers I needed to do the offline conversion (I repeated the above process for one other server model, all the others, thank God, were already supported or the drivers stored on my repository worked), and the conversion process went off without any further hitches.  Every test server converted from this point on worked perfectly.

The process was designed and now I could begin setting up System Center and do some testing in the production environment.  This would be easy…

Check back later for the next part as I begin to discover the horrors of a mangled domain, server settings configured to prevent essentially the exact thing I wanted to do, and a wide variety of other fun “speed bumps” – cough mountains cough – that were and are still currently preventing complete success in this project.

With deadlines coming up on the hardware purchase and emails about to be sent to clients that will be affected by this transition, each obstacle is a heartbreak.  And the further I get in to fixing and reattempting, the more vague the error messages get (Thanks Microsoft, I love you!)

Posted in Technology | Tagged , , , , , , , , | Leave a comment

Launching my Blog

After months (years if you want to get technical) of saying I wanted to put up a blog and stating that I was going to put up a blog, I have started. The key now will be to maintain it, of course.

Initially my feelings about blogs was that there are far too many of them out there and that the internet was swamped with tons of information already – good, bad, insane – and that the addition of any material I could share would quickly be lost in the ocean.  I have since decided that mentality is a bunch of hogwash and regardless of if anyone admits to reading my thoughts and opinions, at least I will feel as if there is a chance that I can make an impact in someone else’s thought process (good or bad depending on whether they agree with me or not.)

To start it off, this is my first real post.  What is to come after?  Who knows…

I work in a technological field – as of one day ago I was asked to take over the position of Director of Operations at the company I work for.  This position is the head of the Infrastructure department which builds, monitors, and maintains the company’s entire network which is spread between a few states and over to Manchester, England.

Many of my blogs most likely will be related to experiences, breakthroughs, and thoughts on things associated with that as well as web hosting and, of course, many other related items.

I am also very politically opinionated.  So I may blog about that – my wife believes that I should actually have a radio talk show to cover all my material.  Too many of those out there already but yeah I could see how that would be fun…

Follow me on twitter @McCartneyCorp

Anthony McCartney

Posted in Uncategorized | 1 Comment