so we started looking into Hyper-V

as a Non-profit we get huge discounts for Microsoft software through some third-party vendors, so it made perfect sense for us to move to Hyper-V and achieve the automated process we were looking for. now, moving from an infrastructure platform to another is not easy, but we plan to run both in parallel, and any upgrade or new VMs will be built on the new Hyper-V platform until we face out VMware completely.

I believe that even if we were not a non-profit company, we would benefit by the move, because Microsoft Hyper-V is a lot cheaper than VMware, and we can basically accomplish the same thing. for a company with thousands of users it probably makes more sense to use VMware, but for a couple hundred users, Hyper-V is more adequate.

it happened a few months ago on a Friday morning. everyone one at the office was happy, drinking coffee, and some of them chit-chatting about plans for the weekend etc. as usual I was walking around the office completing some help desk calls, and as a good practice I always keep the IT phone with me when I’m not at my desk to check emails periodically and respond quickly to any emergency help desk call being logged, and that morning I noticed a couple regarding exchange server being slow. I know many people don’t have patience enough and exaggerate problems sometimes, but when I saw an email from the CFO regarding the same problem, I knew I had to act, and act fast.

the first thing I did was to RDP to exchange and see if there was any problem with the server. sure enough, the RDP connection took about 2 minutes to get established, and the remote desktop icons took another 3 minutes to show up, the server was painfully slow. after checking disks speed, event logs, and other basic things, I opened task manager and immediately noticed the store.exe process taking up all the processor resources. it was 100% all the time, barely spiking up and down.  I know the store.exe allocates a lots of memory, but I never had seen it taking so much processor resources as well before, so I did some research about it, and found little to none information regarding the issue.

We do have other critical applications running on the same ESX Host server Exchange is on, like the accounting application, and our Intranet portal, and soon the developer guy came to my desk to let me know those applications were running slow as well, and he suspected it was hardware problem for which I am responsible for. oh dear, from there on I knew it was going to be a long weekend. I know our portal server runs on a crappy CMS engine, and my first hint to the CIO was that probably it was the portal server that was causing the problem, and that VM needed to be either shutdown or moved to another ESX less critical than where it was. he told me to restart the entire ESX server to see if that cleared the problem, so we did, but the problem was still there. then he told me to proceed with the portal VM move. I new the portal server was fine, because I had checked everything, event logs, disks speed, processor, and memory usage and everything was normal. but the server was painfully slow, the same as the exchange server and I could not find the reason why. I didn’t have a clue of what was the real issue, but on a situation like that, is better to do something than nothing, so I started getting ready another VM for the portal move. I figured that just migrating the VM would not work, because whatever was the problem it was going to be inherited if I just moved the VM,  so a new VM needed to be built, installed the software from scratch, and restore the portal database from tapes etc..

but my brain kept telling me, exchange server is the problem…..I just did not want to listen to myself because I was scared to death to accept that our most important application ( exchange server ) was the problem, and I did not know how to fix it. and because it is Exchange server 2003 on VMware, it is not supported by Microsoft, so I could not get any help from them. I was screwed. after a few reflections I thought there was no point wasting time moving the portal server to a different VM, when Exchange was the real problem.  so I told the CIO I really thought it was exchange server the issue, and we should concentrate on that instead. by now I was losing confidence with the CIO, but he agreed that if exchange store.exe process was taking up so much CPU resources then most likely that was the issue.  so we focus on exchange.

the biggest thing that troubled me the most was, why all the VMs on the ESX host were slow, when only the Exchange server VM was the problem. my understanding was that any performance issue on any VM should not have affected the rest of VMs on the same host. but clearly this was not the case here. it seems that the store.exe process in exchange was slumping the performance on the entire ESX host.

Soon I got with VMware technical people on the phone for help, and after a few minutes of discussion one of them remote into my computer, and fired up the ESXTOP tool to investigate the problem. all our ESX VM disks run from a Dell SAN, so he suspected the SAN was the problem,and sure enough, he found out that the Exchange VM was sending a lot of write/read traffic to the SAN, and there was a big latency with it , so he suggested me to call Dell for support, then he hung up. basically he left me on the same boat, and by now it was late Friday afternoon, and people were getting pissed off about Outlook being so slow for that long.

I was not happy with the VMware guy conclusion, because if the SAN was the problem, then all our other ESX hosts would be slow as well ( because all VM disks run from the same SAN ) , but the other ESX servers were running smooth like a beer in summertime, and it was only that particular ESX host where exchange was on that was slow, so it had to be exchange causing the problem, not the SAN.

I really had to focus on exchange now. especially on the store.exe process.

I stopped the SMTP service, IIS, Exchange information store, system attendant, basically everything. then rebooted a few times, nothing. I disable Symantec antivirus, still nothing, remove it completely and reboot, still nothing…. I was getting very desperate and drinking coffee even with my nose.

after trying a zillion different things to fix the problem my brain was exhausted, and I could hardly think rationally any longer. so I sat on my chair almost motionless wishing for a miracle to happen, of course that was impossible because I hardly believe in God, so I was helpless from all directions.

then I scarcely remembered an Inter-Org tool my colleague in Bogota and I had setup to synch up public folders between our exchange servers, and  I thought to check that up just in case that was high-jacking our SMTP tables and thus making the store.exe process to use up all the processor resources. I started by examining the logs, and I noticed the tool had not run for awhile, but was trying to connect to the remote exchange constantly and it was generating a lot of logs. Holy crap said I, this is surely the  culprit of my nightmare, so I went to the services console and disable the service for the tool immediately and rebooted the server. what a relief I felt when the server came back, and it was fast as usual, with the store.exe process taking almost nothing of the processor resources. till now I have only experienced two intense pleasures in my life, an orgasm, and that moment of relief.

since then we have not had any performance issue with exchange again, but the question still remain unanswered:

why performance on one VM slumped the performance on our entire ESX Host server?

As you probably figured, the hypervisor does not need the service console to run, so that’s what they did with ESXi, they removed the service console from the hypervisor. the good thing is that by doing that the hypervisor is now very small, and the bad thing is that by stripping off the service console ESXi has less support for hardware, and updating drivers on ESXi requires a VMkernel update also.

So, which is better, ESX or ESXi?

you can do the same thing on both hypervisor, they both support the same virtualization features, so it becomes a matter of preference. I personally like ESX for the custom options it offers, and size partitioning. though I trust VMware with the ESXi speed-light installation without any decision on my side, I rather customize the installation myself.

so to summarize everything, the difference between ESX and ESXi is the service console. remember ESX has a service console, ESXi does not.

reference: http://www.vmware.com/products/esxi/uses.html

I tried to shutdown the ESXi server from the VMware infrastructure client when it happened.

weird, but I had to manually shutdown my server, which I really hate to do.

so how you change the password? it is very simple. when you first login to your VMware infrastructure client, you will get some basic tasks recommendation.

click on the Change the default password link you will be prompted to enter the password.

enter the password and click on OK.

now your ESXi server should be passworded.

from your VMware infrastructure client click on the Configuration tab, and then click on DNS and Routing under the software section:

then click on Properties on top:

change the host identification, and DNS if you like on the following window:

after choosing your settings click on OK.

Most likely you will need to restart the server in order for the changes to take effect. the new name should appear after the server comes back.

that’s it. I hope it helps.

 After a small research I found out that ESXi 4.0 does not support 32-bit only processors, and unfortunately  the XEON processor on this server does not support 64-bit processing. I will have to install ESXi 3.5 , and find a newer machine to install 4.0 for my testing.

I have never seen such a freaking simple program doing such a great job. once you download the file from the link above, unzip it, and run it.

when this small program pop-up, choose the Vmdk file on the Source Vmdk option. then choose the destination of the Vhd file that will be created.

then press the convert button.

that should convert your VMDK files to VHD files. then just install virtual server and start managing those VHD files.

Let’s get started.

Preparing the virtual Machine

If  this is the first time you visit this website, I recommend you to read my previous post on how to create a virtual machine Creating a Virtual Machine on VMware 2.0 remember that you will be booting your virtual machine from an ISO image. to do that, right-click on the virtual machine you created in the tutorial above and go to Settings:

click on the CD-ROM icon then choose Use ISO Image then click browse to find your ISO image.

Windows 7 Installation

Click OK. and reset your Virtual machine.

after rebooting the virtual machine, the Windows 7 Installation wizard should begin. the first window that should come up is this:

Choose your Language, and click next.

right after, the Install Now window should pop-up:

Click on Install Now, and let the installation  begin.

After awhile, the Microsoft license agreement should come up. accept it, and click next.

Next, you need to choose which type of installation you want. choose the Custom (advanced) option. the upgrade option is selectable on this beta build. it must be a bug because as far as I know there is no windows installation on this VM.

Click Next.

Select the drive where you want to install Windows 7. if you have an IDE drive probably it will show up here, but if your hard drive is an SCSI or SATA you will need the drivers. the best option is to change the settings on your virtual machine to use an IDE drive instead of a SCSI.

click Next, if your drive is selected.

The file installation should begin.

That will take awhile….

after finishing with the window above, Windows 7 should reboot. you will get the fancy windows 7 starting logo.

it should check the system performance.

after that, it should ask you to select a username.

after choose the username, click next.

then comes the password.

click next, after choosing password and hint.

then the product activation. and because windows 7 is not even out yet, we should not put any registration key here.

Simply click Next.

The Next window is the updates. choose ask me later.

Next, choose your date and time zone.

That’s it. windows 7 should be ready to load the desktop for the first time.

Enjoy Windows 7.

Getting The Server Ready

If you don’t have Windows Server 2008, download the trial version from Microsoft from the following link http://www.microsoft.com/downloads/details.aspx?FamilyId=B6E99D4C-A40E-4FD2-A0F7-32212B520F50&WT.sp=_technet_,dcsjwb9vb00000c932fd0rjc7_5p3t&displaylang=en

You can use Windows 2008 in trial mode for up to 6 months.

Once you downloaded the ISO image, logon to your VMware Infrastructure Web Access, and click on the virtual machine you created following the tutorial above, then right click on theCD/DVD Drive and select Edit:

Then right at the bottom select ISO Image.

Minimize the browser and copy the Windows Server 2008 ISO Image you downloaded from Microsoft to your Virtual Machines directory. in my case that directory is at C:\Virtual Machines\Server 2008. now go back to the browser again, and click on Browse and the Server 2008 ISO image should show up in the directory like the image below:

Select the ISO Image and click OK.  now you should be ready to start.

Installing Windows 2008

Power up the virtual machine.

After a few seconds of loading, the following window should come up:

Choose your settings and click on Next.

The Install Now button will come up.

Click on Install Now.

next, the activation button will come up. and because we don’t have an activation key for this server, we won’t put any. just click Next.

You will get the following warning:

and assuming you are installing Windows Server 2008 just for testing clicking NO should be fine. otherwise you probably will need to buy a license when the trial period expires.

Next, select the edition of Windows you purchased ( downloaded) from Microsoft. Windows Server 2008 Standard ( Full Installation)

check “I have selected the edition of windows that I purchased” and click on Next.

Next, the license terms. check ” I accept the license terms”

Click on Next, after agreeing the license terms.

Next, choose the type of the installation you want. the Upgrade option should be grey-out, so choose the custom (advanced) options.

Choose, Custom (Advanced) option.

Next, Installer will ask you where you want to install Windows Server 2008. of course we only have one drive on the VM machine. so choose that:

click Next, after selecting your drive.

Windows should start installing at this point.

That should take awhile…

the above window should come up after windows rebooting.

Windows completing installation.

After awhile, windows will ask you that the password needs to be changed before logging in the first time:

Click OK.

choose your password for the administrator user.

After changing the password, windows should start preparing your desktop for the first time.

Wait a little bit, and windows will load.

Change time Zone On the Server

the first thing you should do when windows loads, is to change the time zone. on the Initial Configuration Task page that comes up the first time, click on Set Time Zone:

click on Change Time Zone on the window that pops up, and choose your zone, and click OK.

Now server 2008 should be installed on your VM machine.