As you begin to prepare for your migration to System Center Virtual Machine Manager 2012 R2 (SCVMM 2012 R2), one of the best nuggets of advice I can offer is to perform your dry run from end to end. Even if you are a seasoned VMM administrator, there are so many tiny dependences that you need to be aware of, especially if you have chosen to jump from 2012 RTM to the latest R2 Release.
Although the upgrade process might initially read as trivial, it is imperative that you should run at least one pilot upgrade of the environment prior to going forward with the real thing. However if this pilot is to be of any value you must take into account the versions of your SCVMM hosts and libraries, including both the OS and application levels. Ensure you match these source versions for your upgrade pilot, otherwise your pilot will have questionable value when you approach the real migration.
In addition, one of the most important elements to keep in your pilot will be the VMM database and its associated SQL Server instance version. Create a backup of the live database; using a SQL Server Instance of the same version and patch level, restore the backup for use in the pilot executions. If you hit a problem, you can then simply roll back the database and try again.
As you start the execution of the upgrades, you will want to monitor the progress in more detail what exposed from the installation wizard. The logs which you will monitor are located in %ProgramData%\VMMLogs, and using the CMTrace.exe tool you can easily monitor in real time the content of files including SetupWizard.log and vmmServer.log
We will look at just one of the problems you might encounter, especially if you are rolling the upgrade on an environment which has been running for some time.
SCVMM contains a table of performance history for your Hyper-V hosts, which can grow to quite a size. As an example, a small environment of 20 hosts running for just over 18 months will generate about 3.5 million records! This table can induce a failure on an upgrade pass as one of the SQL actions executed is to alter this table, which can take quite an extended amount of time (33 minutes in my lab) and is more than enough to make the setup wizard fail.
The following screen shot, is taken from the vmmServer.log file, and we can see the actions that upset the setup wizard.
Connecting to the SQL Management console, simply cutting and pasting the above commands to execute them one at a time on a new copy of our VMM database, quickly gets me to the point where I can determine that the ALTER TABLE TSQL command is going to be real problem for us (33 minutes 26 seconds to complete).
A second observation while this command is executing: Our SQL transaction log has ballooned by no less than 10Gb in the course of executing this statement.
As the information in this table is historical (and in my case, is approaching 18 months of data, all of which is related to the performance of my managed Hyper-V hosts), a quick peek into some of the records will show what is stored for us and for VMM to use; for example, in reporting and host optimization.
The information is important, but not enough to prevent me from requiring to move forward. I do have backups of the database and have over the last months created reports from this data. At this stage I am ready to cut loose and move forward.
My transaction log is already full, so if I execute a SQL Delete * type command, this will just add to the pain. Therefore, I will just truncate the table.
truncate table [VirtualManagerDB].[dbo] .[tbl_PCMT_PerfHistory_Raw]
And to check this has the desired results, lets run our select query again, this time expecting the table to be empty.
Now, if I go back and run the Alter statement, it will complete in a few milliseconds. Although my historical data is now all gone, I can prepare to run the upgrade again. Do be careful though and do not run the Alter Statement – allow VMM do that for you, otherwise you will have an new error in the logs, due to the table already been altered, and that is not any good for us!
VMM is a critical component in the management of our environment, and it gathers a lot of data. Not all of it is 100 percent of value to everyone, and sometimes hard decisions need to be made when data maintenance is not performed. Due to the critical nature of VMM, take the time to reproduce your production environment prior to any upgrades you might be planning and run some pilots. You are almost guaranteed that there will be some nasty gremlins just waiting for you as the midnight hours approach, so be prepared.
Of course, a rollback plan is always a good investment. But one thing’s for sure: Upgrading and rolling back isn’t going to be trivial. Good luck!