TL;DR - Did a clean install. Recreated users, shares and printers.  Export mailboxes to PSTs.  Copied data over.  Reimported mailboxes. Got on with my life.

For the second run at upgrading the server, I had a few new tools at my disposal.

First, was a virtual copy of their server.  Since their server was already virtualized, I just stopped VMWare, made a copy of the VM and took it back to my office.  I loaded it up on a little Vostro 200 I had laying around (the server was amazingly peppy, all things considered) and then figured I'd do 99% of the migration off-site and then drop the box in and copy what was left of the data and mailboxes over.

I made the smart move and took GHOST images of the new box at every juncture, so I could "easily" roll back.  (Granted, the roll back takes 15 minutes, but it's waaaaay better than having to sit thru the unpacking of files every time something goes wrong.)

I will say it took an inordinate amount of time to get the right drivers to get the Dell T310 work with Sysinternals' ERD Commander 5.0 (Microsoft version; not the pre-buyout version with Firefox in it...) but it was worth it.

Let's say I rolled back more than a few times.  Learning a little more each time; or sometimes just doing everything identically, but having it work the second time around.

This guide came in handy: http://blog.mpecsinc.ca/2009/06/sbs-2003-to-sbs-2008-migration-guide.html

After doing this dance for a couple days, it got to a point where I was fed up with the roadblocks and things not working right (like migrated uses not showing up in the users applet).  From an email I sent to my partner at midnight the night before we were supposed to be on site, where I make the call to punt:

"OK, I've had it. Every time I think I got it, there's some fucking error somewhere. I assume its in the old box somewhere -- a remnant of the P2V or something. So, fuck the migration. Their environment isn't so hard that we can't just do a clean install from scratch, remake the 10 users and the 4 shares and just be done.  Can you come down tomorrow since we're going to have join the machines to a new domain? I think the biggest challenge is going to be exporting/importing [the owner's] 8GB PST. I'll have all the users set up and the groups and most of the data copied over, so it'll just be joining the computers to the new domain and importing the mail."

From there, it was a dream -- everything worked just fine.  We had one machine exporting people's PST files for later import.

(WARNING: You lose Single Instance Storage when you do it this way, so your Exchange store will be bigger than it was due to duplication of attachments, etc. This may not be right for you.  YMMV.)

At about 1:30 AM we were ready to move the new server into the copy room where it was going to stay.  The server asked to apply 59 updates, and like idiots, we let it, and that's where it all went to shit.

The reboot happened and then nothing but "Applying Updates, Stage 3 of 3..."

After an hour of staring at that screen, I sent my partner home.  No sense have two of us stuck in graybar land.

After two hours, I started Googling solutions to the problem.  Eventually I was able to start the server and get to safe mode where I could edit some XML files and cross my fingers and wait more.

After another hour, I just left.  It was almost 4 in the morning by that point, so I got a room at a nearby motel (to save myself the 65 minute each way commute to the client office) and crashed hoping to get back onsite before everyone showed up at 8.

As I wrote in my post-mortem email to Brian at 11:20 AM:

Subject: What a fucking mess

Everything works and nothing works,

Ended up staying at the local Days Inn and crossed my fingers that the thing would roll back while I slept, and when I came back at 8:15, we were at CTRL+ALT+DEL so it seems to be OK,m tho it lists all 59 updates as pending. There's a possibility they'll never take. (Fun!)

"Where are my contacts!?!" (nickname) - bring them over and all the nicknames for the internal people are all wrong, so the mail bounces.  NK2Edit to the rescue.

Scanner set up.

Abacus' DRM is driving me up a fucking wall.

Some documents that people were working on last night never made it over -- feels like there wasn't a sync done, and since we generally deleted the offline files from the workstations to eliminate the error we were getting at logoff sync... DOH. Happily, it seems they were able to recreate them with little effort, so it was an inconvenince, but not a show-stopper.

People don't like to use their address books, I find -- they really REALLY rely on their NK2 files -- so when doing any sort of migration, don't overlook them.  NK2Edit is the greatest tool in the world.

I hate DRM.  I think it only hurts legitimate users, pirates get past it with little effort, so it only serves to create issues for paid users at the worst possible times -- like when people need to use their time and billing systems and the support center isn't open for another hour or two...

The document sync issue wasn't something we thought was an issue -- we made sure offline files were all sync'd up, but users reported some documents didn't copy over, and I'm not sure what else to attribute it to.

In the time since we did this, the 59 updates did in fact take and the server didn't fall apart or fail.

Then I see the Windows Update shield, and I open it up much to my horror:

118 UPDATES?!?

So, at 2:05 PM, I clicked "AGREE" on a licensing agreement, and off we go...

I have a thread going over on Facebook...

JK (me): Shall we take bets as to how long this will take to run?

AF: I'll take the over on 5 reboots.

JS:  i am guess it will take 6 hours total. 5 hours and 45 minutes of annoying pop ups. 15 minutes of actual updating

JK: 2:10 - 2 security updates done... 0 reboots.

JK: ‎2:16 - 10 down, 0 reboots

JC: Don't forget Genuine Advantage.

JK: That's always last, isn't it?

JK: 2:36 19 down... 0 reboots.  (Net Framework installs take the longest.)

JK: 2:46 Update 22. Net Framework 1.1 SP1; see you in an hour.

We're relying on Microsoft's instructions to get us through the night.

It's 7:31 pm on a Friday night.  The air conditioning in the office has been turned off, so it's a bit sultry.  I prepped the source server last night -- just made sure its service packs and patches were up to date.  Now that I'm on site, I'm ready to make my answer file.

To do so, we need to install some tools from the SBS 2011 install media (which happily Dell included).

Sadly, the pre-requsites to using the tool don't run on Windows XP, and this is an XP shop.

Great.

However, I've got two Dell Inspiron 560s ready to be dropped down and they have Windows 7 on them, so let me unbox one and get it powered up so I can get the server migration started.

(I had planned on installing the workstations when the data was migrating... looks like I get a jump on it...)

I'll be back once I'm unboxed...

 7:56 - Machine unboxed, Facebook notified of liveblogging, OOBE completed, waiting for my desktop customizations and personal settings to be applied. Backup of source server in progress.

8:22 - Machine joined to domain, Migration Assistant installed and waiting for backup to finish...

8:31 - Backup can take 7 hours.  Uh-oh.  Wait!  The server is running in a VM!  There was a hardware failure which precipated this migration so we brought the old box back up in a VM.  (Pretty slick apart from a couple licewnsing activation gotchas.  "DRM -- Screwing the honest since 1998.") So, I can just take a snapshot and roll back if anything goes horribly wrong! Score! Back on schedule!

8:37 - Migration Assistant found some issues...

9:08 - Issues persist.  Odd ones  that aren't true - the remote registry service IS running, but the migration assistant refuses to believe it...

9:51 - Ran some more hotfixes on the server after SBS BPA suggested it.

10:16 - Re-read some documentation.  Last night, I read that (what I thought was) the migration assistant needed to be run on a workstation, not the server.  That does not seem to the case.  I've spent the last hour chasing down ghosts.

10:20 - Running the MA on the SBS server worked right out of the box.  FML.  Wasted all this time due to my stupid brain.  Stupid, stupid brain.

LESSON 1: Run the Migration Tool ON THE SOURCE SERVER.

10:42 - Answer file created, copied to USB key, USB key in destination server and the install is now running on the destination server.  Right out of the box, the new Dell server runs thru its little preinstall and then dumps you at a screen that says "Basic or Migration Install?"  I choose Migration, it reads the answer file like magic, and we're off to the races.

I created the answer file to not be unattended. I like clicking NEXT when I do my server installs - just to make sure everything is right one last time before committing.  Good thing too, since I misspelled the domain name on the credentials section of the answer file...

Now we wait what may be 30 minutes for Windows to expand its installation files...

10:53 - Server just rebooted itself...

10:54 - Seemed to apply some updates, back to waiting for expanding files... and in the time it took me to type this, it started rebooting again... and we wait some more...

11:02 - Finally, motion!  The bar on the screen started moving across.  I guess all the updates are done... more waiting...

11:03 - I guess it connected to the domain because a printer mapping came up and failed due to an incompatible driver... more waiting...

11:31 - 80% of the way thru the bar.  "Up to 30 minutes," indeed!"  Luckily, the new workstations (before decrapifying) have the WildTangent games on them.  They're usually the first things to go, but right now, I'm glad they're here...

11:43 - 90% of the way thru... when does the 6 hour "migrating data from old server" start?

11:44 - No sooner than I type that, then Explorer starts putting up its "Customizing Browser Customizations..." dialog, and the SBS Setup program quits with an error.  Swell.  (Looks at logs... might it have something to do with time synchronization?!?)

12:12 - Fantastic.  The source server got hung up (a backup started: LESSON TWO - STOP TASK SCHEDULER) and the clock stopped advancing, which caused Kerberos errors, and then the only way to recover is TO START OVER.  So, rebooted both servers, booted off the DVD on the destination server, reformatted the partitions and am now sitting thru the whole thing again.  Sloppy... but this is why we flat-rate server installs... my mistakes are my own and come out of my end, not the client's.

12:41 - We're back at where we were at 10:42.  Lost almost 2 hours ot the minute.  Dang.  Worst part, is I don't have a key to the building, so I'm trapped here.  Can't even go out for a bite and come back...

1:08 - Server install fails with "JoinDomain_DCPROMO Failed..." - turns out the initial failed install got as far as seizing all the roles from the other server and was already in the domain.  So, instead of trying to reverse everything I did, I just reverted to the snapshot from 8:31.

1:22 - Snapshot still had the anti-virus installed and didn't have the migration assistant run, so we're waiting for the post-AV, pre-MA reboot.  *sigh* I'll be pretty good at this when I finally get it done... tho I am ECSTATIC that the server is a VM and all it took was the snapshot to roll it back.  I don't know what I would have done (likely a clean install, then a manual migration of mail and a visit to all workstations to rejoin them to the domain) if I didn't have that capability...

4:19 - Been a while.  Had to give two it more tries.  Both failures.  This last time, I logged in to the domain once as the new user they want you to create to act as the migration account, and maybe that was the charm, because now -- seven hours after we started -- I can proceed to STEP FOUR of the 19 Step Migration document...

8:34 AM - Looked good for a while, then it didn't.  The Migration Wizard on 2011 isn't as automated as one might hope, and in my bleary eyed state, I made some missteps and by the end, nothing seemed right, so I punted.  Old server restored from snapshot and I will take another run at this soon, once I catch up on some sleep and more reading in the migration guide.

We had an in-house project that needed SQL Server, so we installed SQL Server 2008 R2 Enterprise in Evaluation mode, since we didn't have the product key handy.

Install went fine, server works great, everyone is happy.

170 or so days go by and we run an SSIS package and notice the warning "Evaluation Version, will expire in 8 days"  Wha?  Do a quick Select @@Version and sure enough, we're still running the evaluation version... in a production environment.  (Facepalm!)

Searching around for how to turn my eval version into a licensed version (we finally got around to requesting the key from MS) showed all sorts of solutions, none of them seemed to really fit the bill.

Some people admonished "You're using Eval software in a  production environment!?? Serves you right!  Uninstall/reinstall!" others required odd registry hacks to make the setup program run -- it all seemed so complex for what should be a pretty simple process.

Lo and behold, buried at the very end of the comments attached to one of these complex blog posts, was the simplest of solutions from a chap named Waleed Al-Qudah:

"Go to Microsoft SQL Server Configuration Tools, and navigate to SQL Server installation Center then click the Maintinance link and choose Edition Upgrade."

Two minutes later, we were all legal.

The SQL Server Installation Center is a fairly complicated looking piece of software, and I never noticed that tab nor that option.  Needless to say, I spent a few more minutes looking through it.

For English based systems, the MSMAPI32.DLL file is living in %programfiles%\Common Files\system\MSMAPI\1033 folder.  Other locales will be in a numbered folder other than 1033.

We ran up against this when a client got a computer with a 60 Day Trial of Office 2007 on it, but installed their corporate version of Office 2003 over it.  Uninstalling Office 2007 presented the error.

He sent:

More failures. This was to our accountant and my fiancée. It took two days to receive this failure notice. I will forward the originals.  Have you figured out what is going on or how to fix this?

I replied:

The NDR comes back in 48 hours because the mail server attempts to connect to the other server for 2 days.

Email was never designed to be 100% timely, it was designed to deliver mail come hell or high water; so if there’s a problem with the server you’re trying to send to, our server tries and tries again, hoping the problem is fixed within 48 hours.  If it is not, our server stops trying, rejects the message and lets you know.

4.4.7 errors usually indicate some issue with the recipient’s server (from: http://support.microsoft.com/kb/284204):

Numeric Code: 4.4.7

Possible Cause: The message in the queue has expired. The sending server tried to relay or deliver the message, but the action was not completed before the message expiration time occurred. This NDR may also indicate that a message header limit has been reached on a remote server or that some other protocol timeout occurred during communication with the remote server.

Troubleshooting: This code typically indicates an issue on the receiving server. Verify the validity of the recipient address, and verify that the receiving server is configured to receive messages correctly. You may have to reduce the number of recipients in the header of the message for the host that you are receiving this NDR from. If you resend the message, it is placed in the queue again. If the receiving server is on line, the message is delivered.

Problem was, I didn't see where the recipient’s server was rejecting any emails in the logs; nor could I see why the message was getting hung up.

So, using mails to sent to his fiancee on 2/24 as a test, I found that all mails sent to her were successfully delivered EXCEPT the calendar invites / meeting requests.  So, I had to question why  that was.  Obviously, sending to that mail server wasn't the issue; the user was not blacklisted or anything, but something was getting hung up.

The SMTP logs didn't show anything out of the ordinary, so I figured the items were never even making it into the queue.  Using Message Tracking, we were able to verify that it never hit the queue.  It got stuck in "Message Routed and Queued For Remote Delivery"

And then two days later, the NDR was generated.

So, further investigation made me ask this question:  Are the people bouncing messages using Exchange 2007? I couldn't tell from one of the servers (custom SMTP banner) if it was even running Exchange, but the other said "220+domain.local+Microsoft+ESMTP+MAIL+Service+Version:+2.0 0 0 71 0 47 SMTP."

What made me ask this question was this note from Microsoft we turned up after some research:

Consider the following scenario. A Microsoft Exchange Server 2003 organization and a Microsoft Exchange Server 2007 organization exchange communications by using SMTP. An Exchange 2003 user organizes a meeting and then sends a meeting request to an Exchange 2007 user. Additionally, the Exchange 2007 user accepts the meeting request. Then, the meeting organizer uses Microsoft Office Outlook to send a meeting update message or a meeting cancellation message to the Exchange 2007 user.

In this scenario, the meeting update message or the meeting cancellation message is not delivered to the Exchange 2007 user. Instead, the meeting update message or the meeting cancellation message goes into the SMTP retry queue. If an administrator tries to open the message in the Exchange System Manager console, the administrator may receive the following error message: […]

After some time, the sender of the meeting update may receive the following NDR:

Your message did not reach some or all of the intended recipients.
Subject: test message
Sent: 8/20/2008 11:44 AM
The following recipient(s) could not be reached: user@domain.com on 8/20/2008 11:34 PM
Could not deliver the message in the time limit specified. Please retry or contact your administrator. <server.domain.com #4.4.7>

So, it looked promising -- the article mentioned both calendar invites and error 4.4.7 along with the message getting stuck in the queue, but the initial scenario is not quite correct (it seems to assume the Exchange 2007 recipient got the request and accepted it; something that’s not happening here).

The Microsoft article mentioned a hotfix, which we downloaded and applied.

After the hotfix was applied, I sent a test calendar entry to the client and his fiancee (whose address which was giving us a hard time) and lo and behold, the invite went through:

(In playing with the Message Tracker, I noticed the time was being reported an hour fast.  Turns out, there's a hotfix for that, too.)

However, when I got on-site, there was no malware on the machine. 

A quick glance over the Autoruns turned up nothing out of the ordinary, MalwareBytes turned up nothing, doing a netsh winsock reset did nothing to improve the situation... not feeling like standard malware... was this thing rootkitted?

But then I noticed there was an odd entry in his start menu called "ZoneAlarm!12dc_532f!erased" which had a broken link to a "ZoneAlarm Security Tutorial"

The client does use ZoneAlarm, and prior issues with his machine had been traced back to ZA -- most notably when a Microsoft DNS patch befuddled the firewall, forcing it to block all DNS traffic -- so I didn't rule ZA out of the equation.

Searching all the files on his machine which were created in the past 24 hours, turned up a file in c:\windows\internet logs called fwpktlog.txt (FireWall PacKeT LOG anyone?  And why are we still limiting ourselves to 8.3 naming?  Does anything other than our own laziness still rely on that?)

In this file was a whole bunch of lines denying packets from the local machine to Internet addresses.  I could still access machines on the local network, but nothing outside.

What created that packet filter file?

I looked in the services list and there was nothing for ZoneAlarm in there, but there was a entry for ForceField (ZA's browser security program) and it was running.  Stopping the service and setting it to disabled did nothing.

Open the pod bay doors, Hal! 
I'm afraid I can't do that, Dave.

Also, among the recently created files, I found an installer_02251175403.log file, also in the c:\windows\internet logs folder.  Opening it up, it pointed me to a temp folder it had created: C:\DOCUME~1\Steve\LOCALS~1\Temp\02251175403 within which was a bunch of files including an executable called cpes_clean.exe which is listed as "Check Point Endpoint Security Cleaner" from "Check Point Software Technologies LTD" in the file properties.

Going for broke, I ran the program.

It ate up some CPU according to Task Manager, but didn't show any sort of UI until it popped up a box asking to reboot.

Rebooted the machine, and lo and behold, we could access sites out on the internet.

Pop over to ZoneAlarm's and download the latest build, and 145MB and 15 minutes later (DSL!!!) and the download is corrupt.

Another 15 minute download, and ZA is happily reinstalling itself.

Upon setting up the server, moving the USERS share was always the first order of business.  Even when 12GB seemed roomy, it was obvious 15 users was going to eat their way through the share space in nothing flat, and a move to the data partition was in order.

The Health & Monitoring server was another space hog, with its database growing out of control until it expanded, like a gas, to fill all available space.  So, a quick reinitialization of the database clears up some space...

But a lot of these things are quick fixes, but there's a lot of them... so it was very nice of Microsoft to bundle all of them into a single document: Moving Data Folders for Windows Small Business Server 2003

The 600k Word Document is a tremendous little cookbook! It covers just about everything:

• Step 1: Complete and Verify a Full Backup
• Step 2: Notify Users that Resources will be Unavailable
• Step 3: Move the Users Shared Folders
• Step 4: Move the SharePoint Databases
• Step 5: Move the Monitoring Database
• Step 6: Move Exchange Databases and Log Files
• Step 7: Move the Sent Faxes Folder
• Step 8: Move the ClientApps Shared Folder

Thanks, SBS Team!

Related Links 18

I turned the firewall off  and still couldn't connect.

Turns out TCP/IP isn't turned on by default.  Maybe I knew that at one time, but this time it totally slipped  my mind.

To turn it on, go into SQL Server Configuration Manager, expand "SQL Server Network Configuration" and change "Disabled" to "Enabled" by right-clicking on "TCP/IP" and suddenly, I could connect from my development machine.

Author: Jeff Trumbull

Description:
Backup files that make a Microsoft virtual server with only about 1 minute of down time. Suspends the virtual server, takes a shadow copy , starts the virtual server then copies virtual server files. This could be used to copy any open files. Requires vshadow.exe from vss sdk.

Script: