There's been good response to my earlier post (really, July?!?) about upgrading a client's server from SBS 2003 to SBS 2011 -- people want to know how the story ended.  So, forgive me for taking so long to finish up.  So, this will be short of details since I didn't take notes and am relying on memory.

TL;DR - Did a clean install. Recreated users, shares and printers.  Export mailboxes to PSTs.  Copied data over.  Reimported mailboxes. Got on with my life.

For the second run at upgrading the server, I had a few new tools at my disposal.

First, was a virtual copy of their server.  Since their server was already virtualized, I just stopped VMWare, made a copy of the VM and took it back to my office.  I loaded it up on a little Vostro 200 I had laying around (the server was amazingly peppy, all things considered) and then figured I'd do 99% of the migration off-site and then drop the box in and copy what was left of the data and mailboxes over.

I made the smart move and took GHOST images of the new box at every juncture, so I could "easily" roll back.  (Granted, the roll back takes 15 minutes, but it's waaaaay better than having to sit thru the unpacking of files every time something goes wrong.)

I will say it took an inordinate amount of time to get the right drivers to get the Dell T310 work with Sysinternals' ERD Commander 5.0 (Microsoft version; not the pre-buyout version with Firefox in it...) but it was worth it.

Let's say I rolled back more than a few times.  Learning a little more each time; or sometimes just doing everything identically, but having it work the second time around.

This guide came in handy: http://blog.mpecsinc.ca/2009/06/sbs-2003-to-sbs-2008-migration-guide.html

After doing this dance for a couple days, it got to a point where I was fed up with the roadblocks and things not working right (like migrated uses not showing up in the users applet).  From an email I sent to my partner at midnight the night before we were supposed to be on site, where I make the call to punt:

"OK, I've had it. Every time I think I got it, there's some fucking error somewhere. I assume its in the old box somewhere -- a remnant of the P2V or something. So, fuck the migration. Their environment isn't so hard that we can't just do a clean install from scratch, remake the 10 users and the 4 shares and just be done.  Can you come down tomorrow since we're going to have join the machines to a new domain? I think the biggest challenge is going to be exporting/importing [the owner's] 8GB PST. I'll have all the users set up and the groups and most of the data copied over, so it'll just be joining the computers to the new domain and importing the mail."

From there, it was a dream -- everything worked just fine.  We had one machine exporting people's PST files for later import.

(WARNING: You lose Single Instance Storage when you do it this way, so your Exchange store will be bigger than it was due to duplication of attachments, etc. This may not be right for you.  YMMV.)

At about 1:30 AM we were ready to move the new server into the copy room where it was going to stay.  The server asked to apply 59 updates, and like idiots, we let it, and that's where it all went to shit.

The reboot happened and then nothing but "Applying Updates, Stage 3 of 3..."

After an hour of staring at that screen, I sent my partner home.  No sense have two of us stuck in graybar land.

After two hours, I started Googling solutions to the problem.  Eventually I was able to start the server and get to safe mode where I could edit some XML files and cross my fingers and wait more.

After another hour, I just left.  It was almost 4 in the morning by that point, so I got a room at a nearby motel (to save myself the 65 minute each way commute to the client office) and crashed hoping to get back onsite before everyone showed up at 8.

As I wrote in my post-mortem email to Brian at 11:20 AM:

Subject: What a fucking mess

Everything works and nothing works,

Ended up staying at the local Days Inn and crossed my fingers that the thing would roll back while I slept, and when I came back at 8:15, we were at CTRL+ALT+DEL so it seems to be OK,m tho it lists all 59 updates as pending. There's a possibility they'll never take. (Fun!)

"Where are my contacts!?!" (nickname) - bring them over and all the nicknames for the internal people are all wrong, so the mail bounces.  NK2Edit to the rescue.

Scanner set up.

Abacus' DRM is driving me up a fucking wall.

Some documents that people were working on last night never made it over -- feels like there wasn't a sync done, and since we generally deleted the offline files from the workstations to eliminate the error we were getting at logoff sync... DOH. Happily, it seems they were able to recreate them with little effort, so it was an inconvenince, but not a show-stopper.

People don't like to use their address books, I find -- they really REALLY rely on their NK2 files -- so when doing any sort of migration, don't overlook them.  NK2Edit is the greatest tool in the world.

I hate DRM.  I think it only hurts legitimate users, pirates get past it with little effort, so it only serves to create issues for paid users at the worst possible times -- like when people need to use their time and billing systems and the support center isn't open for another hour or two...

The document sync issue wasn't something we thought was an issue -- we made sure offline files were all sync'd up, but users reported some documents didn't copy over, and I'm not sure what else to attribute it to.

In the time since we did this, the 59 updates did in fact take and the server didn't fall apart or fail.

Visited a client today who was complaining of computer sluggishness among other issues, so I check the Microsoft Security Essentials and it says it's a few days out of date, but a manual attempt at updating failed with an error.  So, of course, I worry about infection getting in the way of doing updates, but it occurs to me he's still on MSSE 1.x and I upgrade him to 2.x and all is well, he can take updates again.  Yay.

Then I see the Windows Update shield, and I open it up much to my horror:

118 UPDATES?!?

So, at 2:05 PM, I clicked "AGREE" on a licensing agreement, and off we go...

I have a thread going over on Facebook...

JK (me): Shall we take bets as to how long this will take to run?

AF: I'll take the over on 5 reboots.

JS:  i am guess it will take 6 hours total. 5 hours and 45 minutes of annoying pop ups. 15 minutes of actual updating

JK: 2:10 - 2 security updates done... 0 reboots.

JK: ‎2:16 - 10 down, 0 reboots

JC: Don't forget Genuine Advantage.

JK: That's always last, isn't it?

JK: 2:36 19 down... 0 reboots.  (Net Framework installs take the longest.)

JK: 2:46 Update 22. Net Framework 1.1 SP1; see you in an hour.

So, we're at a client's and we have to do a migration from SBS 2003 to SBS 2011.

We're relying on Microsoft's instructions to get us through the night.

It's 7:31 pm on a Friday night.  The air conditioning in the office has been turned off, so it's a bit sultry.  I prepped the source server last night -- just made sure its service packs and patches were up to date.  Now that I'm on site, I'm ready to make my answer file.

To do so, we need to install some tools from the SBS 2011 install media (which happily Dell included).

Sadly, the pre-requsites to using the tool don't run on Windows XP, and this is an XP shop.

Great.

However, I've got two Dell Inspiron 560s ready to be dropped down and they have Windows 7 on them, so let me unbox one and get it powered up so I can get the server migration started.

(I had planned on installing the workstations when the data was migrating... looks like I get a jump on it...)

I'll be back once I'm unboxed...

 7:56 - Machine unboxed, Facebook notified of liveblogging, OOBE completed, waiting for my desktop customizations and personal settings to be applied. Backup of source server in progress.

8:22 - Machine joined to domain, Migration Assistant installed and waiting for backup to finish...

8:31 - Backup can take 7 hours.  Uh-oh.  Wait!  The server is running in a VM!  There was a hardware failure which precipated this migration so we brought the old box back up in a VM.  (Pretty slick apart from a couple licewnsing activation gotchas.  "DRM -- Screwing the honest since 1998.") So, I can just take a snapshot and roll back if anything goes horribly wrong! Score! Back on schedule!

8:37 - Migration Assistant found some issues...

9:08 - Issues persist.  Odd ones  that aren't true - the remote registry service IS running, but the migration assistant refuses to believe it...

9:51 - Ran some more hotfixes on the server after SBS BPA suggested it.

10:16 - Re-read some documentation.  Last night, I read that (what I thought was) the migration assistant needed to be run on a workstation, not the server.  That does not seem to the case.  I've spent the last hour chasing down ghosts.

10:20 - Running the MA on the SBS server worked right out of the box.  FML.  Wasted all this time due to my stupid brain.  Stupid, stupid brain.

LESSON 1: Run the Migration Tool ON THE SOURCE SERVER.

10:42 - Answer file created, copied to USB key, USB key in destination server and the install is now running on the destination server.  Right out of the box, the new Dell server runs thru its little preinstall and then dumps you at a screen that says "Basic or Migration Install?"  I choose Migration, it reads the answer file like magic, and we're off to the races.

I created the answer file to not be unattended. I like clicking NEXT when I do my server installs - just to make sure everything is right one last time before committing.  Good thing too, since I misspelled the domain name on the credentials section of the answer file...

Now we wait what may be 30 minutes for Windows to expand its installation files...

10:53 - Server just rebooted itself...

10:54 - Seemed to apply some updates, back to waiting for expanding files... and in the time it took me to type this, it started rebooting again... and we wait some more...

11:02 - Finally, motion!  The bar on the screen started moving across.  I guess all the updates are done... more waiting...

11:03 - I guess it connected to the domain because a printer mapping came up and failed due to an incompatible driver... more waiting...

11:31 - 80% of the way thru the bar.  "Up to 30 minutes," indeed!"  Luckily, the new workstations (before decrapifying) have the WildTangent games on them.  They're usually the first things to go, but right now, I'm glad they're here...

11:43 - 90% of the way thru... when does the 6 hour "migrating data from old server" start?

11:44 - No sooner than I type that, then Explorer starts putting up its "Customizing Browser Customizations..." dialog, and the SBS Setup program quits with an error.  Swell.  (Looks at logs... might it have something to do with time synchronization?!?)

12:12 - Fantastic.  The source server got hung up (a backup started: LESSON TWO - STOP TASK SCHEDULER) and the clock stopped advancing, which caused Kerberos errors, and then the only way to recover is TO START OVER.  So, rebooted both servers, booted off the DVD on the destination server, reformatted the partitions and am now sitting thru the whole thing again.  Sloppy... but this is why we flat-rate server installs... my mistakes are my own and come out of my end, not the client's.

12:41 - We're back at where we were at 10:42.  Lost almost 2 hours ot the minute.  Dang.  Worst part, is I don't have a key to the building, so I'm trapped here.  Can't even go out for a bite and come back...

1:08 - Server install fails with "JoinDomain_DCPROMO Failed..." - turns out the initial failed install got as far as seizing all the roles from the other server and was already in the domain.  So, instead of trying to reverse everything I did, I just reverted to the snapshot from 8:31.

1:22 - Snapshot still had the anti-virus installed and didn't have the migration assistant run, so we're waiting for the post-AV, pre-MA reboot.  *sigh* I'll be pretty good at this when I finally get it done... tho I am ECSTATIC that the server is a VM and all it took was the snapshot to roll it back.  I don't know what I would have done (likely a clean install, then a manual migration of mail and a visit to all workstations to rejoin them to the domain) if I didn't have that capability...

4:19 - Been a while.  Had to give two it more tries.  Both failures.  This last time, I logged in to the domain once as the new user they want you to create to act as the migration account, and maybe that was the charm, because now -- seven hours after we started -- I can proceed to STEP FOUR of the 19 Step Migration document...

8:34 AM - Looked good for a while, then it didn't.  The Migration Wizard on 2011 isn't as automated as one might hope, and in my bleary eyed state, I made some missteps and by the end, nothing seemed right, so I punted.  Old server restored from snapshot and I will take another run at this soon, once I catch up on some sleep and more reading in the migration guide.

A client called this morning saying "I clicked on a security alert and now I can't get on the internet..."  Of course, my spideysense said "Ah!  Classic Malware!"

However, when I got on-site, there was no malware on the machine. 

A quick glance over the Autoruns turned up nothing out of the ordinary, MalwareBytes turned up nothing, doing a netsh winsock reset did nothing to improve the situation... not feeling like standard malware... was this thing rootkitted?

But then I noticed there was an odd entry in his start menu called "ZoneAlarm!12dc_532f!erased" which had a broken link to a "ZoneAlarm Security Tutorial"

The client does use ZoneAlarm, and prior issues with his machine had been traced back to ZA -- most notably when a Microsoft DNS patch befuddled the firewall, forcing it to block all DNS traffic -- so I didn't rule ZA out of the equation.

Searching all the files on his machine which were created in the past 24 hours, turned up a file in c:\windows\internet logs called fwpktlog.txt (FireWall PacKeT LOG anyone?  And why are we still limiting ourselves to 8.3 naming?  Does anything other than our own laziness still rely on that?)

In this file was a whole bunch of lines denying packets from the local machine to Internet addresses.  I could still access machines on the local network, but nothing outside.

What created that packet filter file?

I looked in the services list and there was nothing for ZoneAlarm in there, but there was a entry for ForceField (ZA's browser security program) and it was running.  Stopping the service and setting it to disabled did nothing.

Open the pod bay doors, Hal! 
I'm afraid I can't do that, Dave.

Also, among the recently created files, I found an installer_02251175403.log file, also in the c:\windows\internet logs folder.  Opening it up, it pointed me to a temp folder it had created: C:\DOCUME~1\Steve\LOCALS~1\Temp\02251175403 within which was a bunch of files including an executable called cpes_clean.exe which is listed as "Check Point Endpoint Security Cleaner" from "Check Point Software Technologies LTD" in the file properties.

Going for broke, I ran the program.

It ate up some CPU according to Task Manager, but didn't show any sort of UI until it popped up a box asking to reboot.

Rebooted the machine, and lo and behold, we could access sites out on the internet.

Pop over to ZoneAlarm's and download the latest build, and 145MB and 15 minutes later (DSL!!!) and the download is corrupt.

Another 15 minute download, and ZA is happily reinstalling itself.

So, I'm not a gigantic app person -- my iPod Touch has the essentials for what I need, a few cutsey show-off things or games for the kids, so I figured I'd do the same with my Droid... yes and no. Here's a rundown of whats on my Droid right now, and what I think:

Alarmed Light - Used it because it has a feature where you're forced to answer math questions to turn the alarm off. Kinda forces you to shake the cobwebs out and get started.

Aloqa - Cool app that uses your GPS to let you know what's around. It's integrated with Yelp for food ratings. It has icons on its main screen for "Hot" (whatever that means), Yelp Restaurants, Coffee, last.fm, Music (showing me concerts in Denmark?!?), Playing Tonight (movies), Yelp Bars & Clubs, Real Estate, Wikipedia, ATMs, Pizza, Aloqa, Yelp Fast Food and then "Add more channels." It more or less does what it is supposed to; it's nice to look at.

BeamReader - a PDF viewer. I should uninstall since I bought "Documents to Go"

Bubble Burst Lite -- Windows Mobile Jawbreaker for the Droid.

ConnectFour - decent enough implementation. AI seems a little stupid sometimes.

Documents to Go - open Word, Excel, Powerpoint, PDF. Does good job at rendering PDFs.

Flashlight - turns your screen white. Doesn't seem to adjust for maximum brightness. Passable.

Flickr Droid - Droid needs a good Flickr app. This isn't it, but the best one I could find that uses the Flickr API to let me at my stuff, since a lot of my photostream is friends/family only.

FlightStats Lite - haven't had a chance to play with this. Will in January as I head to CES.

Goggles - Google's latest toy. Varies from wildly successful to "how did you not recognize the Pepsi logo?!?"

GPS Status - essentially a digital compass. Used when I was troubleshooting GPS on the phone.

Flixter Movies - quick and easy to get to where we have to go for Friday Morning Movie Club.

NYC Bus and Subway Map - not as interactive as I'd like. Literally a HiDef graphic of the map, and you can click thru to the MTA website for further details on the lines.

OpenTable - online restaurant reservations from opentable.com . Decent.

Pandora - works well over 3G, tho I imagine it eats thru the quota pretty quick.

PicSay Lite - dopey photo editing thing. Makes speech baloons. Don't know why I downloaded this.

Poke a Mole - whack-a-mole for the phone. Fun game with a Giant Downside - even when phone is muted annoying background music plays. Have to go into game menu to mute it.

Remote RDP Demo - eventually I'll need to really use function keys when I Remote Desktop into a machine from my phone, but until then, the demo version does the trick.

Robo Defense FREE - I do enjoy the tower defense genre of casual gaming... so why not have it on my phone? (Hardly never play it. Seemed like a good idea.)

Shazam - this app still amazes me. Where did they get that song database???

Stopwatch - straightforward.

Sudoku Free - seeing it in my list makes me feel smart until I play it, thenI feel dumb.

The Weather Channel - it has a widget so I can glance at the home screen to see what it's going to be like... or rather I can look at the home screen and tell my wife what it's going to be like... (I had a weather widget on the Treo and missed it.)

TivoRemote - control the Tivo over WiFi. Nice, especially for text entry. The iPhone's version is better.

TRAFFIC! - A test app for me. Not in love.

Trap! - Another game, but its "draw a line" technology gets in the way of gameplay. Fun when it does what you want.

Tunes Remote - Control iTunes from the Droid. YAY! One of the big reasons I got the iPhone touch was to control the iTunes machine hooked to the outdoor speakers.

Twidroid Pro - From what I understand, the best Twitter client for the Droid. I'm happy with it. Does everything I need, but I am far from a Twitter power user, so some might find it lacking. They constantly update it which seems like they're interested developers.

US Traffic - Another traffic app that I tried before I realized Google Maps had a Traffic layer...

wpToGo - Allows me to post to WordPress from the phone... tho I have not had the need to do so.

What have I missed?