TL;DR - Did a clean install. Recreated users, shares and printers.  Export mailboxes to PSTs.  Copied data over.  Reimported mailboxes. Got on with my life.

For the second run at upgrading the server, I had a few new tools at my disposal.

First, was a virtual copy of their server.  Since their server was already virtualized, I just stopped VMWare, made a copy of the VM and took it back to my office.  I loaded it up on a little Vostro 200 I had laying around (the server was amazingly peppy, all things considered) and then figured I'd do 99% of the migration off-site and then drop the box in and copy what was left of the data and mailboxes over.

I made the smart move and took GHOST images of the new box at every juncture, so I could "easily" roll back.  (Granted, the roll back takes 15 minutes, but it's waaaaay better than having to sit thru the unpacking of files every time something goes wrong.)

I will say it took an inordinate amount of time to get the right drivers to get the Dell T310 work with Sysinternals' ERD Commander 5.0 (Microsoft version; not the pre-buyout version with Firefox in it...) but it was worth it.

Let's say I rolled back more than a few times.  Learning a little more each time; or sometimes just doing everything identically, but having it work the second time around.

This guide came in handy: http://blog.mpecsinc.ca/2009/06/sbs-2003-to-sbs-2008-migration-guide.html

After doing this dance for a couple days, it got to a point where I was fed up with the roadblocks and things not working right (like migrated uses not showing up in the users applet).  From an email I sent to my partner at midnight the night before we were supposed to be on site, where I make the call to punt:

"OK, I've had it. Every time I think I got it, there's some fucking error somewhere. I assume its in the old box somewhere -- a remnant of the P2V or something. So, fuck the migration. Their environment isn't so hard that we can't just do a clean install from scratch, remake the 10 users and the 4 shares and just be done.  Can you come down tomorrow since we're going to have join the machines to a new domain? I think the biggest challenge is going to be exporting/importing [the owner's] 8GB PST. I'll have all the users set up and the groups and most of the data copied over, so it'll just be joining the computers to the new domain and importing the mail."

From there, it was a dream -- everything worked just fine.  We had one machine exporting people's PST files for later import.

(WARNING: You lose Single Instance Storage when you do it this way, so your Exchange store will be bigger than it was due to duplication of attachments, etc. This may not be right for you.  YMMV.)

At about 1:30 AM we were ready to move the new server into the copy room where it was going to stay.  The server asked to apply 59 updates, and like idiots, we let it, and that's where it all went to shit.

The reboot happened and then nothing but "Applying Updates, Stage 3 of 3..."

After an hour of staring at that screen, I sent my partner home.  No sense have two of us stuck in graybar land.

After two hours, I started Googling solutions to the problem.  Eventually I was able to start the server and get to safe mode where I could edit some XML files and cross my fingers and wait more.

After another hour, I just left.  It was almost 4 in the morning by that point, so I got a room at a nearby motel (to save myself the 65 minute each way commute to the client office) and crashed hoping to get back onsite before everyone showed up at 8.

As I wrote in my post-mortem email to Brian at 11:20 AM:

Subject: What a fucking mess

Everything works and nothing works,

Ended up staying at the local Days Inn and crossed my fingers that the thing would roll back while I slept, and when I came back at 8:15, we were at CTRL+ALT+DEL so it seems to be OK,m tho it lists all 59 updates as pending. There's a possibility they'll never take. (Fun!)

"Where are my contacts!?!" (nickname) - bring them over and all the nicknames for the internal people are all wrong, so the mail bounces.  NK2Edit to the rescue.

Scanner set up.

Abacus' DRM is driving me up a fucking wall.

Some documents that people were working on last night never made it over -- feels like there wasn't a sync done, and since we generally deleted the offline files from the workstations to eliminate the error we were getting at logoff sync... DOH. Happily, it seems they were able to recreate them with little effort, so it was an inconvenince, but not a show-stopper.

People don't like to use their address books, I find -- they really REALLY rely on their NK2 files -- so when doing any sort of migration, don't overlook them.  NK2Edit is the greatest tool in the world.

I hate DRM.  I think it only hurts legitimate users, pirates get past it with little effort, so it only serves to create issues for paid users at the worst possible times -- like when people need to use their time and billing systems and the support center isn't open for another hour or two...

The document sync issue wasn't something we thought was an issue -- we made sure offline files were all sync'd up, but users reported some documents didn't copy over, and I'm not sure what else to attribute it to.

In the time since we did this, the 59 updates did in fact take and the server didn't fall apart or fail.

Then I see the Windows Update shield, and I open it up much to my horror:

118 UPDATES?!?

So, at 2:05 PM, I clicked "AGREE" on a licensing agreement, and off we go...

I have a thread going over on Facebook...

JK (me): Shall we take bets as to how long this will take to run?

AF: I'll take the over on 5 reboots.

JS:  i am guess it will take 6 hours total. 5 hours and 45 minutes of annoying pop ups. 15 minutes of actual updating

JK: 2:10 - 2 security updates done... 0 reboots.

JK: ‎2:16 - 10 down, 0 reboots

JC: Don't forget Genuine Advantage.

JK: That's always last, isn't it?

JK: 2:36 19 down... 0 reboots.  (Net Framework installs take the longest.)

JK: 2:46 Update 22. Net Framework 1.1 SP1; see you in an hour.

We're relying on Microsoft's instructions to get us through the night.

It's 7:31 pm on a Friday night.  The air conditioning in the office has been turned off, so it's a bit sultry.  I prepped the source server last night -- just made sure its service packs and patches were up to date.  Now that I'm on site, I'm ready to make my answer file.

To do so, we need to install some tools from the SBS 2011 install media (which happily Dell included).

Sadly, the pre-requsites to using the tool don't run on Windows XP, and this is an XP shop.

Great.

However, I've got two Dell Inspiron 560s ready to be dropped down and they have Windows 7 on them, so let me unbox one and get it powered up so I can get the server migration started.

(I had planned on installing the workstations when the data was migrating... looks like I get a jump on it...)

I'll be back once I'm unboxed...

 7:56 - Machine unboxed, Facebook notified of liveblogging, OOBE completed, waiting for my desktop customizations and personal settings to be applied. Backup of source server in progress.

8:22 - Machine joined to domain, Migration Assistant installed and waiting for backup to finish...

8:31 - Backup can take 7 hours.  Uh-oh.  Wait!  The server is running in a VM!  There was a hardware failure which precipated this migration so we brought the old box back up in a VM.  (Pretty slick apart from a couple licewnsing activation gotchas.  "DRM -- Screwing the honest since 1998.") So, I can just take a snapshot and roll back if anything goes horribly wrong! Score! Back on schedule!

8:37 - Migration Assistant found some issues...

9:08 - Issues persist.  Odd ones  that aren't true - the remote registry service IS running, but the migration assistant refuses to believe it...

9:51 - Ran some more hotfixes on the server after SBS BPA suggested it.

10:16 - Re-read some documentation.  Last night, I read that (what I thought was) the migration assistant needed to be run on a workstation, not the server.  That does not seem to the case.  I've spent the last hour chasing down ghosts.

10:20 - Running the MA on the SBS server worked right out of the box.  FML.  Wasted all this time due to my stupid brain.  Stupid, stupid brain.

LESSON 1: Run the Migration Tool ON THE SOURCE SERVER.

10:42 - Answer file created, copied to USB key, USB key in destination server and the install is now running on the destination server.  Right out of the box, the new Dell server runs thru its little preinstall and then dumps you at a screen that says "Basic or Migration Install?"  I choose Migration, it reads the answer file like magic, and we're off to the races.

I created the answer file to not be unattended. I like clicking NEXT when I do my server installs - just to make sure everything is right one last time before committing.  Good thing too, since I misspelled the domain name on the credentials section of the answer file...

Now we wait what may be 30 minutes for Windows to expand its installation files...

10:53 - Server just rebooted itself...

10:54 - Seemed to apply some updates, back to waiting for expanding files... and in the time it took me to type this, it started rebooting again... and we wait some more...

11:02 - Finally, motion!  The bar on the screen started moving across.  I guess all the updates are done... more waiting...

11:03 - I guess it connected to the domain because a printer mapping came up and failed due to an incompatible driver... more waiting...

11:31 - 80% of the way thru the bar.  "Up to 30 minutes," indeed!"  Luckily, the new workstations (before decrapifying) have the WildTangent games on them.  They're usually the first things to go, but right now, I'm glad they're here...

11:43 - 90% of the way thru... when does the 6 hour "migrating data from old server" start?

11:44 - No sooner than I type that, then Explorer starts putting up its "Customizing Browser Customizations..." dialog, and the SBS Setup program quits with an error.  Swell.  (Looks at logs... might it have something to do with time synchronization?!?)

12:12 - Fantastic.  The source server got hung up (a backup started: LESSON TWO - STOP TASK SCHEDULER) and the clock stopped advancing, which caused Kerberos errors, and then the only way to recover is TO START OVER.  So, rebooted both servers, booted off the DVD on the destination server, reformatted the partitions and am now sitting thru the whole thing again.  Sloppy... but this is why we flat-rate server installs... my mistakes are my own and come out of my end, not the client's.

12:41 - We're back at where we were at 10:42.  Lost almost 2 hours ot the minute.  Dang.  Worst part, is I don't have a key to the building, so I'm trapped here.  Can't even go out for a bite and come back...

1:08 - Server install fails with "JoinDomain_DCPROMO Failed..." - turns out the initial failed install got as far as seizing all the roles from the other server and was already in the domain.  So, instead of trying to reverse everything I did, I just reverted to the snapshot from 8:31.

1:22 - Snapshot still had the anti-virus installed and didn't have the migration assistant run, so we're waiting for the post-AV, pre-MA reboot.  *sigh* I'll be pretty good at this when I finally get it done... tho I am ECSTATIC that the server is a VM and all it took was the snapshot to roll it back.  I don't know what I would have done (likely a clean install, then a manual migration of mail and a visit to all workstations to rejoin them to the domain) if I didn't have that capability...

4:19 - Been a while.  Had to give two it more tries.  Both failures.  This last time, I logged in to the domain once as the new user they want you to create to act as the migration account, and maybe that was the charm, because now -- seven hours after we started -- I can proceed to STEP FOUR of the 19 Step Migration document...

8:34 AM - Looked good for a while, then it didn't.  The Migration Wizard on 2011 isn't as automated as one might hope, and in my bleary eyed state, I made some missteps and by the end, nothing seemed right, so I punted.  Old server restored from snapshot and I will take another run at this soon, once I catch up on some sleep and more reading in the migration guide.

However, when I got on-site, there was no malware on the machine. 

A quick glance over the Autoruns turned up nothing out of the ordinary, MalwareBytes turned up nothing, doing a netsh winsock reset did nothing to improve the situation... not feeling like standard malware... was this thing rootkitted?

But then I noticed there was an odd entry in his start menu called "ZoneAlarm!12dc_532f!erased" which had a broken link to a "ZoneAlarm Security Tutorial"

The client does use ZoneAlarm, and prior issues with his machine had been traced back to ZA -- most notably when a Microsoft DNS patch befuddled the firewall, forcing it to block all DNS traffic -- so I didn't rule ZA out of the equation.

Searching all the files on his machine which were created in the past 24 hours, turned up a file in c:\windows\internet logs called fwpktlog.txt (FireWall PacKeT LOG anyone?  And why are we still limiting ourselves to 8.3 naming?  Does anything other than our own laziness still rely on that?)

In this file was a whole bunch of lines denying packets from the local machine to Internet addresses.  I could still access machines on the local network, but nothing outside.

What created that packet filter file?

I looked in the services list and there was nothing for ZoneAlarm in there, but there was a entry for ForceField (ZA's browser security program) and it was running.  Stopping the service and setting it to disabled did nothing.

Open the pod bay doors, Hal! 
I'm afraid I can't do that, Dave.

Also, among the recently created files, I found an installer_02251175403.log file, also in the c:\windows\internet logs folder.  Opening it up, it pointed me to a temp folder it had created: C:\DOCUME~1\Steve\LOCALS~1\Temp\02251175403 within which was a bunch of files including an executable called cpes_clean.exe which is listed as "Check Point Endpoint Security Cleaner" from "Check Point Software Technologies LTD" in the file properties.

Going for broke, I ran the program.

It ate up some CPU according to Task Manager, but didn't show any sort of UI until it popped up a box asking to reboot.

Rebooted the machine, and lo and behold, we could access sites out on the internet.

Pop over to ZoneAlarm's and download the latest build, and 145MB and 15 minutes later (DSL!!!) and the download is corrupt.

Another 15 minute download, and ZA is happily reinstalling itself.

Alarmed Light - Used it because it has a feature where you're forced to answer math questions to turn the alarm off. Kinda forces you to shake the cobwebs out and get started.

Aloqa - Cool app that uses your GPS to let you know what's around. It's integrated with Yelp for food ratings. It has icons on its main screen for "Hot" (whatever that means), Yelp Restaurants, Coffee, last.fm, Music (showing me concerts in Denmark?!?), Playing Tonight (movies), Yelp Bars & Clubs, Real Estate, Wikipedia, ATMs, Pizza, Aloqa, Yelp Fast Food and then "Add more channels." It more or less does what it is supposed to; it's nice to look at.

BeamReader - a PDF viewer. I should uninstall since I bought "Documents to Go"

Bubble Burst Lite -- Windows Mobile Jawbreaker for the Droid.

ConnectFour - decent enough implementation. AI seems a little stupid sometimes.

Documents to Go - open Word, Excel, Powerpoint, PDF. Does good job at rendering PDFs.

Flashlight - turns your screen white. Doesn't seem to adjust for maximum brightness. Passable.

Flickr Droid - Droid needs a good Flickr app. This isn't it, but the best one I could find that uses the Flickr API to let me at my stuff, since a lot of my photostream is friends/family only.

FlightStats Lite - haven't had a chance to play with this. Will in January as I head to CES.

Goggles - Google's latest toy. Varies from wildly successful to "how did you not recognize the Pepsi logo?!?"

GPS Status - essentially a digital compass. Used when I was troubleshooting GPS on the phone.

Flixter Movies - quick and easy to get to where we have to go for Friday Morning Movie Club.

NYC Bus and Subway Map - not as interactive as I'd like. Literally a HiDef graphic of the map, and you can click thru to the MTA website for further details on the lines.

OpenTable - online restaurant reservations from opentable.com . Decent.

Pandora - works well over 3G, tho I imagine it eats thru the quota pretty quick.

PicSay Lite - dopey photo editing thing. Makes speech baloons. Don't know why I downloaded this.

Poke a Mole - whack-a-mole for the phone. Fun game with a Giant Downside - even when phone is muted annoying background music plays. Have to go into game menu to mute it.

Remote RDP Demo - eventually I'll need to really use function keys when I Remote Desktop into a machine from my phone, but until then, the demo version does the trick.

Robo Defense FREE - I do enjoy the tower defense genre of casual gaming... so why not have it on my phone? (Hardly never play it. Seemed like a good idea.)

Shazam - this app still amazes me. Where did they get that song database???

Stopwatch - straightforward.

Sudoku Free - seeing it in my list makes me feel smart until I play it, thenI feel dumb.

The Weather Channel - it has a widget so I can glance at the home screen to see what it's going to be like... or rather I can look at the home screen and tell my wife what it's going to be like... (I had a weather widget on the Treo and missed it.)

TivoRemote - control the Tivo over WiFi. Nice, especially for text entry. The iPhone's version is better.

TRAFFIC! - A test app for me. Not in love.

Trap! - Another game, but its "draw a line" technology gets in the way of gameplay. Fun when it does what you want.

Tunes Remote - Control iTunes from the Droid. YAY! One of the big reasons I got the iPhone touch was to control the iTunes machine hooked to the outdoor speakers.

Twidroid Pro - From what I understand, the best Twitter client for the Droid. I'm happy with it. Does everything I need, but I am far from a Twitter power user, so some might find it lacking. They constantly update it which seems like they're interested developers.

US Traffic - Another traffic app that I tried before I realized Google Maps had a Traffic layer...

wpToGo - Allows me to post to WordPress from the phone... tho I have not had the need to do so.

What have I missed?

I instead told him to use VirtualPC to do the job.

A quick Google later, and I pointed him to these directions as they were as full featured and complete a set as I've seen... and I didn't have to type them myself.

Nice job, Abbas!

When I checked the post-infection System Event Viewer log, however, I got an interesting message:

Event Type:Error
Event Source:Schedule
Event Category:None
Event ID:7901
Date:1/31/2009
Time:9:00:00 PM
User:N/A
Computer:XXX03
Description:
The At46.job command failed to start due to the following error:
The system cannot find the file specified.

Huh? At46.job? I know the machine doesn't use the AT scheduler... let's see...

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.
C:\Documents and Settings\administrator\>at
Status ID Day Time Command Line
-------------------------------------------------------------------------------
 1 Each M T W Th F S Su 12:26 AM C:\WINDOWS\system32\Hi3TR1uq.exe
 10 Each M T W Th F S Su 9:00 AM C:\WINDOWS\system32\Hi3TR1uq.exe
 11 Each M T W Th F S Su 10:00 AM C:\WINDOWS\system32\Hi3TR1uq.exe
 12 Each M T W Th F S Su 11:00 AM C:\WINDOWS\system32\Hi3TR1uq.exe
 13 Each M T W Th F S Su 12:00 PM C:\WINDOWS\system32\Hi3TR1uq.exe
 14 Each M T W Th F S Su 1:00 PM C:\WINDOWS\system32\Hi3TR1uq.exe
 15 Each M T W Th F S Su 2:00 PM C:\WINDOWS\system32\Hi3TR1uq.exe
 16 Each M T W Th F S Su 3:00 PM C:\WINDOWS\system32\Hi3TR1uq.exe
 17 Each M T W Th F S Su 4:00 PM C:\WINDOWS\system32\Hi3TR1uq.exe
 18 Each M T W Th F S Su 5:00 PM C:\WINDOWS\system32\Hi3TR1uq.exe
 19 Each M T W Th F S Su 6:00 PM C:\WINDOWS\system32\Hi3TR1uq.exe
 2 Each M T W Th F S Su 1:00 AM C:\WINDOWS\system32\Hi3TR1uq.exe
 20 Each M T W Th F S Su 7:00 PM C:\WINDOWS\system32\Hi3TR1uq.exe
 21 Each M T W Th F S Su 8:00 PM C:\WINDOWS\system32\Hi3TR1uq.exe
Error 22 Each M T W Th F S Su 9:00 PM C:\WINDOWS\system32\Hi3TR1uq.exe

And so forth, with hourly jobs listed down thru job 72. (It kept adding duplicate schedules...)

You can see we got the infection eradicated before 9 PM, because the 9PM AT job show errors.

For those of you who prefer a GUI, you can see the same thing in the Scheduled Tasks pane in Control Panel

So, don't overlook the AT scheduler as a place where infection might hide in an effort to replicate itself. This is the first time I've seen it there, and it will be a place I look at from here out...

He was getting a 403.6 error -- IP Address rejected.

This didn't make any sense, since we want every IP address to be able to access the site and access to the site was fine earlier in the week.

Even after re-running the CEICW (Configure E-Mail and Internet Connection Wizard) a few times, I couldn't connect to the site from outside the local subnet.

The only thing that had changed recently was our swapping out of their existing router for a new Linksys WRT110.

I've set up enough SBS boxes to know which ports we want to open. So I clicked on the "Applications and Gaming" tab and put in the mappings for ports 25, 80, 110, 143, 443-444, 3389 and 4125.

However, I didn't put them where I thought I did.

When you click on the "Applications and Gaming" tab in a WRT54G router, you're taken to a "Port Range Forwarding" page.

Linksys has been doing itthis way for years.

Not so in the WRT110 series!

Now you're brought to a "Single Port Forwarding" page. It looks kinda similar:

But instead of there being a port RANGE, it's a single port. So when you put 443 in the first box and 444 in the second box of the WRT110, it MAPS 443 to 444, and that causes your SSL authentication to fail and your IP to be rejected. It doesn't work like you think. For that, you need to go here:

And if you were to put 443 and 444 in the boxes, then it all works...

So, the moral of the story is, make sure you're forwarding your ports correctly.

I was working for a client late last night, applying the usual slew of patches Microsoft had given us on Tuesday. I rebooted the server, and when it came back up, it didn't bring Exchange with it.

Much gnashing of ensued. I��ve recovered my fair share of Exchange stores, but this one just wasn��t coming back.

The event viewer was full of errors from the Exchange service:

Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 494

Description:
Information Store (3860) First Storage Group: Database recovery failed with error -1216 because it encountered references to a database, ��x:\pathtoourdatabase\priv1.edb��, which is no longer present. The database was not brought to a Clean Shutdown state before it was removed (or possibly moved or renamed). The database engine will not permit recovery to complete for this instance until the missing database is re-instated. If the database is truly no longer available and no longer required, procedures for recovering from this error are available in the Microsoft Knowledge Base or by following the ��more information�� link at the bottom of this message.

The frustrating part of all of this was the Exchange database was where it should have been (contrary to the above error).

By the time I gave up, it was 4:30 AM and I had hit the wall. I fell asleep on the couch with a laptop next to me, watching BackupExec attempt a restore of the public folder store.

I woke up to BackupExec having failed to restore the backup. (Note to self: make sure the Veritas user is in both the "Backup Operators" and "Administrator" group on SBS 2003. Once we did that, BackupExec did what it should.)

I started an offline backup of the Exchange store and headed to the client site. I knew mail was being spooled on the mail gateway -- the AWESOME spam-killing ESVA Virtual Appliance -- so I wasn��t worried about losing any inbound mail.

In the morning, tho, getting the store to mount was still not happening.

I did a Google search for a firm like mine that specialized in Exchange recovery. The google led me to ExchangeRecovery.org and a great tech named Jon.

The receptionist answered promptly and transferred me over to ��an Exchange specialist.��

Jon was as helpful as a fella could be. I opened a VNC port for him and he ran thru the litany of tests I had done, plus a few more I hadn't. Everything passed the integrity checks. the event viewer had mentioned there were issues in replaying the transaction logs, so he moved the transaction log files out of the MDBDATA directory and that seemed to do the trick. The store came right back up. Since the store was in a clean state when it shutdown, we didn��t lose any mail.

I can��t recommend Jon enough �� he was good humored, professional and a joy to work with. Should we come across any more Exchange issues requiring expert assistance -- or at the least a second set of eyes -- his speed dial is the one I'll be hitting.

On Windows machines, I recommend using a tool like SpaceMonger to profile your hard drive(s) and delete unwanted files, etc. (Note - NEVER DELETE a file unless you know what it is, what it does, and that it's unnecessary or redundant)

On my Mac, I had already analyzed my disk, moved or deleted things that didn't belong on my system drive, but I still didn't have as much free space as I'd like. And that's when I discovered Xslimmer.

Most (90%) of Mac programs are now "Universal Binary" programs, meaning that they can run on older, PowerPC-powered Macs as well as new, Intel-powered machines. Which means (basically) that there's two sets of code on every program that's on your hard drive. Additionally, Mac programs often ship (download) with multiple language packs to support a broader range of users. I only speak English, so I rarely (never) need to run a program in Spanish. Or Dutch. Or whatever they speak in Kazakhstan.

What Xslimmer does is analyzes your applications folder, and strips out the code you don't need. If you have a PowerPC Mac, it will strip out the Intel code from your apps, and vice versa. It also removes unnecessary language packs from your apps. The initial analysis and "slimming" took about an hour (during which time I was still able to work without any memory hit) and it saved me almost 4.5GB of space!

So if you've got some Mac bloat (and lots of applications), Xslimmer might help your Mac fit into its thin jeans again!