We've made the move to Windows 7 and we love it.  However, we haven't really made the move to VB.NET.  I still like developing in VB6; I know that makes me a little bit of a relic, but I don't do that much development these days to justify the investment in fully ramping up on VB.NET.

So, I had to tweak a legacy application I wrote which reads an email from a POP3 mailbox and writes the contents into a database.  The program is maybe 30 lines long, and it's a dream thanks to the w3JMail library and ADO.

I revised the program, ran it on my Windows 7 machine, and all was right in the world.  I went to deploy it back to the Windows 2003 server where it lives, and I was hit with Error 430 errors: "Class does not support Automation or does not support expected interface"

So, after adding line numbers to the code, I was able to track the error down to the line

Set objConn = New ADODB.Connection

That seemed weird.  I tried a bunch of different ADO libraries and nothing.  Then I stumbled upon a MSKB article, with the longest, most specific title I've seen in recent memory: "An ADO application does not run on down-level operating systems after you recompile it on a computer that is running Windows 7 SP 1 or Windows Server 2008 R2 SP 1 or that has KB983246 installed"

Long story short, if you're running Win7 SP1 or Win08R2 SP1, then .NET breaks ADO and you need to register some new type libraries on your local machine and then recompile using those type libraries, NOT the usual ADO libraries.

The KB article shows you how to do it easily enough. Put the files where they tell you.  I had to manually navigate to the folders for some reason, but once I did, they registered up like a charm and my programs ran again.

But Google has come to the rescue with an alternate installer that you can grab from their website: http://support.google.com/chrome/bin/answer.py?hl=en&answer=126299

So, log yoursef on the terminal server as an adminsitrator and download the Alternate installer for all user accounts.

Google is going to freak you out with a big button that says "Accept and Install" after you review the EULA.  DON'T PANIC! You;'ll be allowed to download the installer (ChromeStandaloneSetup.exe) and install it using the Programs app in the Control Panel, as you install all programs on the terminal server.  (You do use that, right?)

However, Windows 2008 Server Backup doesn't like this arrangement.  The wizard wants all targets present at the time of configuration.  Since we only have one dock (and a minimum of two drives), if we try to add additional hard drives to the job, the wizard fails with the error: "The filename, directory name, or volume label syntax is incorrect" and the backup job is not modified.

Happily, Microsoft comes to the rescue with a KnowledgeBase article which outlines the three options:

1. Reattach all disks (not an option in our situation)
2. Delete the missing disk (not an option when there's only one disk configured; you have to delete the whole job... plus, we *want* the missing disk to be part of the job since *all* disks except the one that's connected will be missing)
3. Do some command line voodoo.  (Bingo!)

So, the steps are:

1. Find "Command Prompt" in the start menu and "Run as Administrator" from the right-click menu
2. Get the drive ID of the disk we want to add to the job as we'll need it for the next step:
1. At the command prompt, type wbadmin get disks
2. Each drive attached to the system will be listed; so find the removable disk and copy the long string of numbers (and the brackets) that make up the "Disk Identifier"  Copy this to the clipboard (using Mark, Paste from the Window menu... I use ALT+SPACE, E, M as a keyboard shortcut)
3. At the command prompt, now type WBADMIN ENABLE BACKUP -addtarget:{PASTE-YOUR-DISK-ID-HERE} (I use the keyboard shortcut ALT+SPACE, E, P)
4. You'll get a one or a bunch of prompts (depending on if the drive is formatted for backups or not, if existing backups are present on the disk, etc.) which mimic the questions in the wizard about the disk, answer appropriately.
5. Finally, make sure you answer Y to the question "Do you want to enable scheduled backups with the above settings?"

The script will finish running and eventually leave you back at a command prompt.  You can close the window.

Now, when you look at the Backup screen, you'll see multiple targets in the DESTINATION settings, and the Destination Usage drop down will show the drive that's connected PLUS the disconnected drive.

And we're all set.

TL;DR - Did a clean install. Recreated users, shares and printers.  Export mailboxes to PSTs.  Copied data over.  Reimported mailboxes. Got on with my life.

For the second run at upgrading the server, I had a few new tools at my disposal.

First, was a virtual copy of their server.  Since their server was already virtualized, I just stopped VMWare, made a copy of the VM and took it back to my office.  I loaded it up on a little Vostro 200 I had laying around (the server was amazingly peppy, all things considered) and then figured I'd do 99% of the migration off-site and then drop the box in and copy what was left of the data and mailboxes over.

I made the smart move and took GHOST images of the new box at every juncture, so I could "easily" roll back.  (Granted, the roll back takes 15 minutes, but it's waaaaay better than having to sit thru the unpacking of files every time something goes wrong.)

I will say it took an inordinate amount of time to get the right drivers to get the Dell T310 work with Sysinternals' ERD Commander 5.0 (Microsoft version; not the pre-buyout version with Firefox in it...) but it was worth it.

Let's say I rolled back more than a few times.  Learning a little more each time; or sometimes just doing everything identically, but having it work the second time around.

This guide came in handy: http://blog.mpecsinc.ca/2009/06/sbs-2003-to-sbs-2008-migration-guide.html

After doing this dance for a couple days, it got to a point where I was fed up with the roadblocks and things not working right (like migrated uses not showing up in the users applet).  From an email I sent to my partner at midnight the night before we were supposed to be on site, where I make the call to punt:

"OK, I've had it. Every time I think I got it, there's some fucking error somewhere. I assume its in the old box somewhere -- a remnant of the P2V or something. So, fuck the migration. Their environment isn't so hard that we can't just do a clean install from scratch, remake the 10 users and the 4 shares and just be done.  Can you come down tomorrow since we're going to have join the machines to a new domain? I think the biggest challenge is going to be exporting/importing [the owner's] 8GB PST. I'll have all the users set up and the groups and most of the data copied over, so it'll just be joining the computers to the new domain and importing the mail."

From there, it was a dream -- everything worked just fine.  We had one machine exporting people's PST files for later import.

(WARNING: You lose Single Instance Storage when you do it this way, so your Exchange store will be bigger than it was due to duplication of attachments, etc. This may not be right for you.  YMMV.)

At about 1:30 AM we were ready to move the new server into the copy room where it was going to stay.  The server asked to apply 59 updates, and like idiots, we let it, and that's where it all went to shit.

The reboot happened and then nothing but "Applying Updates, Stage 3 of 3..."

After an hour of staring at that screen, I sent my partner home.  No sense have two of us stuck in graybar land.

After two hours, I started Googling solutions to the problem.  Eventually I was able to start the server and get to safe mode where I could edit some XML files and cross my fingers and wait more.

After another hour, I just left.  It was almost 4 in the morning by that point, so I got a room at a nearby motel (to save myself the 65 minute each way commute to the client office) and crashed hoping to get back onsite before everyone showed up at 8.

As I wrote in my post-mortem email to Brian at 11:20 AM:

Subject: What a fucking mess

Everything works and nothing works,

Ended up staying at the local Days Inn and crossed my fingers that the thing would roll back while I slept, and when I came back at 8:15, we were at CTRL+ALT+DEL so it seems to be OK,m tho it lists all 59 updates as pending. There's a possibility they'll never take. (Fun!)

"Where are my contacts!?!" (nickname) - bring them over and all the nicknames for the internal people are all wrong, so the mail bounces.  NK2Edit to the rescue.

Scanner set up.

Abacus' DRM is driving me up a fucking wall.

Some documents that people were working on last night never made it over -- feels like there wasn't a sync done, and since we generally deleted the offline files from the workstations to eliminate the error we were getting at logoff sync... DOH. Happily, it seems they were able to recreate them with little effort, so it was an inconvenince, but not a show-stopper.

People don't like to use their address books, I find -- they really REALLY rely on their NK2 files -- so when doing any sort of migration, don't overlook them.  NK2Edit is the greatest tool in the world.

I hate DRM.  I think it only hurts legitimate users, pirates get past it with little effort, so it only serves to create issues for paid users at the worst possible times -- like when people need to use their time and billing systems and the support center isn't open for another hour or two...

The document sync issue wasn't something we thought was an issue -- we made sure offline files were all sync'd up, but users reported some documents didn't copy over, and I'm not sure what else to attribute it to.

In the time since we did this, the 59 updates did in fact take and the server didn't fall apart or fail.

Then I see the Windows Update shield, and I open it up much to my horror:

118 UPDATES?!?

So, at 2:05 PM, I clicked "AGREE" on a licensing agreement, and off we go...

I have a thread going over on Facebook...

JK (me): Shall we take bets as to how long this will take to run?

AF: I'll take the over on 5 reboots.

JS:  i am guess it will take 6 hours total. 5 hours and 45 minutes of annoying pop ups. 15 minutes of actual updating

JK: 2:10 - 2 security updates done... 0 reboots.

JK: ‎2:16 - 10 down, 0 reboots

JC: Don't forget Genuine Advantage.

JK: That's always last, isn't it?

JK: 2:36 19 down... 0 reboots.  (Net Framework installs take the longest.)

JK: 2:46 Update 22. Net Framework 1.1 SP1; see you in an hour.

We're relying on Microsoft's instructions to get us through the night.

It's 7:31 pm on a Friday night.  The air conditioning in the office has been turned off, so it's a bit sultry.  I prepped the source server last night -- just made sure its service packs and patches were up to date.  Now that I'm on site, I'm ready to make my answer file.

To do so, we need to install some tools from the SBS 2011 install media (which happily Dell included).

Sadly, the pre-requsites to using the tool don't run on Windows XP, and this is an XP shop.

Great.

However, I've got two Dell Inspiron 560s ready to be dropped down and they have Windows 7 on them, so let me unbox one and get it powered up so I can get the server migration started.

(I had planned on installing the workstations when the data was migrating... looks like I get a jump on it...)

I'll be back once I'm unboxed...

 7:56 - Machine unboxed, Facebook notified of liveblogging, OOBE completed, waiting for my desktop customizations and personal settings to be applied. Backup of source server in progress.

8:22 - Machine joined to domain, Migration Assistant installed and waiting for backup to finish...

8:31 - Backup can take 7 hours.  Uh-oh.  Wait!  The server is running in a VM!  There was a hardware failure which precipated this migration so we brought the old box back up in a VM.  (Pretty slick apart from a couple licewnsing activation gotchas.  "DRM -- Screwing the honest since 1998.") So, I can just take a snapshot and roll back if anything goes horribly wrong! Score! Back on schedule!

8:37 - Migration Assistant found some issues...

9:08 - Issues persist.  Odd ones  that aren't true - the remote registry service IS running, but the migration assistant refuses to believe it...

9:51 - Ran some more hotfixes on the server after SBS BPA suggested it.

10:16 - Re-read some documentation.  Last night, I read that (what I thought was) the migration assistant needed to be run on a workstation, not the server.  That does not seem to the case.  I've spent the last hour chasing down ghosts.

10:20 - Running the MA on the SBS server worked right out of the box.  FML.  Wasted all this time due to my stupid brain.  Stupid, stupid brain.

LESSON 1: Run the Migration Tool ON THE SOURCE SERVER.

10:42 - Answer file created, copied to USB key, USB key in destination server and the install is now running on the destination server.  Right out of the box, the new Dell server runs thru its little preinstall and then dumps you at a screen that says "Basic or Migration Install?"  I choose Migration, it reads the answer file like magic, and we're off to the races.

I created the answer file to not be unattended. I like clicking NEXT when I do my server installs - just to make sure everything is right one last time before committing.  Good thing too, since I misspelled the domain name on the credentials section of the answer file...

Now we wait what may be 30 minutes for Windows to expand its installation files...

10:53 - Server just rebooted itself...

10:54 - Seemed to apply some updates, back to waiting for expanding files... and in the time it took me to type this, it started rebooting again... and we wait some more...

11:02 - Finally, motion!  The bar on the screen started moving across.  I guess all the updates are done... more waiting...

11:03 - I guess it connected to the domain because a printer mapping came up and failed due to an incompatible driver... more waiting...

11:31 - 80% of the way thru the bar.  "Up to 30 minutes," indeed!"  Luckily, the new workstations (before decrapifying) have the WildTangent games on them.  They're usually the first things to go, but right now, I'm glad they're here...

11:43 - 90% of the way thru... when does the 6 hour "migrating data from old server" start?

11:44 - No sooner than I type that, then Explorer starts putting up its "Customizing Browser Customizations..." dialog, and the SBS Setup program quits with an error.  Swell.  (Looks at logs... might it have something to do with time synchronization?!?)

12:12 - Fantastic.  The source server got hung up (a backup started: LESSON TWO - STOP TASK SCHEDULER) and the clock stopped advancing, which caused Kerberos errors, and then the only way to recover is TO START OVER.  So, rebooted both servers, booted off the DVD on the destination server, reformatted the partitions and am now sitting thru the whole thing again.  Sloppy... but this is why we flat-rate server installs... my mistakes are my own and come out of my end, not the client's.

12:41 - We're back at where we were at 10:42.  Lost almost 2 hours ot the minute.  Dang.  Worst part, is I don't have a key to the building, so I'm trapped here.  Can't even go out for a bite and come back...

1:08 - Server install fails with "JoinDomain_DCPROMO Failed..." - turns out the initial failed install got as far as seizing all the roles from the other server and was already in the domain.  So, instead of trying to reverse everything I did, I just reverted to the snapshot from 8:31.

1:22 - Snapshot still had the anti-virus installed and didn't have the migration assistant run, so we're waiting for the post-AV, pre-MA reboot.  *sigh* I'll be pretty good at this when I finally get it done... tho I am ECSTATIC that the server is a VM and all it took was the snapshot to roll it back.  I don't know what I would have done (likely a clean install, then a manual migration of mail and a visit to all workstations to rejoin them to the domain) if I didn't have that capability...

4:19 - Been a while.  Had to give two it more tries.  Both failures.  This last time, I logged in to the domain once as the new user they want you to create to act as the migration account, and maybe that was the charm, because now -- seven hours after we started -- I can proceed to STEP FOUR of the 19 Step Migration document...

8:34 AM - Looked good for a while, then it didn't.  The Migration Wizard on 2011 isn't as automated as one might hope, and in my bleary eyed state, I made some missteps and by the end, nothing seemed right, so I punted.  Old server restored from snapshot and I will take another run at this soon, once I catch up on some sleep and more reading in the migration guide.

We had an in-house project that needed SQL Server, so we installed SQL Server 2008 R2 Enterprise in Evaluation mode, since we didn't have the product key handy.

Install went fine, server works great, everyone is happy.

170 or so days go by and we run an SSIS package and notice the warning "Evaluation Version, will expire in 8 days"  Wha?  Do a quick Select @@Version and sure enough, we're still running the evaluation version... in a production environment.  (Facepalm!)

Searching around for how to turn my eval version into a licensed version (we finally got around to requesting the key from MS) showed all sorts of solutions, none of them seemed to really fit the bill.

Some people admonished "You're using Eval software in a  production environment!?? Serves you right!  Uninstall/reinstall!" others required odd registry hacks to make the setup program run -- it all seemed so complex for what should be a pretty simple process.

Lo and behold, buried at the very end of the comments attached to one of these complex blog posts, was the simplest of solutions from a chap named Waleed Al-Qudah:

"Go to Microsoft SQL Server Configuration Tools, and navigate to SQL Server installation Center then click the Maintinance link and choose Edition Upgrade."

Two minutes later, we were all legal.

The SQL Server Installation Center is a fairly complicated looking piece of software, and I never noticed that tab nor that option.  Needless to say, I spent a few more minutes looking through it.

For English based systems, the MSMAPI32.DLL file is living in %programfiles%\Common Files\system\MSMAPI\1033 folder.  Other locales will be in a numbered folder other than 1033.

We ran up against this when a client got a computer with a 60 Day Trial of Office 2007 on it, but installed their corporate version of Office 2003 over it.  Uninstalling Office 2007 presented the error.

He sent:

More failures. This was to our accountant and my fiancée. It took two days to receive this failure notice. I will forward the originals.  Have you figured out what is going on or how to fix this?

I replied:

The NDR comes back in 48 hours because the mail server attempts to connect to the other server for 2 days.

Email was never designed to be 100% timely, it was designed to deliver mail come hell or high water; so if there’s a problem with the server you’re trying to send to, our server tries and tries again, hoping the problem is fixed within 48 hours.  If it is not, our server stops trying, rejects the message and lets you know.

4.4.7 errors usually indicate some issue with the recipient’s server (from: http://support.microsoft.com/kb/284204):

Numeric Code: 4.4.7

Possible Cause: The message in the queue has expired. The sending server tried to relay or deliver the message, but the action was not completed before the message expiration time occurred. This NDR may also indicate that a message header limit has been reached on a remote server or that some other protocol timeout occurred during communication with the remote server.

Troubleshooting: This code typically indicates an issue on the receiving server. Verify the validity of the recipient address, and verify that the receiving server is configured to receive messages correctly. You may have to reduce the number of recipients in the header of the message for the host that you are receiving this NDR from. If you resend the message, it is placed in the queue again. If the receiving server is on line, the message is delivered.

Problem was, I didn't see where the recipient’s server was rejecting any emails in the logs; nor could I see why the message was getting hung up.

So, using mails to sent to his fiancee on 2/24 as a test, I found that all mails sent to her were successfully delivered EXCEPT the calendar invites / meeting requests.  So, I had to question why  that was.  Obviously, sending to that mail server wasn't the issue; the user was not blacklisted or anything, but something was getting hung up.

The SMTP logs didn't show anything out of the ordinary, so I figured the items were never even making it into the queue.  Using Message Tracking, we were able to verify that it never hit the queue.  It got stuck in "Message Routed and Queued For Remote Delivery"

And then two days later, the NDR was generated.

So, further investigation made me ask this question:  Are the people bouncing messages using Exchange 2007? I couldn't tell from one of the servers (custom SMTP banner) if it was even running Exchange, but the other said "220+domain.local+Microsoft+ESMTP+MAIL+Service+Version:+2.0 0 0 71 0 47 SMTP."

What made me ask this question was this note from Microsoft we turned up after some research:

Consider the following scenario. A Microsoft Exchange Server 2003 organization and a Microsoft Exchange Server 2007 organization exchange communications by using SMTP. An Exchange 2003 user organizes a meeting and then sends a meeting request to an Exchange 2007 user. Additionally, the Exchange 2007 user accepts the meeting request. Then, the meeting organizer uses Microsoft Office Outlook to send a meeting update message or a meeting cancellation message to the Exchange 2007 user.

In this scenario, the meeting update message or the meeting cancellation message is not delivered to the Exchange 2007 user. Instead, the meeting update message or the meeting cancellation message goes into the SMTP retry queue. If an administrator tries to open the message in the Exchange System Manager console, the administrator may receive the following error message: […]

After some time, the sender of the meeting update may receive the following NDR:

Your message did not reach some or all of the intended recipients.
Subject: test message
Sent: 8/20/2008 11:44 AM
The following recipient(s) could not be reached: user@domain.com on 8/20/2008 11:34 PM
Could not deliver the message in the time limit specified. Please retry or contact your administrator. <server.domain.com #4.4.7>

So, it looked promising -- the article mentioned both calendar invites and error 4.4.7 along with the message getting stuck in the queue, but the initial scenario is not quite correct (it seems to assume the Exchange 2007 recipient got the request and accepted it; something that’s not happening here).

The Microsoft article mentioned a hotfix, which we downloaded and applied.

After the hotfix was applied, I sent a test calendar entry to the client and his fiancee (whose address which was giving us a hard time) and lo and behold, the invite went through:

(In playing with the Message Tracker, I noticed the time was being reported an hour fast.  Turns out, there's a hotfix for that, too.)

However, when I got on-site, there was no malware on the machine. 

A quick glance over the Autoruns turned up nothing out of the ordinary, MalwareBytes turned up nothing, doing a netsh winsock reset did nothing to improve the situation... not feeling like standard malware... was this thing rootkitted?

But then I noticed there was an odd entry in his start menu called "ZoneAlarm!12dc_532f!erased" which had a broken link to a "ZoneAlarm Security Tutorial"

The client does use ZoneAlarm, and prior issues with his machine had been traced back to ZA -- most notably when a Microsoft DNS patch befuddled the firewall, forcing it to block all DNS traffic -- so I didn't rule ZA out of the equation.

Searching all the files on his machine which were created in the past 24 hours, turned up a file in c:\windows\internet logs called fwpktlog.txt (FireWall PacKeT LOG anyone?  And why are we still limiting ourselves to 8.3 naming?  Does anything other than our own laziness still rely on that?)

In this file was a whole bunch of lines denying packets from the local machine to Internet addresses.  I could still access machines on the local network, but nothing outside.

What created that packet filter file?

I looked in the services list and there was nothing for ZoneAlarm in there, but there was a entry for ForceField (ZA's browser security program) and it was running.  Stopping the service and setting it to disabled did nothing.

Open the pod bay doors, Hal! 
I'm afraid I can't do that, Dave.

Also, among the recently created files, I found an installer_02251175403.log file, also in the c:\windows\internet logs folder.  Opening it up, it pointed me to a temp folder it had created: C:\DOCUME~1\Steve\LOCALS~1\Temp\02251175403 within which was a bunch of files including an executable called cpes_clean.exe which is listed as "Check Point Endpoint Security Cleaner" from "Check Point Software Technologies LTD" in the file properties.

Going for broke, I ran the program.

It ate up some CPU according to Task Manager, but didn't show any sort of UI until it popped up a box asking to reboot.

Rebooted the machine, and lo and behold, we could access sites out on the internet.

Pop over to ZoneAlarm's and download the latest build, and 145MB and 15 minutes later (DSL!!!) and the download is corrupt.

Another 15 minute download, and ZA is happily reinstalling itself.