I was working for a client late last night, applying the usual slew of patches Microsoft had given us on Tuesday. I rebooted the server, and when it came back up, it didn’t bring Exchange with it.
Much gnashing of ensued. I€™ve recovered my fair share of Exchange stores, but this one just wasn€™t coming back.
The event viewer was full of errors from the Exchange service:
Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 494
Description:
Information Store (3860) First Storage Group: Database recovery failed with error -1216 because it encountered references to a database, €˜x:\pathtoourdatabase\priv1.edb€™, which is no longer present. The database was not brought to a Clean Shutdown state before it was removed (or possibly moved or renamed). The database engine will not permit recovery to complete for this instance until the missing database is re-instated. If the database is truly no longer available and no longer required, procedures for recovering from this error are available in the Microsoft Knowledge Base or by following the €œmore information€ link at the bottom of this message.
The frustrating part of all of this was the Exchange database was where it should have been (contrary to the above error).
By the time I gave up, it was 4:30 AM and I had hit the wall. I fell asleep on the couch with a laptop next to me, watching BackupExec attempt a restore of the public folder store.
I woke up to BackupExec having failed to restore the backup. (Note to self: make sure the Veritas user is in both the “Backup Operators” and “Administrator” group on SBS 2003. Once we did that, BackupExec did what it should.)
I started an offline backup of the Exchange store and headed to the client site. I knew mail was being spooled on the mail gateway — the AWESOME spam-killing ESVA Virtual Appliance — so I wasn€™t worried about losing any inbound mail.
In the morning, tho, getting the store to mount was still not happening.
I did a Google search for a firm like mine that specialized in Exchange recovery. The google led me to ExchangeRecovery.org and a great tech named Jon.
The receptionist answered promptly and transferred me over to €œan Exchange specialist.€
Jon was as helpful as a fella could be. I opened a VNC port for him and he ran thru the litany of tests I had done, plus a few more I hadn’t. Everything passed the integrity checks. the event viewer had mentioned there were issues in replaying the transaction logs, so he moved the transaction log files out of the MDBDATA directory and that seemed to do the trick. The store came right back up. Since the store was in a clean state when it shutdown, we didn€™t lose any mail.
I can€™t recommend Jon enough €” he was good humored, professional and a joy to work with. Should we come across any more Exchange issues requiring expert assistance — or at the least a second set of eyes — his speed dial is the one I’ll be hitting.