I've been working with a client who has been reporting emails that haven't been going through to some recipients.  He was getting frustrated because he couldn't reliably send to personal or professional contacts.

He sent:

More failures. This was to our accountant and my fiancée. It took two days to receive this failure notice. I will forward the originals.  Have you figured out what is going on or how to fix this?

I replied:

The NDR comes back in 48 hours because the mail server attempts to connect to the other server for 2 days.

Email was never designed to be 100% timely, it was designed to deliver mail come hell or high water; so if there’s a problem with the server you’re trying to send to, our server tries and tries again, hoping the problem is fixed within 48 hours.  If it is not, our server stops trying, rejects the message and lets you know.

4.4.7 errors usually indicate some issue with the recipient’s server (from: http://support.microsoft.com/kb/284204):

Numeric Code: 4.4.7

Possible Cause: The message in the queue has expired. The sending server tried to relay or deliver the message, but the action was not completed before the message expiration time occurred. This NDR may also indicate that a message header limit has been reached on a remote server or that some other protocol timeout occurred during communication with the remote server.

Troubleshooting: This code typically indicates an issue on the receiving server. Verify the validity of the recipient address, and verify that the receiving server is configured to receive messages correctly. You may have to reduce the number of recipients in the header of the message for the host that you are receiving this NDR from. If you resend the message, it is placed in the queue again. If the receiving server is on line, the message is delivered.

Problem was, I didn't see where the recipient’s server was rejecting any emails in the logs; nor could I see why the message was getting hung up.

So, using mails to sent to his fiancee on 2/24 as a test, I found that all mails sent to her were successfully delivered EXCEPT the calendar invites / meeting requests.  So, I had to question why  that was.  Obviously, sending to that mail server wasn't the issue; the user was not blacklisted or anything, but something was getting hung up.

The SMTP logs didn't show anything out of the ordinary, so I figured the items were never even making it into the queue.  Using Message Tracking, we were able to verify that it never hit the queue.  It got stuck in "Message Routed and Queued For Remote Delivery"

And then two days later, the NDR was generated.

So, further investigation made me ask this question:  Are the people bouncing messages using Exchange 2007? I couldn't tell from one of the servers (custom SMTP banner) if it was even running Exchange, but the other said "220+domain.local+Microsoft+ESMTP+MAIL+Service+Version:+2.0 0 0 71 0 47 SMTP."

What made me ask this question was this note from Microsoft we turned up after some research:

Consider the following scenario. A Microsoft Exchange Server 2003 organization and a Microsoft Exchange Server 2007 organization exchange communications by using SMTP. An Exchange 2003 user organizes a meeting and then sends a meeting request to an Exchange 2007 user. Additionally, the Exchange 2007 user accepts the meeting request. Then, the meeting organizer uses Microsoft Office Outlook to send a meeting update message or a meeting cancellation message to the Exchange 2007 user.

In this scenario, the meeting update message or the meeting cancellation message is not delivered to the Exchange 2007 user. Instead, the meeting update message or the meeting cancellation message goes into the SMTP retry queue. If an administrator tries to open the message in the Exchange System Manager console, the administrator may receive the following error message: […]

After some time, the sender of the meeting update may receive the following NDR:

Your message did not reach some or all of the intended recipients.
Subject: test message
Sent: 8/20/2008 11:44 AM
The following recipient(s) could not be reached: user@domain.com on 8/20/2008 11:34 PM
Could not deliver the message in the time limit specified. Please retry or contact your administrator. <server.domain.com #4.4.7>

So, it looked promising -- the article mentioned both calendar invites and error 4.4.7 along with the message getting stuck in the queue, but the initial scenario is not quite correct (it seems to assume the Exchange 2007 recipient got the request and accepted it; something that’s not happening here).

The Microsoft article mentioned a hotfix, which we downloaded and applied.

After the hotfix was applied, I sent a test calendar entry to the client and his fiancee (whose address which was giving us a hard time) and lo and behold, the invite went through:

(In playing with the Message Tracker, I noticed the time was being reported an hour fast.  Turns out, there's a hotfix for that, too.)

A client called this morning saying "I clicked on a security alert and now I can't get on the internet..."  Of course, my spideysense said "Ah!  Classic Malware!"

However, when I got on-site, there was no malware on the machine. 

A quick glance over the Autoruns turned up nothing out of the ordinary, MalwareBytes turned up nothing, doing a netsh winsock reset did nothing to improve the situation... not feeling like standard malware... was this thing rootkitted?

But then I noticed there was an odd entry in his start menu called "ZoneAlarm!12dc_532f!erased" which had a broken link to a "ZoneAlarm Security Tutorial"

The client does use ZoneAlarm, and prior issues with his machine had been traced back to ZA -- most notably when a Microsoft DNS patch befuddled the firewall, forcing it to block all DNS traffic -- so I didn't rule ZA out of the equation.

Searching all the files on his machine which were created in the past 24 hours, turned up a file in c:\windows\internet logs called fwpktlog.txt (FireWall PacKeT LOG anyone?  And why are we still limiting ourselves to 8.3 naming?  Does anything other than our own laziness still rely on that?)

In this file was a whole bunch of lines denying packets from the local machine to Internet addresses.  I could still access machines on the local network, but nothing outside.

What created that packet filter file?

I looked in the services list and there was nothing for ZoneAlarm in there, but there was a entry for ForceField (ZA's browser security program) and it was running.  Stopping the service and setting it to disabled did nothing.

Open the pod bay doors, Hal! 
I'm afraid I can't do that, Dave.

Also, among the recently created files, I found an installer_02251175403.log file, also in the c:\windows\internet logs folder.  Opening it up, it pointed me to a temp folder it had created: C:\DOCUME~1\Steve\LOCALS~1\Temp\02251175403 within which was a bunch of files including an executable called cpes_clean.exe which is listed as "Check Point Endpoint Security Cleaner" from "Check Point Software Technologies LTD" in the file properties.

Going for broke, I ran the program.

It ate up some CPU according to Task Manager, but didn't show any sort of UI until it popped up a box asking to reboot.

Rebooted the machine, and lo and behold, we could access sites out on the internet.

Pop over to ZoneAlarm's and download the latest build, and 145MB and 15 minutes later (DSL!!!) and the download is corrupt.

Another 15 minute download, and ZA is happily reinstalling itself.

For a while, Dell was shipping their Small Business Servers with a 12GB System partition.  While that may have seemed like a lot of space at one time, it's not amymore, and we're seeing clients' servers getting stressed out as they're pushing the limits of the partition size.

Upon setting up the server, moving the USERS share was always the first order of business.  Even when 12GB seemed roomy, it was obvious 15 users was going to eat their way through the share space in nothing flat, and a move to the data partition was in order.

The Health & Monitoring server was another space hog, with its database growing out of control until it expanded, like a gas, to fill all available space.  So, a quick reinitialization of the database clears up some space...

But a lot of these things are quick fixes, but there's a lot of them... so it was very nice of Microsoft to bundle all of them into a single document: Moving Data Folders for Windows Small Business Server 2003

The 600k Word Document is a tremendous little cookbook! It covers just about everything:

• Step 1: Complete and Verify a Full Backup
• Step 2: Notify Users that Resources will be Unavailable
• Step 3: Move the Users Shared Folders
• Step 4: Move the SharePoint Databases
• Step 5: Move the Monitoring Database
• Step 6: Move Exchange Databases and Log Files
• Step 7: Move the Sent Faxes Folder
• Step 8: Move the ClientApps Shared Folder

Thanks, SBS Team!

Related Links 18

Recently installed SQL Server 2008 on a Windows 2008 box and was happily adminsitering it from the console of the local server.  When I fired up SSMS (SQL Server Management Studio) from my development machine, it wouldn't connect.

I turned the firewall off  and still couldn't connect.

Turns out TCP/IP isn't turned on by default.  Maybe I knew that at one time, but this time it totally slipped  my mind.

To turn it on, go into SQL Server Configuration Manager, expand "SQL Server Network Configuration" and change "Disabled" to "Enabled" by right-clicking on "TCP/IP" and suddenly, I could connect from my development machine.

This article first appeared at the now-closed Win32Scripting site. Since I find it useful, and my bookmark no longer works, I have been unable to reach its author, Jeff Trumbull, so I hope he doesn't mind that I've archived it.

Author: Jeff Trumbull

Description:
Backup files that make a Microsoft virtual server with only about 1 minute of down time. Suspends the virtual server, takes a shadow copy , starts the virtual server then copies virtual server files. This could be used to copy any open files. Requires vshadow.exe from vss sdk.

Script: