ATTENTION: Recent E-Mail Sending (SMTP) Issues
Updates are in blog style, newest first. To read about the issue from start to finish, start at the bottom.

Update 12-08-06 at 8:45am EST
Just a quick link to read more about Verizon and their silly mail policy.

Update 12-07-06 at 10:25am EST
All WSO servers now have IP based spam filtering in place. This will not eliminate all spam, but it will reduce it significantly. But we need your feedback. How it is working for you? Are you getting less spam? Are people you regularly communicate with able to send you mail (they should be able to)? Have you noticed anything else about the e-mail system that concerns you or makes you think it might not be working correctly? We'd love to hear from you.

We are still not done implementing all of the features yet. One of the final steps is to completely whitelist any IP address that passes the POP-Before-SMTP system we have in place. POP-Before-SMTP's basic idea is that if you can successfully log into a WSO server to check your mail, then you should also be allowed to send mail since you are now a known and trusted user. This is accomplished by storing your current IP address in a database and then allowing that IP address to send mail through our mail servers for a predetermined period of time.

But what happens if your IP address is mistakenly in a spam blacklist because it was just recently assigned to you and the person who had it before you was a spammer? This does happen. So now there is a conflict. POP-Before-SMTP says you are a legitimate user and you should be allowed to send mail through the system, but then our blacklist checks override that and block you anyway.

To fix this, we want to automatically and completely whitelist any IP address that passes the POP-Before-SMTP test, irregardless of the IP's status in any of the spam blacklists we use. The only downside here is that if one of our users really is a spammer, but that is a very remote possibility because we keep such high standards for our clients (seriously). Look for more updates on this implementation soon.

Update 12-05-06 at 10:15am EST
We are going to be putting the final touches on our newly enhanced mail processing system today. So far it has been working well but is currently not deployed across all WSO servers and does not have all the features in place. We should be able to knock most of them out today and continue tweaking as needed over the next few days.

Update 12-03-06 at 10:00pm EST
Verizon finally did respond to our concerns via e-mail and here is what they said.

Verizon Online does a sender verification call back to verify that the from address used to send the message is valid. It does this by connecting back to the MX server of the sending domain. In this instance your MX server is rejecting email from our servers and therefore causing the test to fail. Please ensure that the following IP range is not being blocked: 206.46.252.137-206.46.252.164. Also note UCEProtect has these servers listed for sending spam email. These servers do not have the functionality to send email they can only run the above described test.

So, if we take Verizon at their word it would seem UCEProtect either has made incorrect assumptions about these Verizon IP's or they simply don't like Verizon or this Sender Verification policy. In either case that would seriously jeopardize the integrity of UCEProtect and for that reason we are choosing to drop the use of their blacklist.

Update 12-01-06 at 10:20am EST
A common customer of both WSO and Verizon filled out a request form dealing with this spam and blocking issue and the automated response from Verizon said: "In order to resolve your issue, Verizon Online Security has attempted to contact the company or domain for which you were having problems receiving email." Yet, we have never been contacted by Verizon and we are unable to contact them directly because they have no phone number. A call to Verizon's tech support people doesn't help either as they claim to be unable to connect you to the Online Security department as well.

Through some trial and error it was determined that only the UCEPROTECT blocking list contained the IP's that Verizon wants us to keep open. So, for the time being, we will stop using UCEPROTECT just so that our customers can continue to send mail to Verizon owned e-mail addresses. Hopefully Verizon will fix the problem completely and stop placing demands on companies like us to allow known spammer IP's to remain unblocked.

Update 11-30-06 at 4:00pm EST
More on the Verizon issue. Verizon has a policy in place where they check the FROM address on every piece of mail sent to one of their customers to make sure it is a valid address. This is an attempt to cut down on spammers who use bogus e-mail addresses and it is called "Sender Address Verification." Nice idea, but bad in reality. First, it creates undue strain on the mail system, Verizon has a LOT of customers that include many other companies than just verizon.net, such as bellatlantic.net. Second, for Verizon to do this e-mail check, they must be able to connect back to the mail server that sent the e-mail, and here is where it gets interesting.

Verizon states that if you want to be allowed to send mail to their users, you must allow connections back to your mail server from a set of IP addresses they own, which are 206.46.252.0/24. Funny thing is, lots of the IP's in that range are sources of spam themselves!!

See for yourself, go to http://www.uceprotect.net/en/rblcheck.php?ipr=206.46.252.1 and type in the code in the black box and press the Check IP button. Read about how there are so many IP's in that range that have been reported as spamming that they've included the entire range in their blacklist. These are the same IP's Verizon is demanding we allow connect to our mail servers. Interesting isn't it?

Here's a little more about Verizon. They are listed as the #1 Worst Spam Service ISP in the world.

Why would Verizon even allow spammers on their network in the first place? And then why would they shove it down our throats by forcing us to accept their spam laden IP's just so our customers can send e-mail to Verizon customers? This is insane crazy!

In an attempt to verify the behavior documented above we temporarily removed all spam filtering based on IP address. We then sent a test message to a Verizon customer using an e-mail address that had not been used while our spam filtering was in place. Guess what? It went through just fine. Unfortunately, the e-mail address that was blocked before is still blocked, even after our filtering was removed. We suspect this is because they do not do a Sender Address Verification on addresses which have already been checked recently. Probably in a few hours the address will begin to work again as long as we keep our filtering turned off.

Solution? Not sure yet. We are waiting for a response from Verizon after we sent them all of the same info we've written about above. Their reply should be interesting. In the meantime we are going to try and put our filtering back in place but allow the set of IP's that Verizon is requesting remain open just so that our users aren't inconvenienced. Ultimately Verizon needs to stop these spammers or change their verification methods. They shouldn't force quality, reputable services like ours to accept more spam just so our customers can communicate with their customers.

Update 11-30-06 at 11:45am EST
A problem has been identified sending mail to Verizon customers. Verizon returns e-mails with an error message similar to this: "reason: 550 You are not allowed to send mail:sv28pub.verizon.net". We've checked with Verizon and our e-mail servers are NOT blocked. So we've moved on to a second phase of testing. Updates to follow...

Update 11-30-06 at 9:00am EST
Things started really heating up again this morning. The mail server load was high again and there was some concern. Fortunately a first guess about what might be wrong did the trick. A feature that we are going to implement, but haven't yet, was turned on in our SMTP software configuration file and was causing undue load on the machine. Once turned off, the load was cut by more than 50%.

As a precaution, we rechecked our mail servers for Open Relays using ORDB.org and they were found to be completely clean. Basically it means we have proper protections in place to stop spammers from hijacking our machines and sending mail from them directly.

Update 11-29-06 at 11:45pm EST
Things are looking much better. We've gotten some reasonable spam filtering in place and this has cut down on the spam and the server load it was causing by quite a bit. Of course there are several things more to do, but for now, things are looking much better. Here is some more info about the current filtering.

We are currently using four "blacklist" services to filter incoming mail and immediately reject it. Please visit each web site to learn more about these services. Also, if you are getting mail returned to you or an error message in your e-mail program when you try to send mail, look closely and find which blacklist is responsible. Then you can go to that web site and research more. If you find your IP address is blacklisted you should immediately contact your ISP (the company though which you buy your Internet connection to your home or office) and ask to speak to the highest level tech person available.

List Name: SPAMHAUS SBL & XBL (Spam+Exploits)
WSO E-Mail Reject Tag: SPAMHAUS
Web site: http://www.spamhaus.org
IP Check: http://www.spamhaus.org/sbl/index.lasso

List Name: UCEPROTECT-Network - Level 1 List
WSO E-Mail Reject Tag: UCEPROTECT
Web site: http://www.uceprotect.net
IP Check: http://www.uceprotect.net/en/rblcheck.php

List Name: SpamCop.net Blocking List
WSO E-Mail Reject Tag: SPAMCOP
Web site: http://www.spamcop.net
IP Check: http://www.spamcop.net/bl.shtml

List Name: The Abusive Hosts Blocking List
WSO E-Mail Reject Tag: AHBL
Web site: http://www.ahbl.org
IP Check: http://www.ahbl.org/tools/lookup.php

Update 11-29-06 at 1:00pm EST
This morning we noticed the mail server was low on memory and had nearly maxed out the swap file. It turned out that one of our tuning techniques to keep mail under control was backfiring by leaving delayed mail connections open indefinitely. Once these open processes were killed off and the delay function turned off, the issue went away immediately.

Most of you should be seeing a reduction in the amount of spam getting into your WSO hosted e-mail accounts. We are still working to implement the POP-Before-SMTP auto-whitelist feature as a bypass for IPs mistakenly listed in spam blacklists. Once this is implemented we should be able to use more aggressive blacklists and you should see even less spam. But we need to be clear, no amount of filtering will remove all spam. Our goal is only to reduce the amount of spam to a more manageable level.

Update 11-28-06 at 10:00pm EST
Fine tuning continues on the new spam filtering system put in place today. We already have a POP-Before-SMTP system in place and we're trying to now integrate that into a whitelist system so that if you are authenticated to send mail through a WSO mail server based on the POP-Before-SMTP criteria, then no other blacklist checking will be done. This will help eliminate issues where a WSO client IP address is mistakenly included in a spam blacklist when clearly they are not spammers. If we trust you enough to log into our servers to check your mail, then we trust you enough to send mail through our servers irregardless of what any blacklist says. This new whitelist feature is not ready. When it becomes operational we will again implement more aggressive blacklist filtering comfortably knowing that our legitimate users will be able to send mail through our SMTP server no matter if their IP is in a blacklist or not.

Update 11-28-06 at 4:15pm EST
We've receive a few calls from people who are seeing error messages indicating that the IP address their ISP has assigned them is listed in one of the many spam blacklist databases that mail servers use to filter out known spammers. This can happen for a few reasons. One is that your could be infected with a virus that is silently sending out spam on behalf of the person who wrote the virus. The other common possibility is that you were recently assigned a new IP address and the previous user of that IP address either knowingly or unknowingly sent out spam.

To test your IP address against many common spam blacklists, go here: http://www.dnsstuff.com. Once you are there, look near the top right of the web page and find where it says "Your IP:" Cut and paste this set of numbers into the first box in the second column labeled "Spam Database Lookup" and press the Lookup button. If you see lots of "timeout" in the result column, hit the reload button on your browser until you get proper results.

Look for entries where the entire row is in RED. If you see that it means your IP address is listed with that spam database and may be causing our mail server to reject your mail. Be sure to follow up with the spam database that has your IP listed. Also call your ISP and tell them you want them to get it delisted or you want to be assigned a new IP address that is not listed. If the problem continues, please contact us by phone, fax, or using an alternate e-mail method.

Update 11-28-06 at 2:45pm EST
Nearly all SMTP functionality has been restored. If you are still unable to send mail using your WSO hosted domain as your SMTP server, please pay close attention the the error message displayed by your e-mail program. If the problem continues, please contact us with this exact error message by phone, fax, or using an alternate e-mail method.

Update 11-28-06 at 12:30pm EST
Some WSO customers may be experiencing problems sending mail (SMTP). This issue is a result of overloaded mail servers due to massive spam attacks. We are actively working on the problem. In the meantime, you may be able to change the SMTP (outgoing mailing server) setting in your e-mail program to use your ISP's SMTP server as a workaround to using your own domain name as the SMTP server. This is not guaranteed to work, but is something you can try.