I’m posting this a day late because it took me a whole morning to figure this one out. It appears that Comcast has completely blocked both inbound and outbound traffic on port 25 for my company. For a few years now we have been running a in-house authenticated SMTP server using IIS. It was running on port 587 and basically relayed emails to another server at an off-site location on port 25. Why was it set up this way is a topic for a whole other rant, but it worked well for us until now.
When I came in to work yesterday morning all was well. Few people around the office mentioned something about a slow day, noting their inbox was unusually empty. I didn’t really pay much attention to that chatter, until someone decided to email something to herself and it never came through. Then all hell broke loose.
You see, this problem was essentially hidden from regular users because they could connect to my in-house server on port 587 without any issues. So their emails were leaving their outbox as normal, and then queuing up on the server, never to be seen again. The server itself could not shoot them back failure notifications either, because it could not connect to it’s relay point on port 25. So people were emailing each other all morning without even realizing something was amiss. When they caught on, it was instant panic mode spreading throughout the organization like a fucking wildfire.
For several hours I was methodically checking, re-checking, restarting, and power-cycling every single device and service that had anything to do with email traffic. I was also calling the folks who maintain the off-site server every 5 minutes to see what was their progress. They were convinced the issue was on my side, and I was adamant that it was on their side. After much deliberation, we came to the conclusion that we were both wrong. The off-site server was accessible from everywhere but my location, but there was nothing here in the office which would prevent it from communicating on port 25.
We tested outbound and inbound traffic on their side and it was working just fine so that left only one conclusion – my ISP fucked us over and completely sealed off port 25. Once we realized that, the conclusion was as swift as it was simple. We simply switched the external server to listen on port 587, changed the outbound port in IIS and an avalanche of backed up email started streaming into people’s mailboxes.
Let me run that by you again in case you didn’t notice – once we figured out what the issue was, it took us 5 seconds to reconfigure our shit, and route around it. So if this supposed anti-spam measure is so easy to circumvent, then can someone explain to me how is it supposed to be stopping hard core spammers with their sprawling botnets out there? I’m pretty sure most of semi-modern spam-trojans can be remotely reconfigured to send out emails on alternate ports.
Port blocking has became pretty much an industry standard these days, but I still fail to see how it could ever be effective. What is stopping me from running an email server on port 80 or 443? Will they block these two ports as well? It is just a knee jerk reaction, that might be effective in a short term. It won’t work in the long run though – soon they will run out of ports to block, and regular customers won’t be able to use any kind of non-standard internet services for genuine purposes without bending over backwards.
This is just one of these wholesale, one-click-and-your-done spam solutions. Why do ISP’s do it? Because it’s easy! You block some important ports, and the amount of spam and genuine email routed through your network goes down. You boss is happy, your investors are happy, folks in the security business are clapping their hands marveling at the sudden drop in spam, forgetting it will be back to normal in a month or two as all the spammers will figure out the same thing I did just now.
The only people who are not happy about this are the customers, but Comcast does not really care about them that much anyway as it has blatantly demonstrated in the past with it’s bandwidth throttling, and lackluster tech support.
Also, Twitter > than regular tech support resources it seems:
Despite the fact that we were constrained to 140 characters per pop, talking asynchronously and multitasking, this was still way more pleasant than my experiences with Mr. Rooter and Mr. 125 Times. Not sure if that guy is an actually really affiliated with the company in any way, but he seems to be representing them well in the 140 character conversation universe.
One more reason to love Twitter and hate Comcast! :mrgreen:
[tags]comcast, port 25, email, smtp, spam, comcast sucks[/tags]
God I want to be a guy being paid to use twitter all day.
I’m sure that is not his only responsibility. But yeah, being the Twitter spokesperson for a company wouldn’t be a bad job. :P
I think its very innovative though, to use twitter.
If I was running a company I would use all means like that, myspace the works… that way everyone can stay connected.
er… I assume your company uses a professional connection. Why on earth are they blocking ports on a professional connection? The implication of a professional connection being that there’s a sysadmin of some stripe administering it?
Furthermore, if it is a professional connection, why did they not warn you in advance of changes that may affect business? Shutting off an industry standard port surely falls under that category.
I agree that it makes no sense for an ISP to without prior notice suddenly block ports on a company connection.
However, I don’t think blocking outgoing port 25 is a short-term knee jerk thing. It really makes it impossible for spammers to use hacked local computers as spam relays. If outgoing port 25 traffic is blocked there really is no way to directly contact most of the email servers on the Internet, since they all listen on port 25. The spammer would have to set the hacked computer to go through another relay on another port — and if so, the first relay is useless as it does not expand the bandwidth.
Also, a protip you probably already know: my favorite tool for diagnosing smtp-troubles is ‘telnet’. Simply doing
telnet smtp.offsite.example.com 25
and run the smtp conversation manually to inject an email have helped me find the source of several problems in the past.
[quote post=”2423″]er… I assume your company uses a professional connection.[/quote]
Actually, we seem to be on a residential circuit – go figure. I’m assuming it got set up like that, because it dirt cheep at the time. :(
Oh well, I guess this is a good argument to present to the big boss in explaining to him why the extra expense is actually worth it.
[quote post=”2423″]Also, a protip you probably already know: my favorite tool for diagnosing smtp-troubles is ‘telnet’.[/quote]
Yep, that’s what I was doing. Actually, I have a sheet with the list of relevant POP and SMTP commands hanging in my cube just for the times like this. :)
[quote comment=”8937″][quote post=”2423″]er… I assume your company uses a professional connection.[/quote]
Actually, we seem to be on a residential circuit – go figure. I’m assuming it got set up like that, because it dirt cheep at the time. :(
Oh well, I guess this is a good argument to present to the big boss in explaining to him why the extra expense is actually worth it.[/quote]
Just had one of those jaw-drop moments. Running a business on a residential connection is insane. Business contracts come with better SLAs, better response times, and easier access to assistance. Depending on your SLA, you may even have a right to compensation if your business suffers because of a loss of connection due to avoidable actions at the ISP end. If a morning without email sends your company into a panic, then you shouldn’t be on a residential connection.
That being said, I’ve heard an argument for SMBs having two residential connections with different ISPs over different paths, as some outages affect both business and residential users identically and sometimes the business connections are insanely priced. Basically you use one residential line to cover for outages in the other.
Yeah, but that’s what happens in small companies – the “why do we need the expensive business plan if we could buy two residential ones for the same price” factor comes into play. :P
take a printout of your Mr Rooter and Mr 126 times along with you to management. Point out that this one event cost 3 hours of your wages… and then point out that replacing the modem 3 times the previous year cost more of your wages, downtime, any other charges you can think of, all on bad ‘residential’ advice. Point out also that the time you spend working around the blocks put on a residential service effectively costs the company as well, time that could be better spent
chatting to the receptionistworking diligently for increased profit margins.Pingback: Twitter Haters « Terminally Incoherent
