Sam Pointer

I recently had cause to liase with the support team at a hosting company, who shall remain anonymous to protect the guilty.

I was having some problems with getting the SMTP server on the VM image they had there to properly accept email. As far as I knew I had everything in place:

• correct MX, A and SPF records for the domain published and properly propagated, confirmed by a ‘dig’ from an external server
• domain listed in the local domains (Cw) section of the Sendmail configuration
• Sendmail restarted to ensure everything had taken effect
• SMTP port 25 allowed in the local iptables configuration

Everything should have been working, and yet I was receiving the dreaded "Connection Refused" back from my sending SMTP server after the requisite time-out period.

From a local machine I attempted to connect to port 25 of the receiving SMTP server with netcat, reasoning that if I could get at least a TCP connection to the port I’d know that my iptables configuration was good and something must be up with my Sendmail configuration. No joy, connection refused.

So I was getting somewhere. In order to make sure it wasn’t just a local problem I confirmed this from another remote server. Same result. I then took advantage of the port monitoring facility of the HyperVM virtualization manager the host was using to see if I could connect to port 25 from there, my reasoning being that it would narrow the problem down to the hosting company’s infrastructure and rule out any sillies at my end. Sure enough, expected ports such as 80 for http and 22 for ssh were open and accepting connections, 25 wasn’t.

So, I lodged a support ticket with my findings, what I was trying to achieve and what I needed doing, namely opening up whatever part of their infrastrucuture was blocking port 25 to the VM in question. Being in the UK and the hosting company being in the states I left it at that and expected to come back the next day to find everything sorted.

Two hours later I got an email from the monitoring service on the VM. xinetd had failed, which has odd because xinetd virtually never fails.

Connecting to the machine for some investigation revealed that the Dovecot POP3 server was constantly trying and failing to bind to port 110, and the rapid respawing had caused xinetd to fail leaving a trail of pid files and subsystem lock files in it’s wake. This was quite confusing, as I was using a custom-compiled version of qpopper launched via xinetd to handle POP3 mail, and had never installed Dovecot.

All was revealed when an email appeared shortly afterwards from the support desk of the hosting company saying "I’ve installed the Dovecot pop3 email server, let me know how you get on now".

Fail number one.

The support guy at the other end had obviously not read the support ticket, which made no mention of pop3. He’d also assumed that I wasn’t running another pop3 server - a quick `netstat -a | grep -i listen` would have told him that I was.

So, I promptly told him all of this an reiterated my original request that, thankyou very much I know what’s going on (and here’s the evidence to prove it), and would you kindly remove the POP3 server and open port 25 please. Oh, and please uninstall Dovecot as I’ve already got a POP3 server and I don’t want a repeat of the xinetd failure.

I then receive another reply from a different technician saying sorry for the confusion, and he’d enabled SMTP Authentication within Sendmail, and could I please try again. Should things still not be working, could I please forward them my email address and password so they could configure their mail client the same way and attempt to send an email via my Sendmail instance.

Fail number two.

I’d never mentioned collecting email, sending email via the SMTP instance (which was configured only to relay from the local machine) or any requirement for authentication. I’d only asked for the port to be unblocked in their firewall.

This change had the compound effect of causing the ActionMailer portion of a Rails app that I was running to promptly fail, as it expected to relay via a local SMTP server without authenticating.

The helpdesk had now succeeded in knocking out the ability to collect email for the domain, caused an application sending email from the domain to fail, and still not solved the original problem.

Eventually after stating in explicit terms that they should undo everything they had done aside from keeping port 25 open the problems were fixed. It brought home to me a lesson that I had learnt long ago, and that these anonymous helpdesk technicians would do well to learn themselves.

First and most importantly, read what is in front of you. I had provided enough information in my original support ticket, backed up with sensible evidence of what was wrong, and made an explicit request for a single task to remedy this.

I have dealt with a wide number of financial institutions in my time. Some of these are large multi-national banks, others are small shops with barely a single IT literate person in the company. You learn to pitch your questions and your answers to the audience that you are addressing. For some people you have to explicitly spell everything out. For others you need only tell them what to do in a few concise sentences, and provide the technical detail (IP addresses, paths, passwords, whatever) that they could not know themselves and let them get on with it. The first group appreciate the hand-holding as they don’t have to ask questions and perhaps sheepishly asking for more help. The second group appreciate being treated as a professional and you getting out of their way to get the job in hand done. The real skill is learning how to pitch to the shades of gray in-between.

Being a specialized hosting company I addressed the helpdesk guys as professionals, and they resolutely showed that whilst they might have the technical cojones, they faled at good helpdesk interaction and judging how to pitch and receive information. It was obvious from the information I provided in the support ticket that I knew my stuff and had a simple request that they could have carried out and closed the ticket. If I had written, "hello there, I am not able to get my email. An error keeps popping up and it says refused, or something" then it would have well been in reason to go about installing a POP3 server and asking me to test it out. But they didn’t pick up on the subtle cues of content, syntax and context which would have enabled them to deduce what was going on.

In the technical consultancy field we should never forget who we’re talking to and what we’re talking about, and we should pitch our interactions as such. And like the test we’ve all done at school where the last instruction is to only write your name, always read what is in front of you.

After a recent discussion, I realised that as a techie my take on security might be a little skewed. I used to subscribe to the industry journal SC Magazine, which used to arrive and make it’s way promptly to the office toilet to be added to the pile of, er, break-time reading.

Typically for trade publications it was pitched at the CTO level, and was essentially a glossy sponsored advert for various boxes that could be plugged into the network and automagically solve all network ills, complete with nicely priced support contract.

I used to be quick to distain this type of publication as simply box-shifting nonsense, reasoning that real security was to be found in my iptables configuration, SSH keyrings and carefully crafted ACLs. It’s temping to think about security as a configuration task. One that’s a process that must be continually reviewed, but a technical configuration task nonetheless.

A number of years ago I assisted with a drive to bring a hosting service up to scratch and compliant with BS7799. That security standard has now been ratified by the ISO as ISO/IEC 27002: Information technology - Security techniques - Code of practice for information security management.

Whilst a little painful, the exercise exposed a few glaringly obviouis and not-so-obvious failings in the way the company conducted itself in terms of security. If you take the time to read over the outline of the standard you may find that you’re doing quite a bit of the required work without even knowing it. By taking a few short steps and submitting to the formal certification you’ll gain extra confidence that you’re following industry best-practice, and can pass this assurance onto your clients.

For example, got a DR site? You’re well on your way to satisfying section 11 of the standard: "Business continuity management". Tied your Solaris authentication into your Windows Active Directory LDAP tree and adopted a change and review procedure to altering access? You’re well on your way to satisfying section 8.

I’ll wager that most shops that are already managing their systems in a sensible way would not have to put too much extra work in to get a bona fide certification that you’re doing things the right way.

Of course it should be remembered that a framework such as ISO/IEC 27002 is just that. It won’t tell you how to configure tcp wrappers, it won’t protect your private SSH keys and it won’t make sure you’ve patched everything under the sun to not use sequential transaction IDs and/or source ports in DNS requests.  

I was walking past the Lehman Brothers building in Canary Wharf complex of London today. Outside there was a Lamborghini on a trade stand, complete with bimbo model.

Ironic, eh?

Yeah, I chuckled a little inside too before returning to the same thought I’ve been having all week. The traditional ivory tower techie attitude has it that "the business" are a bunch of baffoons who know nothing about technology, and should just get back to wowing each other with Powerpoint effects. The rest of us understand that, for the most part, we need "them" as much as they need "us".

It’s a worrying time for anyone involved in financial IT. Budget freezes, canned projects, collapse of whole organisations. Even if you’re not directly working within the finance industry you and your clients need money, and a healthy economy in which to operate. Droves of newly unemployed people who presumably didn’t anticipate it are going to drive real competition in the recruitment market.

In sentence: things are looking bad.

What to do? Well, as all the best faces on Bloomberg like to say, "there is some hope". Share trading systems generate revenue no matter whether the stock is rising or falling. A trade is a trade, and they still get paid, so to speak. 

Plus, the UK at least still has a strong gambling market - and a modern online gambling infrastructure is not that far removed from any other financial trading platform. As Cal Henderson said in his recent Django 2008 keynote, speed seems to be the key in that sense (it’s also a refreshing good talk - funny bloke with some good ideas).

The Perl to Rails re-write of www.dailyunixtip.com is drawing near.

Almost all of the functionality required to have a working site is there, there’s a decent suite of tests (that now pass!), and the back-end infrastructure is almost there, thanks to the wonderful people over at hostingrails.com.

So, what have I learnt so-far?

Centos isn’t a patch on RHEL-proper

(But only if you actually pay-up for that support contract). I really mean this.

In previous roles in corporate environments RHEL versions 3 and 4 were prevalent. Big software, big gear, serious stuff, nice men from IBM.

Although I recognise that dailyunixtip.com isn’t even tugging at the coat-tail of that kind of infrastructure I have still be immensely frustrated at how much of a mish-mash the world of Centos package management is. To be able to apply .rpms you know have been tested and will work from a trusted repository with everything in it is a god-send. If you find yourself on Centos, go here for the nearest thing to it.

Of course, I’m not going to practice what I preach and actually buy a support contract, so in that sense Centos fills a niche for small no-budget exercises such as this. You get a de-baged RHEL for the price of nothing, but like repo parts for your Ford Escort: it won’t fit right from the factory.

I suppose that is what is at the core of what has irked me about Centos. I’m all for fiddling and deliving and generally getting deep into the minutae of the technology in front of me. But I’m also a business. I want my time to be spent in my code fettling my process and the infrastructure I have built. That’s what I’m paying or being paid for, and that’s ultimately what’s got to be done to generate the money that makes the whole exercise worthwhile.

Any external distraction (read: configuring external rpm repositories to install a FastCGI-enabled copy of lighttpd) is costing the project money.

Infrastructure changes take time, even in small outfits

In a past life I use to administer to a fairly complex BIND 9 installation, which was used as part of a failure and resilience infrastructure for an ASP service. Having thought about running a small named to take care of dailyunixtip.com and the various sub-domains, I rejected this in favour of leaving the whole thing to my domain registrar.

Again, in a small setting such as this any time spent fiddling with named.conf files would be a waste.

But, there’s a big "but".

You need to do a least a modicum of planning to ensure that all of your domains and subdomains and forward and reverse records are pointing the right way in very good time before you intend to launch. Running your own nameserver is a blessing in disguise for the first part of a new project, when short TTLs can help you rapidly deploy and test supporting infrastructure components such as SMTP servers, Apache and the like.