Thursday, February 01, 2007

Proposal for connection signing reputation system for email: TECS

IMPORTANT: This blog post is deprecated. Please read Trusted Email Connection Signing (rev 0.2) instead


The motivation behind TECS (Trusted Email Connection Signing) is that what managers of MX servers on the public Internet really care about is the ability to distinguish a good connection (coming from a legitimate sender and which will be used to send wanted email) from a bad connection (coming from a spammer). If you can identify a bad connection (today, you do that using an RBL or other reputation service based on the IP address of the sender) you can tarpit or drop it, or subject the mails sent on the connection to extra scrutiny. If you can identify a good connection it can bypass spam checks and help reduce the overall false positive rate.

Currently, the options used to identify a bad connection are rather limited (RBLs, paid reputation services and grey listing), and good connections are hard to manage (whitelists on a per-recipient basis, or pay-per-mail services). What's needed is a different approach.

There are also ideas like SPF, Sender-ID and DomainKeys which all attack the problem of protecting the integrity of the From: portion of a message.

TECS is different. The idea is to identify and determine the reputation of the entity connecting to a mail server in real-time without resorting to a blacklist or whitelist. This is done by signing the connection itself. With the signature on a per-connection basis a mail server is able to determine who is responsible for the connection, and then look up that entity's reputation in a database.

Current reputation databases are based on IP addresses. This is a very inflexible system: IP addresses must be added to blacklists very fast as spammers churn through zombie machines, and any legitimate emailer needs to make sure their mail servers are whitelisting with multiple email providers (e.g. Yahoo!, Gmail, Brightmail, ...) to ensure delivery. And if a legitimate mailer wants to bring on line new servers, with new IP addresses they have to run through the entire whitelisting process again.

This is inefficient. The mapping between IP address and entities (e.g. knowing that Google's Gmail services uses a specific set of IP addresses) is unwieldy to manage and the wrong level of granularity. Google should be free to add and remove email servers at will, while carrying their good reputation with them.

That's what TECS gives you.

Now for the how. To work TECS requires two things: a reputation authority and an algorithm. Let's start with the second.

Connection Signing

When a mail sender connects to an SMTP server wishing to sign its connection it issues the EHLO command and if that SMTP server is capable a new extension command TECS will be available. After the EHLO the mail sender then signs the connection using the TECS command.

The TECS command has two parts: an identifier (this is the unique identifier of the entity signing the connection, and thus taking responsibility for the messages send across the connection) and a signature.

Each entity has an RSA key public/private key pair. When signing a connection the entity generates a SHA-256 hash of the tuple . The destination IP/port pair is the IP address and port on the mail server that the mail sender is currently connected to; similarly the source IP/port pair is the IP address and port of the connection being used by the
mail sender. The epoch is the standard Unix epoch rounded to the nearest 30 seconds.

The entity making the connection then encrypts the hash with their private key, turns that into a hex string and uses that string as the second parameter to the new SMTP TECS command.

For example, an entity with the unique identifier 1b46ef4 might sign a particular connection like this:

TECS 1b46ef3d 5dde82a341863c87be1258c02ce7f80bf214192b

to which the receiving server could reply 200 OK if the signature is good (which they verify by generating the same hash and decrypting using the entity's public key), or with an error if the signature is bad (and they should probably drop the connection).

To get the entity's public key the receiving server needs to query the reputation authority.

Reputation Authority

The TECS reputation authority would be a non-profit organization that sells public/private key pairs and allocates entity IDs to verified entities. Money gathered from selling keys would be used to maintain the database of reputation information for each entity, and in ensuring the only reputable entities can obtain keys.

In the example above the receiving server would query the DNS TXT record of the domain name produced by concatenating identifier given in the TECS command with the name of the authority. Suppose that the authority was tecs.jgc.org then a DNS TXT query would go to 1b46ef3d.tecs.jgc.org.

The reply would consist of the ascii-armored public key for that entity and a reputation measure indicating the reliability of that user. The reputation measure would take one of 4 states: unknown (a recently issued key would not have any reputation), good (only a small number of complaints against this ID), medium (some complaints), bad (large number of complaints, probable spam source). The receiving server can verify the signature and use the reputation information to decide on the handling of the connection.

The authority would accept ARF formatted complaints consisting of abusive messages giving connection information, and the full text of the TECS command. They would then investigate to ensure that the reputation database contained up to date and useful information.

How much is a key pair going to cost?
I think it should be cheap for individuals ($25?), fairly cheap for non-profits and charities ($100?), and then a sliding scale for for-profit companies based on size (say $100 for a small company, $1000 for a big one?). The goal would be to make enough money to run the list.

What about mailing lists that forward mail?
By signing their connections they take responsibility for the mails they are sending. So mailing lists would need to have appropriate email policies in place for unsubscriptions, and deal themselves with spam to the list. Since the connection is signed any concern about munging of From: addresses for VERP handling, or adding headers/footers to email are irrelevant.

Is this compatible with SPF, Sender-ID, DomainKeys?
They are orthogonal. There's no direct interaction.

Will this reduce spam?
I'm not going to make any predictions. The goal would be to build a database that makes it easier to recognize someone who is legitimate, and scrutinize those who abuse the system or who choose not to sign.

What about anonymity?
Anoymous remailers are unaffected. They could sign their outbound connections with the system but that would not affect any changes they make to anonymize messages since its the conneciton, not the message content that's signed.

What if I change the mail servers or IP addresses I am using?
There's no effect. Keep signing the connections and you can take responsibility for any IP address you want to.

I think you are wrong, right, stupid, a genius.
Please comment here, or write to me directly.

4 comments:

fanf said...

SMTP already has STARTTLS which supports client certificates.

John Graham-Cumming said...

I agree, but doing STARTTLS means that you want to have an SSL/TLS encrypted session through which you pipe your messages. Sure you can have a certificate, but I think it's too heavyweight (you don't need session encryption) and it doesn't solve the reputation part.

John.

None said...

Great, so spammers have proven that they can afford to buy domains, set up dns with SPF records, etc. WTF would stop them from buying these? Central authorities have already proven they cannot be trusted to prevent the bad guys from buying up domain names and they cannot be trusted to be stewards of the info about the non-badguys (Verisign, ICANN, etc). A single central authority doesn't work well for the whole world due to national legal boundaries [even if they could be trusted]. Since buying a cert doesn't stop the spammers, it follows that it only stops legit "poor" users.

And please, any response based on the fact that their would be some kind of "checking" or verification of the business is total BS - you can't tell me if a LLC from Nevada USA or some small Asian country is legit.

For whatever cases self certification does not work, those cases also fail for small $ amount payment for certs or other big brother stamps of approval. This is proven by the current state of trafficing of domain names, hosting, colo space, bandwidth, IP allocations, BGP adverts, etc.

So given that, you might as well solve the problem for the self certified case which means explaining how to make DomainKeys or DKIM work. Self generated certs create useful and valuable positive and negative reputation information - google uses DomainKeys currently, so we can assign positive reputation points to their key, similarly we can assign negative rep points to a never-before seen DK and create a greylisting system based on DK key.

A midway between your system and DK could be where a client attempting to connect to your server would sign a string that was a concatenation of your server IP, the public part of the DK key (that they are using to sign mail for the current connection) and the time. Since they know these 3 pieces of data prior to the connection, they can spin it up (a proof of work for free to you). They could then include that at the end of their ELHO response line. The ability to tie the connection signature with the individual DK sigs in each email would strengthen both and the proof of work would be a gesture of non-malicious intent like hashcash.

Signing in for comments sucks.

John Graham-Cumming said...

Your comment (and others I received via other means) makes me think that having an authority is the wrong way to go. It seems to me that self-signing is the answer.

My initial way of doing this would be to say that you self sign using a domain name as the identifier, e.g.

TECS google.com [hex string]

and the recipient looks up the TXT DNS record of say _tecs.google.com to get the public key to be able to verify the same hash as described in my article.

The entire question of how you assign reputation to domains is then left up to the user of the system. Could be a central database, could be totally private. Self-signing like that makes is more flexible and leaves the options open.

I'm not sure that I fully understand your last paragraph. Could you go into a bit more detail, it sounds like an interesting approach, but I'm confused by the line: 'They could then include that at the end of their ELHO response line.'. Surely, its the destination server that is responding to the EHLO. Where does the client SMTP server place their signed string for verification?

Sorry about the no anonymous comments policy, I was getting too much abuse.

John.