Today I'm officially launching spamorham.org. It's Hot or Not for email.
The basic idea is to get humans (that means you) to read a small number of messages (some are ham; some are spam) and decide what they are. I'm doing this because there are currently two usable corpuses of spam and ham: the SpamAssassin Public Corpus (which was hand sorted) and the TREC 2005 Public Corpus (which was machine sorted).
The TREC 2005 Public Corpus is the first target of spamorham.org. With your help we can all verify that the machine sorted messages in the corpus were correctly identified as spam or ham. Given that there are so few public test resources, it's essential that those that are out there are accurate.
Starting today you can visit spamorham.org and be shown rendered and unrendered emails from the TREC 2005 Public Corpus. Once I've got enough human decisions (I'd love to get 10 per message; that means almost 1,000,000 human classifications) I'll make all the data public. And specifically I'll highlight any emails where people disagree with the current classification published by Gordon Cormack.
Hopefully, this effort will ensure that the TREC 2005 Public Corpus is rock solid. And, I expect it'll through up some interesting data... for example, just how good are humans are sorting spam? Since we'll be able to look at where the corpus and the humans disagree we'll be able to spot machine errors and human errors.