Saturday, April 29, 2006 launched

Today I'm officially launching It's Hot or Not for email.

The basic idea is to get humans (that means you) to read a small number of messages (some are ham; some are spam) and decide what they are. I'm doing this because there are currently two usable corpuses of spam and ham: the SpamAssassin Public Corpus (which was hand sorted) and the TREC 2005 Public Corpus (which was machine sorted).

The TREC 2005 Public Corpus is the first target of With your help we can all verify that the machine sorted messages in the corpus were correctly identified as spam or ham. Given that there are so few public test resources, it's essential that those that are out there are accurate.

Starting today you can visit and be shown rendered and unrendered emails from the TREC 2005 Public Corpus. Once I've got enough human decisions (I'd love to get 10 per message; that means almost 1,000,000 human classifications) I'll make all the data public. And specifically I'll highlight any emails where people disagree with the current classification published by Gordon Cormack.

Hopefully, this effort will ensure that the TREC 2005 Public Corpus is rock solid. And, I expect it'll through up some interesting data... for example, just how good are humans are sorting spam? Since we'll be able to look at where the corpus and the humans disagree we'll be able to spot machine errors and human errors.


If you enjoyed this blog post, you might enjoy my travel book for people interested in science and technology: The Geek Atlas. Signed copies of The Geek Atlas are available.


<$BlogCommentDateTime$> <$BlogCommentDeleteIcon$>

Post a Comment

Links to this post:

<$BlogBacklinkControl$> <$BlogBacklinkTitle$> <$BlogBacklinkDeleteIcon$>
Create a Link

<< Home