Skip to main content


Showing posts from 2015

The secret message hidden in every HTTP/2 connection

If you spy on an HTTP/2 connection starting up you'll notice that it sends an almost-but-not-quite valid HTTP request at the very start of the connection. Like this:

Written a little more clearly that's:

The HTTP verb is PRI and the body contains just SM. Put them together and you get... PRISM. This occurs right at the start of the connection to ensure that the server really supports HTTP/2.0. It is detailed in Section 3.5 of RFC7540 as follows:

In HTTP/2, each endpoint is required to send a connection preface as a final confirmation of the protocol in use and to establish the initial settings for the HTTP/2 connection. The client and server each send a different connection preface. The client connection preface starts with a sequence of 24 octets, which in hex notation is: 0x505249202a20485454502f322e300d0a0d0a534d0d0a0d0a That is, the connection preface starts with the string "PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n&q…

Possibly the machines that injected JavaScript in pre-revolution Tunisia

Back in 2011 as the Tunisian Revolution was underway I blogged about JavaScript that was being injected into web pages visited by Tunisian's using HTTP. The JavaScript was designed to steal usernames and passwords of Facebook and Google Mail users.

With the release by Wikileaks of the archive of email from Hacking Team it's possible to lift the lid on this just a little.

Ten days before my blog post there was an article in Fast Company titled TUNISIAN GOVERNMENT ALLEGEDLY HACKING FACEBOOK, GMAIL ACCOUNTS OF DISSIDENTS AND JOURNALISTS which talked about this piece of JavaScript. This news article is discussed briefly by folks within Hacking Team.

One engineer comments on the JavaScript being injected by sending out a quotation just followed by the smiley ":P". Another responds in English "Truly remarkable" and finally someone responds "La foto l'ha fatta un nostro tecnico con cellulare!!!" (Photo take by one of our engineers with his cellphone…

Dirty Go: eliminating vertical space

Having written a lot of Go I sometimes find myself exasperated by functions like this:

because of all the repetition and vertical space. Working on something the other day I realized I could (ab-)use Go's switch like this:
with a small change to the functions being called. Instead of having doFirst (etc.) return an error, have it take a pointer to an error and return a boolean indicating success. That's so much more compact, but feels a little dirty.

PS You can play about with that here.

The effectiveness of Turing's Vigenère cipher breaking technique

In Turing's paper The Applications of Probability to Cryptography he describes a technique for breaking the Vigenère Cipher using Bayes Theorem. The paper was declassified in 2012 and a computer typeset version has been released (Turing used a typewriter and pen to produce the original).

Having used Bayes Theorem quite a few times (for example in POPFile) I was interested to see how it was being applied to Vigenère by Turing. Bayes Theorem is surprisingly powerful even when great simplifications (such as assuming, for example, independence of the occurrence of words or letters in an English sentence: something that's clearly not the case).

Briefly Turing's technique is to use Bayes Theorem to 'weigh the evidence' that each of the 26 letters of the alphabet could be one of the letters in the key and output weights for each letter. His technique outputs weights for each letter in each position in the key and by picking the most likely it's possible to have a goo…

Another way to find the value of GNU make's -j parameter

The other day I wrote a blog post about finding the value of -j in GNU make and it received a lot of commentary. Daniel Frey wrote in with a different solution that goes like this. It's worth studying if you are interested in GNU make wrangling.

.PHONY: default FORCE token recursive
default: foo.out
    @$(eval JOB_COUNT := $(shell cat $<))
    @echo Job count: $(JOB_COUNT)

foo.out: FORCE
    @timeout 1 $(MAKE) --no-print-directory token recursive 2>/dev/null | wc -l >[email protected]


    @echo X && sleep 2 && false

    @$(MAKE) --no-print-directory token recursive

One weird trick that will give you makefile X-ray vision

Sometimes when you are running a make you'd like to see what commands get executed (even if they've been muted by prefixing them with @) and where in the makefile those commands reside. But you can't. There's no way to override @ and make -n doesn't actually run the make.

But here's 'one weird trick' that makes it possible to see inside a make as its running. Just add this to the makefile:

SHELL = $(warning [[email protected]])$(_SHELL) -x

Before digging into how that gives you makefile X-ray vision, let's take a look at it in action. Here's an example makefile that builds an executable from foo.o and bar.o (which are built from corresponding foo.c and bar.c files).

.PHONY: all
all: runme

runme: foo.o bar.o ; @$(LINK.o) $^ -o [email protected]

foo.o: foo.c
bar.o: bar.c

%.o: %.c ; @$(COMPILE.C) -o [email protected] $<

Running simply make on this performs the compiles and link but produces no output because output has been suppressed by the @ signs on the compile and link comm…

Plain web text offenders: sending my location over HTTP when HTTPS was possible

The BBC Weather App sends the location of the user via HTTP to the BBC in order to determine the best location for weather information. It does this with roughly 100m accuracy, in the parameters of an unencrypted GET and even though the same API endpoint is available using HTTPS.

I discovered this accidentally using Charles Proxy to snoop on traffic from my iPhone at home. Here's the Charles Proxy view of that app interacting with the BBC's APIs:

It's hitting the endpoint with parameters la and lo containing the three decimal digit latitude and longitude (which give roughly 100m precision) over HTTP as a GET request.

The API then returns a JSON object containing nearby locations that the app can get weather information about.

Sadly, this API could have been accessed over HTTPS. Just switching the protocol from HTTP to HTTPS works fine. A legitimate wildcard certificate for * is used.

So, the app could have used HTTPS.

The one line you should add to every makefile

If you're using GNU make and you need help debugging a makefile then there's a single line your should add. And it's so useful that you should add it to every makefile you create.


    print-%: ; @echo $*=$($*)

It allows you to quickly get the value of any makefile variable. For example, suppose you want to know the value of a variable called SOURCE_FILES. You'd just type:

    make print-SOURCE_FILES

If you are using GNU make 3.82 or above it's not even necessary to modify the makefile itself. Just do

    make --eval="print-%: ; @echo $*=$($*)" print-SOURCE_FILES

to get the value of SOURCE_FILES. It 'adds' the line above to the makefile by evaluating it. The --eval parameter is a handy way of adding to an existing makefile without modifying it.
How that works The line

    print-%: ; @echo $*=$($*)

defines a pattern-rule that matches any target in the form print-% (the % is the wildcard character). So when you run make print-SOURCE_FILES that…

"The GNU Make Book": probably more than you ever wanted to know about make

So, it's finally here. My second book, entitled The GNU Make Book and published by No Starch Press later this month (electronically it's available now). If you are a long time reader of my blog then you'll know that I've had a very long interest in GNU make and written quite a bit about it. For a while I had self-published all my articles on GNU make as a print on demand book.

The folks at No Starch Press kindly took my self-published effort and did the serious work of actually editing, organizing, indexing and generally sprucing it up. I added some articles that weren't in the old version and updated for GNU make 4.0.

And now it's here.

The book is not intended for people learning GNU make from scratch. If you need to do that go get the FSF's GNU make manual and read it. It covers all that you need to get started with GNU make.

Once you've done that you'll be ready to read mine. It's based on years of working with real makefiles, on having wri…

GNU make insanity: finding the value of the -j parameter

The other day at work a colleague posed the seemingly innocent question "Can I find out the value of the GNU make -j parameter inside a Makefile?". Simple, right? Recall that -j (or --jobs) specifies the maximum number of jobs that GNU make can run in parallel. You've probably used it to speed up a build by typing something like make -j16.

But it turns out the actual value supplied is not available in any GNU make variable. You can verify that by doing something like make -p -j16 which will dump out pretty much everything defined by GNU make and the Makefile. You won't find any reference to the -j16 anywhere.

What follows falls into the "It's crazy, but it just might work!" category.
But it is possible to calculate its value if you are willing push GNU make hard. Here's a Makefile that gets the value given to -j (or --jobs) into a variable called JOB_COUNT. 
There's quite a lot of magic in that Makefile. I'll explain how it works bit by bit.…

Failing to RTFM for Go: defer statements

So, a colleague wanders over the says: "You know that commit you just made to project X? It doesn't work" and the proceeds to remind me of that fact that the defer statement in Go is rather special.

The argument of the defer statement is a function that will be called when the defer statement executes. But its arguments are evaluated when the defer statement is encountered (here's where you, or at least I, RTFM).

A function like this won't do what you expect:

func deferPrintf() error {
var err error
defer fmt.Printf("deferPrintf: err has value %v\n", err)

err = errors.New("Error 2")
return err

That will always output deferPrintf: err has value because the function being called (fmt.Printf) by the defer has its arguments (which include err) evaluated when the defer is encountered.

Oops. I'd forgotten that.

The solution is pretty simple. Wrap the fmt.Printf in a closure like this:

func useAClosure() error {
var err error
defer func() {