Thursday, December 22, 2011

A simple illustration of the use of goroutines and channels in Google Go

For a long time I've been meaning to spend some quality time with Google Go and I finally have the chance. For the little home project I'm working on I needed a way to generate unique IDs. In Go there's a really nice and simple way of doing this: assign a single goroutine as an ID generator and use a channel as the way to grab a new ID.

Here's the code:
    idMaker := make(chan string)

    go func() {
        var counter int64 = 0
        for {
            idMaker <- fmt.Sprintf("%x", counter)
    } ()
The first line makes a channel that will be used to communicate strings. In this case, I'd decided to have unique string IDs.

Then there's a go function (in this case an anonymous function) that contains a 64 bit integer counter that is incremented every time an ID is generated. When an ID is generated it is formatted as a hex number and made available on the channel. It just loops around forever providing IDs and only updating the counter when an ID has been used.

Any other part of the program can grab an ID like this:
    id := <-idMaker
Because reading from a channel is atomic and only one part of the program can read from the channel at once this eliminates any headaches about threads or processes needing to share some unique ID generating code. In fact, multiple goroutines can use exactly the same channel to get unique IDs trivially.

This works because Google Go takes its channel synchronization idea from CSP where communication is how synchronization is done. There's no need for any sort of locking around the var counter because it only gets updated in one place (inside the goroutine) and it only gets updated when its value has been consumed by reading from the channel.

Thursday, December 15, 2011

A pre-Christmas Plan 28 update

Despite the lack of public announcements from Plan 28 on the project to build Charles Babbage's Analytical Engine there's plenty going on behind the scenes.

The big news is the Plan 28's registration as a charity in the UK is now complete and has been published by the Charity Commission. Plan 28 is now registered charity number 1145043.

Every charity has to have 'objects' which are a statement of its aims. Plan 28's objects are as follows:
3. Objects

3.1 The Charity's objects ("The Objects") are for the public benefit

3.1.1 The advancement of education, in particular in the fields of science and the history of science, through the construction of Charles Babbage's Analytical Engine; and

3.1.2 The advancement of science, in particular through:
a) Research in the fields of computer science, engineering and mathematics; and
b) The raising of public awareness of Charles Babbage's Analytical Engine and related scientific theories and developments.

3.2 Article 3.1 may be amended by special resolution but only with the prior written consent of the commission.
What this means is that early in the New Year Plan 28 will enter a fundraising phase and all those folks who pledged money will now be able to donate it. It also means that Plan 28 will be eligible for Gift Aid for UK donors.

Here's to getting the Engine project moving in early 2012.

PS A big thank you to the solicitors at Stone King who worked on the charity registration process with me. They've been simply brilliant to work with and I would highly recommend them for other people starting a charity.

Tuesday, December 13, 2011

The Sainsbury's Wheel of Death

The British supermarket Sainsbury's labels its own food products with something that resembles a Trivial Pursuit game piece containing colored slices showing the 'healthiness' of the food. They call this the Multiple traffic-light labeling. I prefer to refer to it as the "Wheel of Death".

There are five categories on the wheel of death: total sugars, calories, fat, saturated fat and salt. When visiting Sainsbury's it's fun to see if you can find a full, all red 'Wheel of Death' on a product. I haven't managed to photograph one yet.

Here's a partial Wheel of Death:

And getting a bit closer to a full Wheel of Death on a chocolate cake mix packet:

And one more on a ready made dessert:

But has anyone seen an all red Wheel of Death? A product that combines high salt, fat, saturated fat, calories and sugar? If so, please email me.

Another piece of Apple design that drives me nuts

The Apple wireless keyboard and its side loading battery slot. Here's mine. The batteries needed replacing.

To get at the batteries you open a little circular cover on the right hand side and slide the batteries out. Or not. If one of the batteries has split open (even a small amount) it becomes jammed inside the slot and there's so little room that it's very hard to get it out.

There's a battery in there. Poking something into the hole doesn't help because you end up jamming the battery in further. The only viable option is to hit the keyboard on a hard object (e.g. the table) repeatedly and violently until the battery comes out.

It was a painful experience taking a piece of Apple hardware and whacking it on the table.

Tuesday, December 06, 2011

A back channel confirms that I'm right... sort of

Through a little circuitous back channel I received an unofficial follow up to a blog post about the unused machine code in the GCHQ Code Challenge part 2. The follow up assured me that the unused code was left over after some clean up was done, and that the rest of the data in the file was random filler (as I'd already heard).

But as I'd already determined that it was not actually random (at least at one level) I sent a carrier pigeon out with a question about my follow up blog post.

Listening for secret messages transmitted during The Archers I received a reply indicating that I had indeed 'broken' the encryption on that stream of ASCII data (by guessing the algorithm and reverse engineering the key stream), but that the underlying data was actually random ASCII generated and encrypted using the following Python code:
b = ''.join([chr(randint(0x20, 0x7f)) for i in range(0, 16 * 7)])
c = codec(b, 0x1f, s=5)
The first part generates 112 bytes of random ASCII text and the codec function is apparently the encryption function with the key I had identified (see the 0x1f and 5).

There are a couple of oddities about this code. The randint could generate an 0x7F which is non-printable (and doesn't appear in the decrypted text) and it generates precisely 112 bytes of data (whereas the part 2 actually contains two blocks of 112 and one of 102).

I questioned that by leaving a note in a tree in St. James's Park and received a response by exchanging briefcases at King's Cross Station to the effect that the 102 byte block had simply been truncated to make the code look interesting.

Of course, that could all be completely non-suspicious, but having been told that first it was random filler and then that actually it was blocks of encrypted random printable ASCII filler it wouldn't surprise me if the truth was even more complex (and perhaps rather mundane).

But why go to all this trouble on 'random filler' text? And why update the web site to say "The Challenge Continues"?

I am left slightly baffled, which I'd imagine is just how people working in secret places would like it to be.

Monday, December 05, 2011

Down the GCHQ rabbit hole (or I think there really is a part 4)

Update: Confirmation that this is correct

Early this morning I blogged that I thought there was a hidden part 4 to the GCHQ Code Challenge because of the amount of non-random data in part 2. Through someone who knows someone who knows someone at GCHQ I got the response the data was "a random filler and that they ran out of time to do anything interesting".

Well, I don't believe that's the whole story. There's a distinct pattern worth investigating.

Firstly, here are the three blocks of data that are untouched in part 2:
0150: 7d 1f 15 60 4d 4d 52 7d 0e 27 6d 10 6d 5a 06 56   }..`MMR}.'m.mZ.V
0160: 47 14 42 0e b6 b2 b2 e6 eb b4 83 8e d7 e5 d4 d9   G.B.............
0170: c3 f0 80 95 f1 82 82 9a bd 95 a4 8d 9a 2b 30 69   .............+0i
0180: 4a 69 65 55 1c 7b 69 1c 6e 04 74 35 21 26 2f 60   JieU.{i.n.t5!&/`
0190: 03 4e 37 1e 33 54 39 e6 ba b4 a2 ad a4 c5 95 c8   .N7.3T9.........
01a0: c1 e4 8a ec e7 92 8b e8 81 f0 ad 98 a4 d0 c0 8d   ................
01b0: ac 22 52 65 7e 27 2b 5a 12 61 0a 01 7a 6b 1d 67   ."Re~'+Z.a..zk.g

0200: 37 7a 07 11 1f 1d 68 25 32 77 1e 62 23 5b 47 55   7z....h%2w.b#[GU
0210: 53 30 11 42 f6 f1 b1 e6 c3 cc f8 c5 e4 cc c0 d3   S0.B............
0220: 85 fd 9a e3 e6 81 b5 bb d7 cd 87 a3 d3 6b 36 6f   .............k6o
0230: 6f 66 55 30 16 45 5e 09 74 5c 3f 29 2b 66 3d 0d   ofU0.E^.t\?)+f=.
0240: 02 30 28 35 15 09 15 dd ec b8 e2 fb d8 cb d8 d1   .0(5............
0250: 8b d5 82 d9 9a f1 92 ab e8 a6 d6 d0 8c aa d2 94   ................
0260: cf 45 46 67 20 7d 44 14 6b 45 6d 54 03 17 60 62   .EFg }D.kEmT..`b

0270: 55 5a 4a 66 61 11 57 68 75 05 62 36 7d 02 10 4b   UZJfa.Whu.b6}..K
0280: 08 22 42 32 ba e2 b9 e2 d6 b9 ff c3 e9 8a 8f c1   ."B2............
0290: 8f e1 b8 a4 96 f1 8f 81 b1 8d 89 cc d4 78 76 61   .............xva
02a0: 72 3e 37 23 56 73 71 79 63 7c 08 11 20 69 7a 14   r>7#Vsqyc|.. iz.
02b0: 68 05 21 1e 32 27 59 b7 cf ab dd d5 cc 97 93 f2   h.!.2'Y.........
02c0: e7 c0 eb ff e9 a3 bf a1 ab 8b bb 9e 9e 8c a0 c1   ................
02d0: 9b 5a 2f 2f 4e 4e 00 00 00 00 00 00 00 00 00 00   .Z//NN..........
I say three (rather than two blocks), because as you'll see below there's a real pattern that indicates that we are looking at three blocks of data encrypted using the same method and the same key.

The decryption scheme used for the other parts of part 2 (that actually get decrypted) is based on calculating the following where xn is the nth byte (0 based) of the block to be descrypted: f(xn) = xn xor ( a * n + b ). a and b are constants. The first decryption uses a = 1 and b = 0xAA. The second decryption uses a = 3 and b = 0x32.

Working under the assumption that a similar scheme was used for these three blocks I ran various tests and noticed a peculiar rhythmic pattern where whole runs of random ASCII characters would appear followed by runs of non-printable characters. It occurred to me that this could imply that (a) the underlying data is in ASCII (using printable characters) and that (b) the pattern I was seeing was a pattern in the top bit of key. If the key is following the scheme above it would go through runs of 0s followed by 1s and if the plain text is ASCII the top bit is always 0 and so for that one bit the 'key' comes through.

A quick calculation of just the top bits of the three blocks above showed the following pattern:
Immediately, it's apparent this this data is not random (at least in the top bit), and that it follows a pattern of 0s and 1s alternating with the following counts:
150: 20 (0s), 25 (1s), 26 (0s), 26 (1s), 15 (0s)
200: 20 (0s), 25 (1s), 26 (0s), 26 (1s), 15 (0s)
270: 20 (0s), 25 (1s), 26 (0s), 26 (1s), 15 (0s)
(Note that the last block is short because the data is truncated and ends with 0s). It also might indicate that there are three separate blocks of data encrypted using the same key.

Theorizing that that was generated using the f(xn) function above meant finding a and b that fit the bill. It's not hard to calculate this spotting that the 25 occurs when the 0xFF boundary is passed and that the 20 occurs because we start in the middle of a 26 block. The actual coefficents are a = 5 and b = 0x1F. That generates the following sequence to use as the key:

1f 24 29 2e 33 38 3d 42 47 4c 51 56 5b 60 65 6a 6f 74 79 7e 83 88 8d 92 97 9c a1 a6 ab b0 b5 ba bf c4 c9 ce d3 d8 dd e2 e7 ec f1 f6 fb 00 05 0a 0f 14 19 1e 23 28 2d 32 37 3c 41 46 4b 50 55 5a 5f 64 69 6e 73 78 7d 82 87 8c 91 96 9b a0 a5 aa af b4 b9 be c3 c8 cd d2 d7 dc e1 e6 eb f0 f5 fa ff 04 09 0e 13 18 1d 22 27 2c 31 36 3b 40 45 4a 4f 54 59 5e 63 68 6d 72 77 7c 81 86 8b 90 95 9a 9f a4 a9 ae b3 b8 bd c2 c7 cc d1 d6 db e0 e5 ea ef f4 f9 fe 03 08 0d 12 17 1c 21 26 2b 30 35 3a 3f 44 49 4e 53 58 5d 62 67 6c 71 76 7b 80 85 8a 8f 94 99 9e a3 a8 ad b2 b7 bc c1 c6 cb d0 d5 da df e4 e9 ee f3 f8 fd 02 07 0c 11 16 1b 20 25 2a 2f 34 39 3e 43 48 4d 52 57 5c 61 66 6b 70 75 7a 7f 84 89 8e 93 98 9d a2 a7 ac b1 b6 bb c0 c5 ca cf d4 d9 de e3 e8 ed f2 f7 fc 01 06 0b 10 15 1a 1f

Which has the required sequence of 0s and 1s and begins: 20 (0s), 25 (1s), 26 (0s), 26 (1s), 25 (0s). So, at this stage I've identified a pattern and a way to generate that sequence using the encryption scheme used in part 2.

The surprising thing that happens when you decrypt those three blocks using that key stream is that not only is it 7 bit (which was bound to happen because of the way the key was generated) but the entire thing is printable ASCII. No characters below 0x20 and no 0x7F. Here are the three decrypted blocks in ASCII (I added line breaks):

\*^[email protected],Dd=83;?e0bnP3R$ZF:V,L~O 5wS&[km?6x5M;7A+X-(^.?,%Ugu;O4x;"?
0%qjE)RcVax:/xsk}*.=u[\[email protected]/N7aHpA_$5H'LCW76XHtRA*krs|WZxu|U;

The fact that this decrypts as printable ASCII reinforces my belief that this is correct. Of course, it looks like gibberish, but it's printable. There are 89 characters used from the 95 possible. The missing characters are # 1 > G Q l.

It seems unlikely that the sequence of steps I have taken would have resulted in a block of perfectly printable ASCII characters had this not been the correct thing to do.

Perhaps this is a base-89 encoding of some other binary data. Whatever it is it has high-entropy and could be encrypted with some robust scheme, or it could be the result of compression. Of course, it could be that this data is random and was encrypted by GCHQ as I specified as a sort of dead end.

PS If anyone from GCHQ is reading... can you email me a simple 'carry on' or 'stop wasting your time'. Need to sleep...

Is there another GCHQ Code Challenge?

Update: The answer is Yes, sort of.

Last week GCHQ issued a 'code challenge' as a way of attracting candidates via a web site called Can You Crack It?. I did the challenge, which actually consists of three parts and has more to do with assembly language and reverse engineering skills than cryptography (in fact, the only time you come across a cryptographic function, you can completely ignore it).

The cat is completely out of the bag now because some spoilsport published complete details on the web showing how to break it, but I think there may be one more mystery worth investigating.

Part 2 of the challenge involves writing a virtual machine that executes some code to decrypt an in memory buffer containing an HTTP command. Here's a dump of the memory of the virtual machine after execution:
0000: 31 04 33 aa 40 02 80 03 52 00 72 01 73 01 b2 50   1.3.@...R.r.s..P
0010: 30 14 c0 01 80 00 10 10 00 00 00 00 00 00 00 00   0...............
0020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0090: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
00a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
00b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
00c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
00d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
00e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
00f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0100: 32 00 75 0c 31 08 33 32 40 02 80 03 52 00 72 01   2.u.1.32@...R.r.
0110: 73 03 b2 00 c3 b0 00 30 1b c0 01 ff 00 00 00 00   s......0........
0120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0130: 00 00 75 10 01 00 00 00 00 00 00 00 00 00 00 00   ..u.............
0140: cc 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0150: 7d 1f 15 60 4d 4d 52 7d 0e 27 6d 10 6d 5a 06 56   }..`MMR}.'m.mZ.V
0160: 47 14 42 0e b6 b2 b2 e6 eb b4 83 8e d7 e5 d4 d9   G.B.............
0170: c3 f0 80 95 f1 82 82 9a bd 95 a4 8d 9a 2b 30 69   .............+0i
0180: 4a 69 65 55 1c 7b 69 1c 6e 04 74 35 21 26 2f 60   JieU.{i.n.t5!&/`
0190: 03 4e 37 1e 33 54 39 e6 ba b4 a2 ad a4 c5 95 c8   .N7.3T9.........
01a0: c1 e4 8a ec e7 92 8b e8 81 f0 ad 98 a4 d0 c0 8d   ................
01b0: ac 22 52 65 7e 27 2b 5a 12 61 0a 01 7a 6b 1d 67   ."Re~'+Z.a..zk.g
01c0: 47 45 54 20 2f 64 61 37 35 33 37 30 66 65 31 35   GET /da75370fe15
01d0: 63 34 31 34 38 62 64 34 63 65 65 63 38 36 31 66   c4148bd4ceec861f
01e0: 62 64 61 61 35 2e 65 78 65 20 48 54 54 50 2f 31   bdaa5.exe HTTP/1
01f0: 2e 30 00 00 00 00 00 00 00 00 00 00 00 00 00 00   .0..............
0200: 37 7a 07 11 1f 1d 68 25 32 77 1e 62 23 5b 47 55   7z....h%2w.b#[GU
0210: 53 30 11 42 f6 f1 b1 e6 c3 cc f8 c5 e4 cc c0 d3   S0.B............
0220: 85 fd 9a e3 e6 81 b5 bb d7 cd 87 a3 d3 6b 36 6f   .............k6o
0230: 6f 66 55 30 16 45 5e 09 74 5c 3f 29 2b 66 3d 0d   ofU0.E^.t\?)+f=.
0240: 02 30 28 35 15 09 15 dd ec b8 e2 fb d8 cb d8 d1   .0(5............
0250: 8b d5 82 d9 9a f1 92 ab e8 a6 d6 d0 8c aa d2 94   ................
0260: cf 45 46 67 20 7d 44 14 6b 45 6d 54 03 17 60 62   .EFg }D.kEmT..`b
0270: 55 5a 4a 66 61 11 57 68 75 05 62 36 7d 02 10 4b   UZJfa.Whu.b6}..K
0280: 08 22 42 32 ba e2 b9 e2 d6 b9 ff c3 e9 8a 8f c1   ."B2............
0290: 8f e1 b8 a4 96 f1 8f 81 b1 8d 89 cc d4 78 76 61   .............xva
02a0: 72 3e 37 23 56 73 71 79 63 7c 08 11 20 69 7a 14   r>7#Vsqyc|.. iz.
02b0: 68 05 21 1e 32 27 59 b7 cf ab dd d5 cc 97 93 f2   h.!.2'Y.........
02c0: e7 c0 eb ff e9 a3 bf a1 ab 8b bb 9e 9e 8c a0 c1   ................
02d0: 9b 5a 2f 2f 4e 4e 00 00 00 00 00 00 00 00 00 00   .Z//NN..........
02e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
02f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
There are three colors used there to highlight areas: memory in blue was executed, memory in red was both read and written, memory in green was read, written and executed. As you can see the program starts running at 0x0000. There there's small program that decrypts the block of memory between 0x100 and 0x14F and jumps to 0x100.

The decrypted program decrypts the memory at 0x1C0 to reveal the HTTP command that is used to execute part 3 of the GCHQ challenge.

But what about all the rest of the unused memory? No other part of the GCHQ challenge wastes bytes, they are all used for something, but here there's a ton of memory that's filled with data and that isn't referred to anywhere else.

Is this a hidden fourth part of the GCHQ challenge? Perhaps even the real challenge?

If you look into the first block of data that's been decrypted there are two intriguing pieces of additional information. At 0x132 there are three bytes: 75 10 01. Those three bytes are actually valid code in the virtual machine. They mean
75 10   add   ds, 0x10
01      jmpr  r1
This seems like perfectly reasonable code. It moves the data segment forward by 0x10 and then jumps to R1. At the end of execution of the original program the data segment is at 0x1c. Thus moving it forward puts it at 0x2c (i.e. the block starting at 0x02c0 above). An alternative explanation is that based on the initial conditions of the VM that data segment would now be 0x20 (i.e. pointing immediately after the decrypted HTTP command at 0x0200).

Also at the end of running the program R1 contains 0x08. Doing the jump would drop straight into the decryption loop just past the initialization and into the XOR portion. That could be totally valid. The jump would take the instruction pointer to:
10:08 movm r0, [ds:r2]
10:0a xor r0, r3
10:0c movm [ds:r2], r0
10:0e add r2, 01
10:10 add r3, 03
10:12 cmp r2, 00
10:14 jmpe r3
10:15 cmp r0, 00
10:17 movr r0, 1b
10:19 jmpe r0
10:1a jmpr r1
The only other clue is that there's the byte 0xCC in the decrypted memory.

And one more thing. Notice the 10:14 jmpe r3. That never gets taken because r2 is never 00. But if you investigate the circumstances under which r2 would be 00 you find that it's when r3 would be 0x32 (i.e. when that jump would take you right to the 75 10 01 sequence that's been decrypted).

I haven't had enough free time to investigate this further. Perhaps it's a red herring, but it looks awfully suspicious. Especially given that the main loop will take that jump instruction when it's completely exhaused 0x100 bytes of data. This little 'subroutine' then moves the data segment on by 0x10 (i.e. 0x100 bytes) and the decryption will continue until a 00 byte is written. So it looks valid and is designed to cope with not having hit a 00 before the end of a segment.

PS It's been pointed out to me that the Can You Crack It? web site has been altered since the beginning to add the words "The challenge continues".

Wednesday, November 30, 2011

Downton Abbey Series 1 Episode 1 Morse Code

The very beginning of Downton Abbey's first ever episode begins with a close up of a Morse code key being used to send a telegram that will announce the critical news of two deaths. If you're like me you instantly wondered what was being keyed.

Here's what I hear:
dah                   T 
dah dah dah           O

di dah di dit         L
dah dah dah           O
di dah dit            R
dah di dah dit        C

dah dah dit           G
di dah dit?           R
So, it appears that the message begins: "TO LORC GR". I assume that is meant to be "TO LORD GRANTHAM" and that the C is actually D and has been incorrectly transcribed.

Here's the audio as an MP3: Downton Abbey Series 1 Episode 1 Morse Code (first part).

Then over loud music it continues:
dah dit               N
dah dah dit           G
dit                   E
di dah dit            R

di dah di dit         L
di dit                I
di di dit             S
dah                   T

di dah                A
di dah                A
di dah                A

di dah dah dah        J
di dah                A
I think that's a fragment of "NGER LIST AAA JA". Since AAA is a full stop and this telegram is referring to the sinking of the RMS Titanic where Patrick and James Crawley were on the passenger list (as we later learn in the episode) it's likely that this is a fragment of "PASSENGER LIST. JAMES".

Here's the audio as an MP3: Downton Abbey Series 1 Episode 1 Morse Code (over music).

Anyone else hear something different?

PS It's pleasing that the producers must have gone to the trouble of composing the complete telegram and the recording its transmission using Morse code just for that small segment at the beginning of the first episode.

PPS A little Google search reveals that it was G3YXZ [PDF] who was recruited to send the CW and he notes that for television reasons the key was adjusted to increase the gap size thus making sending more difficult (hence, no doubt, the mistake with C and D).

Getting a little tired of 'security researcher' bluster

There's been a flap recently about a company called CarrierIQ because of claims made by a 'security researcher' about the software. The software is installed on millions of smartphones around the world and (the company says) is used by carriers to look at network and phone performance by tracking usage.

Two articles have appeared based on this researcher's work. has Security researcher responds to CarrierIQ with video proof and Wired has Researcher’s Video Shows Secret Software on Millions of Phones Logging Everything.

Now, I'll admit that I find it worrying that my smartphone might be logging information about what I'm doing and sending it to some third party, and both articles make a very scary claim:
Wired: From there, the data — including the content of text messages — is sent to Carrier IQ’s servers, in secret. This video has demonstrated a truly significant volume of information is being recorded. Passwords over HTTPS, the contents of your text messages, and plenty more are recorded and sent to the customers of CarrierIQ.
That would be worrying if true, but if you watch the 'security researcher's' video you'll find that nowhere does he make the claim that content that the application sees is leaving the device. And from the video he doesn't appear to try. At no point does he enter a debugger and look inside the CarrierIQ application, and at no point does he run a network sniffer and look at what data is being transmitted to CarrierIQ.

And I don't understand why. It would be a huge story if millions of smartphones worldwide were secretly sending the content of text messages to a US-based company. But that's not the story here because the 'security researcher' does not appear to have tried to find out.

The story as told by the 'researcher' is that the CarrierIQ application gets called when keys are pressed, when text messages are received and when the web is browsed. What isn't delved into is what the application does with the information. Without that it's not possible to tell if there's something really scary going on or not. (I've not discussed the privacy policy implications here as I think that's a separate, non-technical issue).

And here's where I have a problem with 'security researchers'. The story here is a little too sensational and the researcher really needs to dig deep to get to the truth. We've seen this before with sensational claims that Samsung was installing keyloggers on all their laptops made by a 'security researcher'. It turned out to be rubbish.

I'd really like to see some real research into when the CarrierIQ application is doing with the information it is seeing, otherwise these claims about all my keystrokes being sent to some third-party company are just claims without any substance to back them up.

To quote Carl Sagan: "Extraordinary claims require extraordinary evidence". Let's see the evidence.

Tuesday, November 29, 2011

A nice bottle of Lorem Ipsum 2009

Going out to lunch with friends, having a nice bottle of wine in a gastropub. Good way to spend the weekend. Then you decide the wine was really nice and you'd take a quick photograph of the label so you don't forget it.

And then you notice the text on the bottle of Domaine Tissier Sancerre Rouge 2009. Yes, it's Lorem Ipsum filler text!

Interestingly it's the version that's popular in English speaking countries and not in France. I called the vineyard in France who were only able to tell me that the bottle of wine was genuine, that there's very little of the 2009 red left, and that it's only available in the UK to the restaurant trade.

So, if you want to get a bottle as a souvenir or Christmas present for the designer in your family you'll have to persuade a restaurant to part with the bottle.

Monday, November 28, 2011

Computer programming isn't driving or car maintenance

Today on BBC Radio 4 there was a segment about the importance of teaching computer programming in British schools. While I'm delighted to hear this being aired on such an importance program, some of the discussion was ridiculous.

Here's a key question:
John Humphrys: We don't need to know what goes on under the car's bonnet. Why do we need to know what goes on inside the computer?
This brings to mind the wonderful Charles Babbage responding to a question about his mechanical engines: "I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question."

The question somehow equates a car with a computer.

What the question completely overlooks is that the computer is not a machine like a car with a single purpose, it's a meta-machine. A machine that can masquerade as other sorts of machine. A sort of chameleon device that can be persuaded to do its programmer's bidding. In the words of Alan Turing: "a universal machine".

You can't imagine someone turning a car into a music player, a machine that beats you at chess, a camera, a word processor, or a communication device. A car does one thing: moves. You need to know how to operate the movement. Well, a computer does one thing: lets you program it to do things.

If you teach someone to operate a word processor (as is done in the UK's stupid ICT classes) you are not teaching them to use a computer at all. You are teaching them to use a word processor. It's a bit like teaching them to use a typewriter only this one's a bit more sophisticated. In fact, there's be far more outrage if the UK's current ICT classes were called what they actually are: secretarial skills (that's not to demean secretaries as I went to learn how to be a secretary so that I'd be able to touch type).

Humphrys went on:
Isn't there a worry as well, Alex Hope, that we're scared (an awful lot of us anyway) are scared of computers? They seem to be such mysterious objects to us. I mean we can sort of vaguely understand what makes an internal combustion engine tick.
But actually writing code is a massive mystery to most of us.
I'd wager that that's the sort of argument you'd get from people who've never tried it. It's a bit like speaking French. That's really hard (the subjunctive really is a massive mystery), yet we do teach it in schools. And guess what? You're more likely to find yourself communicating with a computer and have need for real computer skills, then speaking to a French person.

More than 20 years ago I taught the LOGO programming language to primary school children. They were able to get the concepts and write programs. Why is today any different?

Tell you what: give me an hour with John Humphrys and I'll get him programming like a primary school child.

Saturday, November 26, 2011

Why I'm not supporting the campaign for a pardon for Alan Turing

There's a new petition to the British government that's asking for a pardon for the convinction of Alan Turing. It says:
We ask the HM Government to grant a pardon to Alan Turing for the conviction of 'gross indecency'. [...] A pardon can go to some way to healing this damage. It may act as an apology to many of the other gay men, not as well known as Alan Turing, who were subjected to these laws.
Regular readers will know that I was the person behind the 2009 Alan Turing apology petition that resulted in the government apology. It's understandable that people have today been asking me what I think about this current campaign.

I think it's a mistake.

Firstly, in recent years the British government pardoned the soldiers who were executed for cowardice or desertion during the First World War. The pardon came about because of campaigning by the soldiers' families and because there was serious doubt that these boys and men were actually cowardly. They were most likely suffering from shell-shock and other traumatic stress. In Turing's case there's really no argument that he simply broke the law. There aren't any circumstances that change that. The law itself was awful (hence my campaign), but it's not clear to me that a pardon is appropriate.

Secondly, even if a pardon is appropriate, a pardon for simply Turing would be unjust to the other gay men who suffered under the law. There were many, many others. And there are men alive today living in Britain with a criminal record because of offenses committed during the time the laws were in force. I could get behind a petition for a pardon for all those people, especially since living people are still hurt by that law, but not just for Turing. Pardoning him doesn't help the living.

But even that's unnecessary. Subsequent to the 2009 apology campaign the UK government introduced legislation that actually does roll back the criminal convictions of gay men. The Protection of Freedoms bill has already passed all stages in the House of Commons, two readings in the House of Lords and enters (this coming Monday) committee stage. That means it's close to being law.

Chapter 4 of that Act specifically allows for the disregarding of convictions under the old law that was used against Turing. Once disregarded the law causes their convictions to be deleted. It's not quite the same thing as a pardon, but its effect is to lift the burden of a criminal record from these living men.

So, while I'm sure the current campaign is a heartfelt attempt to express utter outrage at what happened to Alan Turing, I can't support it.

Thursday, November 24, 2011

Anatomy of a CRUTEM3 bug

Back in February 2010 I spotted (with the help of a reader) a bug in the software used to generate the CRUTEM3 data from the Met Office. This was subsequently confirmed by the Met Office as having a small effect of overestimating the error bounds on the temperature trend.

Back then I wrote:
I have no idea why the correction given in this blog post by Ilya and I works: perhaps it indicates a genuine bug in the software used to generate CRUTEM3, perhaps it means Ilya and I have failed to understand something, or perhaps it indicates a missing explanation from Brohan et al. I also don't understand why when there are less than 30 years of data the number 30 appears to still be used.
I had no idea, because until today I had never seen the source code used for the generation of CRUTEM3. But the recent release of another batch of emails from CRU led to an intriguing web site that had been hidden from view, but publicly accessible if you knew the URL. In one of the emails a Met Office scientist called Philip Brogan proposes (on October 8, 2007) to release the CRUTEM3 source code having made a copy of it onto his personal site:
We could also put the source code online - I've copied the internal software and documentation to a temporary site so you can see it (http://xxxxxxxx/xxxxxx/xxxxxx/crutem3/xxxxxxx/ - please keep this URL secret) - this could be added to if that would help to damp the conspiracy theories. What do you think?
Amazingly, as reported here the 'secret URL' is still working four years later with a complete archive (apparently) of the CRUTEM3 code.

One particular file called contains the function that performs the 'station error' calculation I had looked at way back:
# Calculate the station error for a grid cell
sub station_error {

    my $Climatology = shift;
    my $Sd          = shift;

    # Climatology error
    my $Clim_error = 0;
    for ( my $i = 0 ; $i < scalar(@$Climatology) ; $i++ ) {
        if ( $Climatology->[$i] eq 'D' ) {
            $Clim_error += ( $Sd->[$i]**2 ) / 30;
        }    # Small error in data-based climatology
        elsif ( $Climatology->[$i] eq 'E' ) {
            $Clim_error += $Sd->[$i]**2 / 15;
        }    # Modest error in extrapolated climatology
        elsif ( $Climatology->[$i] eq 'W' ) {
            $Clim_error += 0.3 * $Sd->[$i]**2;
        }    # Largest error in WMO climatology
        else { die "Dud value for climatology flag: $Climatology->[$i]"; }
    $Clim_error /= scalar(@$Climatology);

    # Observations error
    my $Obs_error =
      ( 0.03**2 ) / scalar(@$Climatology)
      ;      # 0.03C per station observational error

    # Homogeneity error
    my $Hom_error =
      ( 0.4**2 ) / scalar(@$Climatology);   # 0.4C per station homogeneity error

    # Return the total error
    my $Total_error = sqrt( $Obs_error + $Clim_error + $Hom_error );

    return $Total_error;
The code is written in Perl (as was the code that the Met Office released for calculating partial CRUTEM3 results). It's satisfying to see this file because it answers my questions above and is a simple example of how errors happen in real code. I can easily imagine myself making a similar error.

First, the answer to my question "I also don't understand why when there are less than 30 years of data the number 30 appears to still be used." is simple. The number 30 is hard-coded. That's a small difference with the published paper.

The answer to why the correction (dividing the number 30 by the number of stations in the grid square) works requires following through the code and its related mathematics. Brace yourself from some forumlae. What I believe happened was that the programmer sensibly realized that there was no point doing all these square roots and then squaring the result and the forumla could be simplified as it was being turned into code. This sort of simplification is a good idea because it helps reduce floating point errors that might get introduced. But in the simplification process a term got lost.

Here's a review of the key terminology and the relationship with the code:

And here's a simplification of the formula that shows how to go from what's in the paper to what's in the code:

The $Hom_error and the $Obs_error are correctly calculated by the code, but the $Clim_error is not. The critical line is $Clim_error /= scalar(@$Climatology);. The line either needs to be repeated or changed to $Clim_error /= (scalar(@$Climatology)*scalar(@$Climatology));.

The effect is that rather than dividing by n^2 the formula is divided by n. And that explains what I saw all that time ago. It's as if the value of m (30 in the code) is divided by the number of stations n. If that's done the 1/n cancels one of the ns in n^2.

What had puzzled me at the time was why the programmer would have introduced 'dividing the number of stations by n'. He didn't, it's just that the effect is the same.

Monday, November 21, 2011

Beware the Alan Turing fetish

Tonight Channel 4 screens a wonderful new docudrama called Britain's Greatest Codebreaker about the life, work and death of Alan Turing. I was privileged to be invited to see it at a private screening at the BAFTA and was particularly moved and astonished by the scenes between Turing and psychiatrist Franz Greenbaum.

Leading up to tonight's screening a number of press articles have appeared about Turing, including a long one entitled Outcast who gave us the modern world in The Sunday Times. The article reflects a worrying trend in talking about Turing: a sort of Turing fetish.

Of course, I'm partly responsible for all this. Having campaigned for the public apology for the treatment of Turing which resulted in the 2009 government apology. What I wanted from that campaign was national (and, perhaps, international recognition) for Turing. That part worked, but we need to be mindful not to go too far.

The Sunday Times article says:
Had he lived, he might have been able to jump-start a new industrial revolution 20 years early — and in his homeland rather than 5,000 miles away in Silicon Valley.
It is fascinating to ponder what might have happened had he lived. Turing spent a couple of years in America in the late 1930s and showed no sign of wanting to stay. It is unlikely that he would have succumbed to the brain drain.
This brilliant, charming, odd, driven workaholic could have turned the old industrial heartlands of Lancashire into a British Silicon Valley and perhaps America’s brightest and best would have flooded east across the Atlantic.
The problem with this line of thinking is that it overlooks the realities of the size of the computer market in the UK and US, the invention of the transistor at Bell Labs, the incredible power of the military-industrial complex and it denigrates those who did work in early computing in the UK. It also overlooks the fact that post-World War II Britain was in serious dire straits: heavily indebted to the US and Canada (a debt Britain didn't finish repaying until 2006) and with its economy tattered.

Sure, Turing was damned important (I wouldn't have campaigned for recognition if I didn't think so), and his contributions to computer science, AI, code-breaking and morphogenesis were massive. But to think his death is somehow why Silicon Valley isn't in Britain is a mistake.

I talked this over with a scientific historian the other day and he made the point that the US computing market (both private and military) was massive at the time Turing was alive. By 1950 IBM had over 30,000 employees and greater than $250m in revenue. The combination of a massive home market and enormous Cold War spending (for example, on SAGE) meant that the US was running fast in the computing game.

At the same time an Alan Turing fetish means we might overlook the other great people who worked in early computing: Tommy Flowers was the eminently practical man who built Colossus; Maurice Wilkes was the man behind EDSAC. And what of British computing projects such as the Pilot ACE, LEO, and Manchester Baby and Manchester Mark I?

The LEO was up and running while Turing was still alive. And it was a business computer running in the UK in 1951.

It's a simple, and tempting, story to say that had Alan Turing lived that Britain would have been years and years ahead (and perhaps so far ahead to have beaten the US), but I think it's a fantasy. The Sunday Times article even goes so far as to suggest that Turing might have emerged as the leader of a British Google or Apple. The article posits that Turing would not have moved to the US.

This Alan Turing fetish extends outside computing. His work on Enigma overshadows the amazing work on cracking the Lorenz cipher which led to the Colossus machine. The cracker of that cipher, Bill Tutte, did leave the UK (in 1962) for Canada.

Let's put a halt to the fantasy and focus on the reality: Turing was a brilliant man and we should celebrate that and the work that he did do. The work he did was amazing enough without burdening his legacy with typical British bemoaning of our lack of megainfluence in the world (in this case, in the world of computing).

So, watch tonight's program: it's good television.

PS A reader points out that The Sunday Times overlook the importance of the British company ARM. Very true, few people outside computing have heard of the company, but they probably have an ARM processor in their pocket.

Thursday, November 17, 2011

The Lack of Affordance Problem

Yesterday, I complained about the lack of an eject button on an Apple USB DVD drive. That design philosophy problem goes much deeper than just a single drive. It pervades Apple products, and, although I love them, it drives me spare!

If I could say one thing to Jonny Ive it would be: "In designing to make the every day use of Apple's products a wonderful experience you're missing the moment when your users need you most. It's when everything goes wrong that the user needs the maximum help. It's in those moments (of despair) that a well designed product will surprise and delight the user."

The DVD drive is an example of the opposite. When it's working it's lovely to look at, but when it goes wrong it's as enigmatic as the 2001 monolith.

The mystical incantations necessary to remove the DVD from the drive (such as the hold down the mouse button while rebooting your machine) have to be passed on from the Apple elders in secret whispers (or found by a desperate Google search). Surely, there's a better way? The DVD drive could have had a discreet eject button. And Apple software could be better.

Take the MacBook Air. If it's gone wrong and I need help I'm going to need the serial number. How do I get it? There are (at least) two ways. First it's printed on the bottom of the machine in tiny type printed in almost the same colour as the case. It's as if the designer wanted to make it hard to read. For goodness sake it's on the bottom of the machine, it wouldn't have killed you to use bigger type, or a different colour.

The second way is in software. But knowing that requires more mystical knowledge and reveals something truly nasty. Here's the dialog you get by clicking Apple -> About This Mac.

If you click on the grey text that says Version 10.6.8 the text changes, first to the build number and then to the serial number. There's is no indication that this will happen. The cursor doesn't change, it's normal text that the user wouldn't expect to see change like this. (You can also click More Info... to get the System Profiler where you can find the same information).

Why do this Apple? Why not let me know this is clickable? What's needed is an affordance. Something that lets the user know that this is interactive, that leads the user to the right action.

This situation gets worse with touch screens. Because there's no 'hover' possibility you are left guessing which inscrutable user interface element might be interactive, or what gestures are possible.

I swear that one day the iPhone will do something interesting if you take the Tai Chi "Carry Tiger to Mountain" position.

And don't even get me started on the abomination that is the Apple keyboard USB extension cable fiasco which is a USB cable with a special little indentation in the USB plug so you can't use it for anything else. That just feels like that designers sat around laughing at the purchaser.

And then there's the problem of video connectors.

PS A commentator on Hacker News pointed out that the USB specification prohibits extension cables and so although millions of the things exist, Apple probably did the right thing by making the extension a non-USB cable.

Wednesday, November 16, 2011

How to forcibly eject a CD/DVD from a MacBook Air USB SuperDrive

So you've got a DVD stuck in the external USB SuperDrive that connects to the MacBook Air? And you've tried hitting the magic Eject button on the keyboard to no avail?

And you've tried drutil tray eject at the command-line? And you've tried booting with Option held down and clicking eject? And you've tried plugging the drive into other machines (Macs, Windows PCs, Linux machines) to no avail?

What do you do?

Clearly you hit the physical eject button, right? Every DVD drive has one of those. On the SuperDrive you first look for it on the front:

Hmm. Nothing there. Perhaps it's one of those pinhole things on the back.

Or on one side:

Or, heart in hand, the other side:

I know, I know. Jonny Ive hid it on the bottom out of sight, right? It's so obvious now. Put a pinhole on the bottom where it will only be seen when needed. Form and functionality in one. Sleek lines when in use. An accessible emergency button when needed. Genius.

Nope. But, phew, at least there's an Apple logo there. I was beginning to worry.

So, the easy procedure for removing the disk is as follows. First you pry open the case with some thin available tool:

Then you remove the six screws holding the drive into the case:

Then you turn the drive over in anticipation of seeing your DVD only to be thwarted by another cover and a set of screws:

And then, finally, you peel back the cover to see your DVD and remove it.

Once reassembled the drive complains about the missing disk and promptly goes back to working normally.

And then you wonder to yourself. The drive inside the case is a Sony NEC Optiarc AD-5630A. And every other drive that Sony sells has a physical eject button, even the slot loading ones. Did Apple hide it? Did they ask Sony to not provide a button? I'm guessing that the Cypress Semiconductor microcontroller inside there knows all about doing an emergency eject.

Of course, you stop worrying about that and wonder how the cable has managed to fray on this barely used drive:

PS Some people have pointed out that there may be a secret lever inside the drive accessible from the front by sticking a thin screwdriver inside. I can't find any reliable mechanism (although jamming the DVD in place using a wad of paper and rebooting does work).

The Great Manual Repository

Why aren't all product manuals released under some Creative Commons-style license on the web in PDF form and archived in a Great Manual Repository? If you go looking for the user or installation guide for a TV, a washing machine, a digital camera you're faced with either navigating a particular vendor's web site (they're almost all awful), or trawling through a myriad of sites whose design is fetid and that either end in a demand for money or an endless list of links to other sites.

It's time this came to an end. What's need is The Great Manual Repository. It shouldn't be hard, and it should be possible to do it in a way that has low cost. After all, the manuals are already being produced.

Here's what would be cool:

1. A web site to which manufacturers release manuals under a suitable license. There could even be an embargo if the manufacturer wanted to limits its competitors' access to manuals for a period (say manuals would be released one year after the device).

2. The web site organizes the manuals by suitable categories and, of course, allows free text search to quickly find a manual without forcing you into the endless drop-down menu forest manufacturers seem to love.

3. The manuals are made available for free download as PDFs (not some in browser nightmare technology like scribd). Perhaps the site covers its cost by selling printed manuals, or by offering product spares, or advertising.

4. The web site is designed to be usable with the minimum of fluff.

I'm sure that someone could build this pretty easily. Storage is cheap and technically it's not complex. All it would take is someone with the energy to explain to Miele, Sony, Phillips, Kenwood, etc. why they should release their manuals.

Surely there's a 'start-up idea' in there?

"The new standard for product demos is that we cry"

That's what one leading VC said to me during a company pitch at Accel Partners. Of course, he didn't mean that he'd literally descend into a Steve Jobs-style blubbing session if the product was good, but he had a serious point: gone are the days where poorly written slides, or a roughly put together product, were good enough for a top-tier VC. If you're pitching at a high level then you need to show you can compete, and in today's market that means being and looking sharp.

Apple, of course, set a very high-standard for devices and software and it's had a trickle down effect. Now that everyone expects their hand-held device to look gorgeous and work without friction they expect applications and web sites to do the same.

In fact, any application or web site that doesn't look like it's had the benefit of an experienced UX designer work on it is cannon fodder.

Of course, VCs still look for the quality of the team, its pedigree, deep or interesting technology, but the bottom line is that the overall standard for software and user interface in particular has gone up. And it's not just lipstick on a pig either, software has to be easy to navigate and above all productive for the end user.

Cheap apps and free web services mean that switching is easy and to compete your product or service needs to make your end users cry with joy.

Tuesday, November 15, 2011

A Christmas wish list wish list

I'm very uncomfortable with online (or other) wish lists for presents. Whenever I see a Christmas wish list, a baby shower registry, or a wedding list it makes me uncomfortable. The reason is that wish lists take all the pleasure out of choosing and buying a gift for a loved one and turn it into a financial transaction devoid of almost any loving meaning.

What I'd like to see is a new type of wish list. A general wish list that expresses things that would please me while allowing people the space to choose personal gifts. For example, my wish list might contain:
1. A lightweight camera tripod. Something I can easily carry about, but that puts the camera about a metre off the ground. Should have a pan and tilt head.

2. Poetry. I like Sir John Betjeman, e e cummings, Oscar Wilde and many of the war poets. Make me discover something new.

3. A book about ancient history. I don't know enough about the Greek civilization.

4. A cover for my iPhone. It's a 4S. Find something that doesn't spoil the phone's lines but will protect it from being shoved in my pocket.

5. Waterproof cycling clothes. Something I can wear over existing clothing. Perhaps waterproof or resistant shoes.
Compare that to a similar list that contained specific items from specific shops. With my general list a loved one can see things I might like and do their own search and discovery taking into account what they know about me. Or they can read the list and by inspired to buy something else.

The gifts I best remember are ones that people surprised me with because of the thought that had gone into them. A new wallet that was carefully chosen and replaced one that I loved (because it had been stolen and returned) but that had reached the end of its days. A tea set from my parents that closely matched the sort of style I would have bought. Those presents express affection through thoughtfulness. The best presents come from observing a loved one and seeing the things that they don't know that they want, or that they wouldn't ask for. My wish list helps that process along.

Imagine how different it would have been if I had chosen my own presents and presented a wish list.

With a specific list it becomes an exercise in "which of these can I afford?" rather than "what would please this person and make them happy?". By restricting the freedom to choose to a specific list of items, the loved one's ability to express love or thoughtfulness is severely restricted. It's only slightly better than sending money to choose from a wish list.

So, who's going to fulfil my final Christmas wish?
6. An online wish list service that's built around general descriptions of items (perhaps with examples from online shopping sites) with all the usual features for marking items bought etc.

Monday, November 14, 2011

Leaving comments in real life

As a programmer I like to write comments because I know that some future person (often me) is going to need to know something about my code that won't be immediately obvious from reading it. Here's a recent example from my homebrew display's code:
// protocol_init: Set up a newly powered-on string of lights.  The lights are arranged
// in an array wired diagonally starting from the bottom left.  For ease of programming
// it's better if they are given numbers which correspond to coordinates.  Given that
// each LED has a 6 bit address it's possible to address the entire array using 3 bits
// for X and 3 bits for Y (the 50th LED will be (7,0) and the display itself will be
// numbered starting (0,0) up to (6,6).
// For example, the first (single LED) diagonal is (0,0), the second diagonal is (1,0) and
// (0,1), the third is (2,0), (1,1) and (0,2)... the loop inside this function expolits this
// to program the LEDs.
// The actual display is wired as follows 
// 21 33 34 42 43 47 48
// 20 22 32 35 41 44 46
// 10 19 23 31 36 40 45
// 09 11 18 24 30 37 39
// 03 08 12 17 25 29 38
// 02 04 07 13 16 26 28
// 00 01 05 06 14 15 27 
// The initializer sets these up with the following addresses (corresponding to X in the bottom
// 3 bits and Y in the top 3 bits)
// 06 14 22 30 36 46 54
// 05 13 21 29 37 45 53
// 04 12 20 28 36 44 52
// 03 11 19 27 35 43 51
// 02 10 18 26 34 42 50
// 01 09 17 25 33 41 49
// 00 08 16 24 32 40 48
// Thus the order in which the (x, y) coordinates are programmed is:
// (0, 0), (0, 1), (1, 0), (2, 0), (1, 1), (0, 2), (0, 3), ...
// Also notice that the sum of the coorindates in each diagonal is related to the direction
// of wiring: even diagonals are wired upwards, odd diagonals downward.
// Initially all the LEDs are set to RGB color (0,0,0) with no brightness (i.e. they are 
// each sent an address and told to be off).
void protocol_init()
  digitalWrite( DISPLAY_PIN, LOW );
  // This is done so we see that the bus is low before we do anything on it
  delay( 1000 );
  // See comment preceding this function for explanation of this loop
  for ( int sum = 0; sum <= 12; ++sum ) {
    if ( ( sum % 2 ) == 0  ) {
      int x = (sum < 8)?0:(sum-6);
      for ( int y = sum - x; y >= 0; --y, ++x ) {
        if ( x < 7 ) {
          init_led( x, y );
    } else {
      int x = sum;
      for ( int y = 0; ( y < 7 ) && ( x >= 0 ); ++y, --x ) {
        if ( x < 7 ) {
          init_led( x, y );
This works well in code, but I've found it also works well in real life. Whenever you spend any time debugging something, or figuring out how something works, leave a note. Leave a physical note. You'll appreciate it years later when you discover you've documented your home wiring, plumbing, plants in the garden, etc.

In one of my old apartments any person who opens up the cable TV wiring will find little notes attached to the cabling using paper luggage tags. The notes detail which cables link to which boxes in the apartment, and why they are connected or not.

I've done something similar with plumbing (especially leaving notes about which stopcocks isolate different parts of the plumbing system) and low-voltage lighting. It's also useful to leave yourself (and future home owners) notes about things that are useful to know (for example, when a particular piece of equipment was last serviced). There's a note attached to the water filter of my washing machine with the last date I cleaned it.

I do this because (a) I'll never remember the details of plumbing, electrics, etc and (b) there's nothing like a note left close to the thing you're investigating.

One day, I suppose, it'll be possible to use an app-for-that, to leave virtual notes. But until then, and perhaps afterwards, there's nothing like a little hand-written note or diagram on paper.

Sunday, November 13, 2011

Back from the dead with a power supply repair: my BBC Micro Model B

A couple of weeks ago I plugged in my BBC Micro Model B and it went up in a plume of smoke.

Unfortunately, the 30 year old machine had undergone one of the most common failures. A capacitor that had been sitting there all these years failed, burst open, emitted smoke and leaked all over the circuit board.

Today, I repaired the damage. This blog post is for anyone who faces the same thing.

To fix the damage I replaced three capacitors that are likely to die over time (including the one that had burst) using a kit costing £2.70 from this site and a copy of the BBC Microcomputer Service Manual.

The kit has one electrolytic capacitor (which is polarized and has to be inserted the right way round) and two ordinary capacitors which can be inserted in any way. The three capacitors to be changed are marked C1, C2 and C9 on the circuit board. On my machine it was C2 that had failed.

The first step is to remove the power supply from its case. This is carefully described in the service manual (including two cable ties that need to be cut and replaced). Here's the supply without its case. I've marked the three capacitors to be replaced.

And here's the reverse of the circuit board (clearly hand soldered) and I've marked the six joints that have to be desoldered. I used my normal soldering iron and a solder sucker to remove all three capacitors.

And here's a shot of the three capacitors after they've been removed with a close up of the ruptured C2.

With the capacitors removed you can see the capacitor juice that had leaked onto the PCB. I cleaned it off with some alcohol and Q-tips. You can also see that C2 actually had four holes drilled for it. Two were used for the supplied capacitor and the other two are used in the repair as the new capacitor has leads that are slightly closer together (good of Acorn to plan 30 years ahead!).

Soldering the three in is not hard (make sure the electrolytic is the right way round). Here's my finished circuit board:

And the reverse side showing the three new capacitors in place:

And here are the two cable ties that need to be replaced. One holds the low voltage cables going to the main PCB in place (it's attached to the power supply casing), the other holds the mains cables in place (it's attached to the power switch).

And putting it all back inside the case... it works!

Monday, November 07, 2011

Turning GE Color Effects G-35 Christmas Lights into a 7x7 display with Arduino

Some time ago I saw some GE Color Effects G-35 Christmas lights and was fascinated by the combination of color patterns that they were capable of and the fact that they only had three wires. That meant that something digital was going on.

Happily, a chap called Robert Quattlebaum did the hard work last year of hacking these lights to reveal that they are linked by a simple self-clocked serial protocol that can easily be driven by a microcontroller. That blog post gives the full details of the protocol.

Each lamp can be set to a specific brightness and an RGB color (4 bits per color). And the protocol is clocked at 30µs per bit so it's possible to refresh the entire 50 LED string at a rate of 24Hz. Perfect for hacking fun.

So I got a set and plugged them in just once (still in their packaging) to check that they worked. Here's a video of the lights getting their one real work out before the soldering iron got to them.

The first thing I did was remove the lovely plastic blub covers (these are being saved to brighten up a simple set of Christmas lights) to reveal the LED inside. This picture shows all three states: one has the original plastic cover on, the one in my hand has it removed but the plastic base intact, the bottom one is the naked LED and controller.

Once you've ripped off the protective plastic you are down to a simple LED with three wires going in (+5V, Data and GND) and three coming out. With 50 of these on a string I decided to make a 7x7 display (with one LED 'spare' should I break one).

Here's the finished display. The entire chain of lights has been cut up, mounted, resoldered and is now controlled by an Arduino Pro. All that's needed is a 5V power supply to plug into the power socket.

It's controlled by a small piece of Arduino software that contains a simple frame buffer implementation, code to implement the LED protocol via bit banging, code for scrolling text and for making faces. You can get the source code here. The key file is protocol.cpp which initializes the display and allows simple setting of the color and brightness of each LED. The most interesting part of that code (at least to me) is the initialization of the LED addresses. On power up the LEDs wait to learn their 6 bit address. They wait in turn and have to be programmed for use. I exploit this to make addressing them easy:

To build this I first cut up the entire string of lights to get just the LEDs with a small amount of cable attached to each. Then I prepared a piece of plywood, marked it up, drilled guide holes and then drilled the 49 mounting holes for the lights.

Then I glued in place each of the lights taking care to orient them for easy soldering into the chain. Then just 144 solder joints and little bits of heat shrink later I had the chain reattached and capable of being driven by the GE supplied controller.

The next step was some sort of opaque diffuser to put in front of the display. For this I used a cutting board that I cut to the right size. With the board in various positions it's possible to get different diffusion effects.

The final step in the hardware was to get a picture frame made to measure (I got a cheap, but deep, one in simple white wood from these folks) and then stuck the cutting board inside, used some old nylon nuts as spacers, mounted the LED board and finally used a piece of MDF that was supplied with the frame as a backing. The back is fixed on using some parts from an old servo.

Electronically this modification to the original lights was pretty simple. Having eliminated the original power supply and controller all that was needed was a new controller in the form of an Arduino Pro and a 5V power supply. The supply provided with the lights gave 5V at 3A (which seemed at lot). I measured the actual power usage and the highest current seen was 1.68A with all LEDs on white at maximum brightness. Thus I used an off-the-shelf power supply for the lights (plus Arduino Pro) capable of providing 2.25A.

The Arduino Pro is screwed to the back of the picture frame and there's only one component added: a 10K pull down resistor between GND and pin 7. Pin 7 is driving the serial data through the LEDs and it is low when not being driven with data. You can see that here:

If you're eagle-eyed you might have spotted a capacitor as well. I got this Arduino Pro free because it had a manufacturing fault and I repaired it using an old capacitor that I'd desoldered from the control board of a dead toaster. In that photo the Arduino Pro is connected via an FTDI cable for programming.

The other slightly tricky thing with this project was getting the timing of the serial protocol right. The simple serial protocol uses alternating high/low pulses of 20µs and 10µs (or low/high). In protocol.cpp I've made use of C #define statements to inline code for speed and also third-party code for fast changes to the I/O ports and accurate delays.

When working on this I used the Salae Logic analyzer to view the actual serial protocol in action. Here are two screen shots. The upper one shows the standard GE controller sending the initialization packets to the each of the LEDs in turn (with a gap of 40ms between them), the lower one shows a zoom into a specific packet being sent.

Finally, here's a video of the complete unit running through a simple "Hello World", followed by a few facial expressions and finishing with a "Thanks GE" for such a nice hackable string of Christmas lights.

PS If anyone from GE is reading this... how about making a 220V version of these? All you need to change is the wall wart to one that does 110-220V and you'd have all us Europeans interested in these lights.

PPS Other Arduino-based projects you might enjoy: GAGA-1: High Altitude Balloon Project and Cansole: Video games console in a can. Other non-Arduino projects: Revealing the secrets of the Ikea Lillabo wooden train set and Building a 'Ponyo' boat.