Tuesday, July 25, 2006

The Open Source Elephant in the room: the source sucks

Firstly, since I'm about to slam Free and Open Source Software let me point out that I am not a Microsoft fan boy, that I use Firefox, Thunderbird, Ubuntu, EMACS, all the GNU tools, the GIMP, Apache, TRAC, OpenOffice, etc. on a daily basis. I live and work on FOSS.

But there's a real problem with most FOSS: for something that prides itself on the source being readable by everyone, and even cames up with 'laws' like 'given enough eyeballs all bugs are shallow', the actual source code of most FOSS is horrible, unreadable, garbage. Actually, I wonder if 'Linus' Law' shouldn't actually be something like 'Linus' Necessity': given that the source is so horrible we need lots of people so that one of them will be able to figure out what the hell it was we wrote.

When I started my well-known open source project I decided that I'd better make the code readable for two reasons: firstly, I was sure that I wouldn't get to work on it often so I'd have to come back and read old code and comments and other coding standards would make that easier; secondly, I was sure that other people were going to read my code.

The second thing turned out to be really important for two reasons: firstly, other people were able to read my code and contribute and I kept them to a similar coding standard and style and hence the code is (reasonably, I'm not claiming I'm perfect) readable. Perhaps more importantly one day I was being interviewed for a job and the interviewer said: "Yes, we've all read your code". They'd downloaded my project and checked me out. (I got the job).

Now, I'm not trying to slam all FOSS here and for the purposes of this entry I have not examined some of the most famous projects (e.g. Linux kernel, Apache, Firefox, ...), but I decided to take a look at the top 50 most downloaded projects of all time on SourceForge.

Then I would pick at random two source files (each source file had to be fairly large, i.e. more than 100 lines of code) and score them being as generous as possible using the following categories and assigned a score to each. I weighted the scores heavily towards doing simple things that have a high benefit (for example, describing the purpose of a file of function):
  • File Description (FD): did the file I open have some sort of description (near the top) of what the purpose of the file was for. I wasn't asking for a detailed explanation, but just a little helper so that a new reader could get going on the purpose. Score: +5 (if present), -5 (if not)
  • Function/Interface Description (FID): did any of the functions, or interfaces, in the file have a description. I would have liked to have seen all the arguments specified and return codes and caveats explained, but I was extremely generous: even if one function had a little header with a minimal description of the function it got into this category. Score: +5 (if present), -5 (if not)
  • Useful Comments (UC): did the file contain at least one useful comment. A useful comment points out something that isn't obvious to the reader, or some trap for the unwary. Score: +1 (if present), -1 (if not)
  • Stupid Comments (SC): did the file contain at least one stupid comment like 'increment i' or 'loop through records'. Score: -1 (if present), +1 (if not)
  • Understandable (U): did I feel like I would be able to understand most of the code given 30 minutes of reading the file and browsing the rest of the source. This was very subjective, but was used to take into account things like clearly named functions, or really well named member variables. Score: +5 (if understandable), -5 (if not)
  • Commented out code (COC): people we have source code control systems. Don't // out your code, or #if 0 it. ok? Score: -1 (if present), +1 (if not)
  • Bonus (B): I had a special bonus category which I could hand out if I felt like it. A positive score here was for particularly well documented, and written code, neutral for most code and negative for really hideous stuff. Score: +10 (loved it), -10 (yuck), 0 (in general)
Of the top 50 projects one (XAMPP) was excluded because it's a distribution of other code and not new code.

What I found was not a pretty picture:
  • 65% don't bother describing even in the most minimal way even one of the functions I saw
  • 60% of the projects don't bother with describing the purpose of a file
  • 59% of the projects scored negatively using my system
  • 53% contained useless comments
  • 40% looked incomprehensible to me without major effort
  • 33% contained commented out or #if 0 code
There was one bright spot: 85% contained at least one useful comment. But given that my percentages underestimate the problems (because I was very generous) these figures are horrible.

The best projects were (in order of score): GNUWin32 (thanks GNU Project!), GTK+ and The GIMP installers for Windows, NASA World Wind, Ghostscript, WINE, Miranda, MinGW (thanks GNU Project!), Erases, and DC++.

Come on FOSS people. Have some pride in your work! Remember, writing some decent comments is a gift you are given to people who read your code, and to yourself.

(Note that if you are the author of one of the projects above it's possile that I made a mistake and just happened to pick the wrong files to read. Send me examples of how great your code is and I'll publish a rebuttal here).

Here's a table with all the data:



















































ProjectFDFIDUCSCUCOCBScorePop.
eMule-1-111110-61
Azureus -1-111110-62
BitTorrent -1-1-1-1-1-10-143
DC++ 11111-10164
Ares Galaxy 1-11-1-1-10-25
CDex -1-111110-66
VirtualDub -1-11-1-1-10-127
Shareaza -1-111110-68
eMule Plus 1-11-111069
GTK+ and The GIMP installers for Windows 111-11-112810
7-Zip -1-11-1-1-10-1211
FileZilla -1-1-1-1-1-10-1412
guliverkli 1-1-11-1-10-613
Gaim 1-11-11-10814
Audacity 1-11-1-1-10-215
phpBB -1-1111-10-416
ZSNES -1-11-1110-417
TightVNC -1-111110-618
phpMyAdmin 1111-1-10619
NASA World Wind 111-11-101820
ABC [Yet Another Bittorrent Client] -11111-10621
Dev-C++ -1-11-11-10-222
aMSN -1111110423
ffdshow -1-1-1-1-1-1-1-2424
WinSCP -1-11-1-1-10-1225
JBoss.org 111-1-1-10826
AC3Filter -1-111-1-10-1427
Ghostscript 11111-101628
[email protected] -1-111-110-1629
Webmin 11-1-1-1-10630
PDFCreator -1-111-1-10-1431
VisualBoyAdvance -1-11-1-110-1432
MinGW - Minimalist GNU for Windows 11111-101633
eMule Morph -1-111-110-1634
GnuWin32 111-11-112835
The CvsGui project 11-111-101436
FlasKMPEG -1-11-1110-437
VirtualDubMod -1-11-1-110-1438
Gallery 1-11-1110639
DOSBox DOS Emulator -1-1111-10-441
Miranda 11111-101642
DScaler Deinterlacer/Scaler -11111111443
Celestia -1-11-11-10-244
PeerGuardian -11111-10645
XOOPS Dynamic Web CMS -1-11-11-10-246
Eraser 111-11101647
Wine Is Not an Emulator 11111-101649
burst! -1-1-1-1-1-10-1450

9 comments:

Erik L. said...

Hi-

I don;t contribute to an open sourceproject but I do have my own long running project that I may open source someday. I looked through your list and I think mine is pretty well commented per your stadards-

two complaints though- 1) Deducting for stupid comments makes no sense unless they are excessive 2) Commenting out code is a perfectly valid technique and I do not agree with the "no commenting out code" fetsih some people seem to have. Yes, we have version control but you have to think to look at the old version. Some commented out code is instructive and some commented out code is a warning not to try the sa,e thing again. Eventually one should get rid of it if it doesn;t serve the previous mentioned purposes.

HHM said...

You should see the Ogre 3d engine; it's probably the project with the greatest quality code I've ever seen. It would surely get an interesting score in your system.

anders said...

Did you have some kind of point you were trying to make or a hypothesis that you were testing?

About all I could deduce from your data is that there isn't a correlation between code quality and usefulness. That shouldn't really come as a surprise to anyone. So if you were trying to make the argument that code readability is unimportant for the success of open source projects, your data would support that.

Pretty much all the proprietary code I've ever seen would have scored even worse by your standards. So I can't really see this as any kind of indictment of open source software quality.

(I'm going to back you up on the "no commenting out code" thing though. That drives me *nuts*.)

[email protected] said...

AMEN!

|2dman| said...

Regarding stupid comments, I don't know if you should be so harsh. Two things spring to mind:

(1) When designing an algorithm, then writing the code, some write out the stages of their algorithm in a series of comments, then fill out the code in between. I like reading this kind of code, because you can see the structure of the algorithm separately from parsing the code -- and this often includes lines like "for each record..."

(2) When looking at a cross-linguistic project in a language with which I am not particularly familiar, it can be very helpful to have a stupid comment here and there so you can spot idioms. This isn't a huge advantage, since most of the stuff you probably looked at is in C, but I read a lot of code in other languages, some very niche languages, and it's helpful to find a dumb comment now and then.

-2dman-

Curt Sampson said...

I'm not sure that this is terribly fair. First of all, as someone else mentions above, most commercial or proprietary code is just awful, too. In fact, overall, I'd say that Open Source code is in general better than most proprietary code.

Second, you're ignoring cultural issues with coding. In an extreme programming project, for example, comments are generally considered to be a bad thing, even the "helpful" ones. But that's because the reaction of an XP developer, upon seeing a comment, is, "How can I change the code to better express its ideas so that the comment is not necessary." Because it avoids comment obsolescence, this is a better approach than using comments to describe code, yet it would be penalized by your count.

Anonymous said...

Sorry to disagree with the statement "proprietary code is worse".

I have worked for several startups, and IBM and Microsoft in the past (yeah, go on, booh me). This includes OS work, tune-up software, codecs and misc UI work.

In all companies (yeah even MS) there was an agreed-to coding style that (for example) INCLUDED a short header for EVERY function, labelling the IN/OUT parameters and use of comments for non-intuitive code.

Sorry to burst you guys' bubbles, but you must have worked in some pretty crap companies with proprietary code.

The FOSS projects I have looked at (GTK+ and others are some of the exceptions) would have resulted in an eventual dismassal as work that was "unsatisfactory".

Anonymous said...

Good code is its own best documentation. As you're about to add a comment, ask yourself, 'How can I improve the code so that this comment isn't needed?' Improve the code and then document it to make it even clearer.

Steve McConnell Code Complete

era said...

Thanks for a useful and thought-inspiring analysis. If you really spent at least 30 minutes on each of those projects, this has been a rather big investment on your part as well -- thanks for that. (I'm sure you are perfectly able to disregard this "anders" kook who apparently doesn't immediately recognize the usefulness of this exercise.)

I have to say, though, that I would have picked slightly different biases. How about this set of scores:

File description: +/- 3
Function / interface desc: +/- 3
Useful comments: +/- 3
Stupid comments: +1 if not present, 0 if present
Understandable: +/-10
Commented out: +1 if not, 0 if yes
Bonus: as before

BTW your result table doesn't add up, or I don't understand your notation at all. If for example eMule gets -5 for FD, -5 for FID, 1 for UC, 1 for SC, 5 for U, 1 for COC, and 0 for B, the sum should be -2, not -6. Then BitTorrent gets -18, Dc++ gets 16 (correct), Ares gets 0 ... Is this correct?

Provided, then, that I have interpreted the raw data correctly, the correct sums are:

ffdshow: -18
BitTorrent: -17
burst!: -17
FileZilla: -17
guliverkli: -15
7-Zip: -7
Ares Galaxy: -7
Audacity: -7
VirtualDub: -7
Webmin: -7
WinSCP: -7
AC3Filter: -5
Celestia: -5
Dev-C++: -5
Gaim: -5
PDFCreator: -5
XOOPS Dynamic Web CMS: -5
DOSBox DOS Emulator: -3
phpBB: -3
The CvsGui project: -3
JBoss.org: 3
VirtualDubMod: 3
VisualBoyAdvance: 3
eMule Morph: 5
eMule Plus: 5
FlasKMPEG: 5
Gallery: 5
NASA World Wind: 5
phpMyAdmin: 5
[email protected]: 5
ZSNES: 5
GnuWin32: 6
GTK+ and The GIMP installers for Windows: 6
ABC [Yet Another Bittorrent Client]: 7
Azureus: 7
CDex: 7
DC++: 7
eMule: 7
Ghostscript: 7
MinGW - Minimalist GNU for Windows: 7
Miranda: 7
PeerGuardian: 7
Shareaza: 7
TightVNC: 7
Wine Is Not an Emulator: 7
Eraser: 15
aMSN: 17
DScaler Deinterlacer/Scaler: 18

And my alternate biases yield:

BitTorrent: -19
burst!: -19
ffdshow: -19
FileZilla: -19
7-Zip: -13
Ares Galaxy: -13
Audacity: -13
Celestia: -13
Dev-C++: -13
Gaim: -13
guliverkli: -13
VirtualDub: -13
Webmin: -13
WinSCP: -13
XOOPS Dynamic Web CMS: -13
AC3Filter: -7
DOSBox DOS Emulator: -7
GnuWin32: -7
GTK+ and The GIMP installers for Windows: -7
JBoss.org: -7
NASA World Wind: -7
PDFCreator: -7
phpBB: -7
The CvsGui project: -7
ABC [Yet Another Bittorrent Client]: -1
DC++: -1
Ghostscript: -1
MinGW - Minimalist GNU for Windows: -1
Miranda: -1
PeerGuardian: -1
phpMyAdmin: -1
Wine Is Not an Emulator: -1
eMule Plus: 7
FlasKMPEG: 7
Gallery: 7
VirtualDubMod: 7
VisualBoyAdvance: 7
ZSNES: 7
Azureus: 13
CDex: 13
eMule: 13
eMule Morph: 13
Eraser: 13
Shareaza: 13
TightVNC: 13
[email protected]: 13
aMSN: 19
DScaler Deinterlacer/Scaler: 19