Another way to find the value of GNU make's -j parameter

The other day I wrote a blog post about finding the value of -j in GNU make and it received a lot of commentary. Daniel Frey wrote in with a different solution that goes like this. It's worth studying if you are interested in GNU make wrangling.

.PHONY: default FORCE token recursive
default: foo.out
    @$(eval JOB_COUNT := $(shell cat $<))
    @echo Job count: $(JOB_COUNT)

foo.out: FORCE
    @timeout 1 $(MAKE) --no-print-directory token recursive 2>/dev/null | wc -l >[email protected]

FORCE:

token:
    @echo X && sleep 2 && false

recursive:
    @$(MAKE) --no-print-directory token recursive

One weird trick that will give you makefile X-ray vision

Sometimes when you are running a make you'd like to see what commands get executed (even if they've been muted by prefixing them with @) and where in the makefile those commands reside. But you can't. There's no way to override @ and make -n doesn't actually run the make.

But here's 'one weird trick' that makes it possible to see inside a make as its running. Just add this to the makefile:

_SHELL := $(SHELL)
SHELL = $(warning [[email protected]])$(_SHELL) -x

Before digging into how that gives you makefile X-ray vision, let's take a look at it in action. Here's an example makefile that builds an executable from foo.o and bar.o (which are built from corresponding foo.c and bar.c files).

.PHONY: all
all: runme

runme: foo.o bar.o ; @$(LINK.o) $^ -o [email protected]

foo.o: foo.c
bar.o: bar.c

%.o: %.c ; @$(COMPILE.C) -o [email protected] $<

Running simply make on this performs the compiles and link but produces no output because output has been suppressed by the @ signs on the compile and link commands. Running a make -n gives this output:

% make -n
g++    -c -o foo.o foo.c
g++    -c -o bar.o bar.c
cc   foo.o bar.o -o runme

That's handy, but doesn't tell the whole story. For example, which of those lines correspond to which targets? And where are the actual commands found in the makefile?

But add the two lines above to the makefile and do a normal make and everything becomes clear:

% make
Makefile:9: [foo.o]
+ g++ -c -o foo.o foo.c
Makefile:9: [bar.o]
+ g++ -c -o bar.o bar.c
Makefile:4: [runme]
+ cc foo.o bar.o -o runme

It's easy to see that foo.o is built by the rule on line 9 of a makefile called Makefile (the %.o: %.c pattern rule) and that the link is on line 4. As each rule runs the location of the rule (with the corresponding target) is output followed by the actual commands.

How that works

The first line of the trick defines a variable called _SHELL and captures the value of the built-in SHELL variable using :=. SHELL contains the path (and parameters) for the shell that will be used to execute commands.

Then SHELL itself is redefined (this time using = and not :=) to contains a $(warning) and the original shell command (from _SHELL) with -x added. The -x causes the shell to print out commands being executed.

Since SHELL gets expanded for every recipe in the makefile, as it runs the $(warning) gets expanded and outputs the specific makefile line where the recipe can be found and [email protected] will be valid and contain the name of the target being built.

Caveat: doing this will slow down a build as GNU make doesn't use the shell if it can avoid it. If SHELL is untouched in a makefile the GNU make short-circuits the shell and execs commands directly if it can.

More?

If that sort of thing interests you, you might enjoy my book: The GNU Make Book.

Plain web text offenders: sending my location over HTTP when HTTPS was possible

The BBC Weather App sends the location of the user via HTTP to the BBC in order to determine the best location for weather information. It does this with roughly 100m accuracy, in the parameters of an unencrypted GET and even though the same API endpoint is available using HTTPS.

I discovered this accidentally using Charles Proxy to snoop on traffic from my iPhone at home. Here's the Charles Proxy view of that app interacting with the BBC's APIs:

It's hitting the endpoint http://open.live.bbc.co.uk/locator/locations with parameters la and lo containing the three decimal digit latitude and longitude (which give roughly 100m precision) over HTTP as a GET request.

The API then returns a JSON object containing nearby locations that the app can get weather information about.

Sadly, this API could have been accessed over HTTPS. Just switching the protocol from HTTP to HTTPS works fine. A legitimate wildcard certificate for *.bbc.co.uk is used.

So, the app could have used HTTPS.

Perhaps "This app would like to use HTTP for its API" should be a permission that the user has to explicitly give.

The one line you should add to every makefile

If you're using GNU make and you need help debugging a makefile then there's a single line your should add. And it's so useful that you should add it to every makefile you create.

It's:

    print-%: ; @echo $*=$($*)

It allows you to quickly get the value of any makefile variable. For example, suppose you want to know the value of a variable called SOURCE_FILES. You'd just type:

    make print-SOURCE_FILES

If you are using GNU make 3.82 or above it's not even necessary to modify the makefile itself. Just do

    make --eval="print-%: ; @echo $*=$($*)" print-SOURCE_FILES

to get the value of SOURCE_FILES. It 'adds' the line above to the makefile by evaluating it. The --eval parameter is a handy way of adding to an existing makefile without modifying it.

How that works

The line

    print-%: ; @echo $*=$($*)

defines a pattern-rule that matches any target in the form print-% (the % is the wildcard character). So when you run make print-SOURCE_FILES that rule will execute and the % will match SOURCE_FILES.

The command executed by print-% is @echo $*=$($*). Here I've used a semicolon to separate the target name and the recipe (commands to be executed). That makes this into a one-liner. In more traditional make syntax (where a tab is used to introduce the recipe) that would be written.

    print-%:
        @echo $*=$($*)

Using semicolon makes this easy to copy and paste.

The automatic variable $* matches the % in print-% (so when executing print-SOURCE_FILES, $* will be SOURCE_FILES). So $* contains the name of the variable that we want to print out.

The $($*) uses gets the value of a variable whose name is stored in $*. For example, $* might expand to SOURCE_FILES and then GNU make would get the value of $(SOURCE_FILES). Getting a variable by storing its name in another variable turns out to be a useful technique in many makefiles.

More?

If that sort of thing interests you, you might enjoy my book: The GNU Make Book.

"The GNU Make Book": probably more than you ever wanted to know about make

So, it's finally here. My second book, entitled The GNU Make Book and published by No Starch Press later this month (electronically it's available now). If you are a long time reader of my blog then you'll know that I've had a very long interest in GNU make and written quite a bit about it. For a while I had self-published all my articles on GNU make as a print on demand book.

The folks at No Starch Press kindly took my self-published effort and did the serious work of actually editing, organizing, indexing and generally sprucing it up. I added some articles that weren't in the old version and updated for GNU make 4.0.

And now it's here.

The book is not intended for people learning GNU make from scratch. If you need to do that go get the FSF's GNU make manual and read it. It covers all that you need to get started with GNU make.

Once you've done that you'll be ready to read mine. It's based on years of working with real makefiles, on having written a complete emulation of GNU make and having created the GNU Make Standard Library project.

It's thoroughly practical and I hope the ideas and techniques in it will prove useful to real-world makefile writers and maintainers. If you're interested there's sample chapter and Table of Contents.