Saturday, March 28, 2015

GNU make insanity: finding the value of the -j parameter

The other day at work a colleague posed the seemingly innocent question "Can I find out the value of the GNU make -j parameter inside a Makefile?". Simple, right? Recall that -j (or --jobs) specifies the maximum number of jobs that GNU make can run in parallel. You've probably used it to speed up a build by typing something like make -j16.

But it turns out the actual value supplied is not available in any GNU make variable. You can verify that by doing something like make -p -j16 which will dump out pretty much everything defined by GNU make and the Makefile. You won't find any reference to the -j16 anywhere.

What follows falls into the "It's crazy, but it just might work!" category.

But it is possible to calculate its value if you are willing push GNU make hard. Here's a Makefile that gets the value given to -j (or --jobs) into a variable called JOB_COUNT

There's quite a lot of magic in that Makefile. I'll explain how it works bit by bit. First the general idea. The Makefile attempts to run up to 32 jobs in parallel (the actual value 32 can be adjusted on line 10 if you want to be able to detect -j values greater than 32). It will get GNU make to run as many jobs in parallel as possible and each job will pause for one second and then fail. The result is that exactly N jobs will run for -jN.

Where do the 32 jobs come from? They are targets named par-0, par-1, par-2, ..., par-31 and are created by the pattern rule on line 11. The pattern rule uses $(eval) to append a single x to a variable called PAR_COUNT. Each time GNU make runs one of the rules PAR_COUNT is appended.

Within the rule there's echo $(words $(PAR_COUNT)) which will echo the number of xs in PAR_COUNT when the rule runs. The result is that 1, 2, 3, ..., N is echoed for -jN.

The actual target names par-0par-1par-2, ..., par-31 are created on line 10 as prerequisites to a target called par by adding the prefix par- to the sequence of numbers 0 1 2 3 ... 31. Where that sequence comes from is interesting.

The sequence is created by the function to_n defined on line 7. To understand how to_n works consider what happens when $(call to_n,4) is executed (it should return 0 1 2 3).


Inside GNU make there's essentially a functional programming language that operates on space-separated lists. By building up a list of xs it's possible to output the numbers in sequence by using $(words) to count the number of elements in the list and by appending to the list on each recursive call to to_n. to_n finishes when the list has the required length. The equality test is done using $(filter-out) as a kind of 'string not equal function'.

So, that's par explained. To actually retrieve the number of jobs a sub-make is used on line 4. The sub-make will inherit the -jN (without the actual value being passed down) because of the way GNU make's jobserver functionality works (you can read about it here).

The output of the sub-make is written to a file called .parallel (which is always created because of the FORCE target) and then it is read at line 9 and the maximum value retrieved. Because the sub-make will actually fail (failure is used to detect the maximum parallelism available) there's an || true to hide the error from GNU make and stderr is redirected.

One last thing. To get the value into JOB_COUNT it's necessary to actually run the parallel target. I've done that using an order-only prerequisite in line 1 (notice the | before parallel?). That causes the parallel target to execute but its execution doesn't affect whether all runs or not.

The only remaining question remaining is... should you do this type of thing?

PS If you are into this sort of thing you might enjoy my new book from No Starch: The GNU Make Book.

6 comments:

pjb said...

This makefile is at least ten time bigger than a patch to GNU make to export a variable with the -j number!

DUH! This is open source! Send patches!

Michael said...

That's a little unreasonable. Even if you were to get a patch included in make (which is unlikely/would take a long time), most people still probably won't be able to update to a newer version with this feature.

Besides that, it's a cool hack - don't knock it because it isn't the "right way" to do this.

Emanuele said...

Hey!

the right way to solve your problem is quite simple: you take the source code of GNU Make, read it, and add a #J variable.

I did it for you (feel free to offer me a beer):

https://github.com/esantoro/make

And here it is in action:

http://santoro.tk/~manu/gnumake.png

It's doing exactly the thing you want.

Josh Hittie said...
This comment has been removed by the author.
Josh Hittie said...

What in the world, how many hours did this take?

Ralph Corderoy said...

I enjoyed the post, though think the underlying mechanism is obscured by using to_n rather than just listing them explicitly.

Two problems that I can see. If -k is in use then the false doesn't stop make creating more children up to the 32 explicitly targeted. And it assumes that all the first round of children will be running before the first child finishes with failure after a second's sleep. On a busy machine, this might not happen.