Skip to main content

GNU make insanity: finding the value of the -j parameter

The other day at work a colleague posed the seemingly innocent question "Can I find out the value of the GNU make -j parameter inside a Makefile?". Simple, right? Recall that -j (or --jobs) specifies the maximum number of jobs that GNU make can run in parallel. You've probably used it to speed up a build by typing something like make -j16.

But it turns out the actual value supplied is not available in any GNU make variable. You can verify that by doing something like make -p -j16 which will dump out pretty much everything defined by GNU make and the Makefile. You won't find any reference to the -j16 anywhere.

What follows falls into the "It's crazy, but it just might work!" category.

But it is possible to calculate its value if you are willing push GNU make hard. Here's a Makefile that gets the value given to -j (or --jobs) into a variable called JOB_COUNT

There's quite a lot of magic in that Makefile. I'll explain how it works bit by bit. First the general idea. The Makefile attempts to run up to 32 jobs in parallel (the actual value 32 can be adjusted on line 10 if you want to be able to detect -j values greater than 32). It will get GNU make to run as many jobs in parallel as possible and each job will pause for one second and then fail. The result is that exactly N jobs will run for -jN.

Where do the 32 jobs come from? They are targets named par-0, par-1, par-2, ..., par-31 and are created by the pattern rule on line 11. The pattern rule uses $(eval) to append a single x to a variable called PAR_COUNT. Each time GNU make runs one of the rules PAR_COUNT is appended.

Within the rule there's echo $(words $(PAR_COUNT)) which will echo the number of xs in PAR_COUNT when the rule runs. The result is that 1, 2, 3, ..., N is echoed for -jN.

The actual target names par-0par-1par-2, ..., par-31 are created on line 10 as prerequisites to a target called par by adding the prefix par- to the sequence of numbers 0 1 2 3 ... 31. Where that sequence comes from is interesting.

The sequence is created by the function to_n defined on line 7. To understand how to_n works consider what happens when $(call to_n,4) is executed (it should return 0 1 2 3).


Inside GNU make there's essentially a functional programming language that operates on space-separated lists. By building up a list of xs it's possible to output the numbers in sequence by using $(words) to count the number of elements in the list and by appending to the list on each recursive call to to_n. to_n finishes when the list has the required length. The equality test is done using $(filter-out) as a kind of 'string not equal function'.

So, that's par explained. To actually retrieve the number of jobs a sub-make is used on line 4. The sub-make will inherit the -jN (without the actual value being passed down) because of the way GNU make's jobserver functionality works (you can read about it here).

The output of the sub-make is written to a file called .parallel (which is always created because of the FORCE target) and then it is read at line 9 and the maximum value retrieved. Because the sub-make will actually fail (failure is used to detect the maximum parallelism available) there's an || true to hide the error from GNU make and stderr is redirected.

One last thing. To get the value into JOB_COUNT it's necessary to actually run the parallel target. I've done that using an order-only prerequisite in line 1 (notice the | before parallel?). That causes the parallel target to execute but its execution doesn't affect whether all runs or not.

The only remaining question remaining is... should you do this type of thing?

PS If you are into this sort of thing you might enjoy my new book from No Starch: The GNU Make Book.

Comments

pjb said…
This makefile is at least ten time bigger than a patch to GNU make to export a variable with the -j number!

DUH! This is open source! Send patches!
Michael said…
That's a little unreasonable. Even if you were to get a patch included in make (which is unlikely/would take a long time), most people still probably won't be able to update to a newer version with this feature.

Besides that, it's a cool hack - don't knock it because it isn't the "right way" to do this.
Emanuele said…
Hey!

the right way to solve your problem is quite simple: you take the source code of GNU Make, read it, and add a #J variable.

I did it for you (feel free to offer me a beer):

https://github.com/esantoro/make

And here it is in action:

http://santoro.tk/~manu/gnumake.png

It's doing exactly the thing you want.
Unknown said…
This comment has been removed by the author.
Unknown said…
What in the world, how many hours did this take?
Unknown said…
I enjoyed the post, though think the underlying mechanism is obscured by using to_n rather than just listing them explicitly.

Two problems that I can see. If -k is in use then the false doesn't stop make creating more children up to the 32 explicitly targeted. And it assumes that all the first round of children will be running before the first child finishes with failure after a second's sleep. On a busy machine, this might not happen.
There is a simpler solution do find out -j value: https://stackoverflow.com/a/48865939/412080
malcook said…
new comment on an old thread: upgrade to version 4.2.1 and parse -j out of MAKEFLAGS

Popular posts from this blog

Your last name contains invalid characters

My last name is "Graham-Cumming". But here's a typical form response when I enter it:


Does the web site have any idea how rude it is to claim that my last name contains invalid characters? Clearly not. What they actually meant is: our web site will not accept that hyphen in your last name. But do they say that? No, of course not. They decide to shove in my face the claim that there's something wrong with my name.

There's nothing wrong with my name, just as there's nothing wrong with someone whose first name is Jean-Marie, or someone whose last name is O'Reilly.

What is wrong is that way this is being handled. If the system can't cope with non-letters and spaces it needs to say that. How about the following error message:

Our system is unable to process last names that contain non-letters, please replace them with spaces.

Don't blame me for having a last name that your system doesn't like, whose fault is that? Saying "Your last name …

Importing an existing SSL key/certificate pair into a Java keystore

I'm writing this blog post in case anyone else has to Google that. In Java 6 keytool has been improved so that it now becomes possible to import an existing key and certificate (say one you generated outside of the Java world) into a keystore.

You need: Java 6 and openssl.

1. Suppose you have a certificate and key in PEM format. The key is named host.key and the certificate host.crt.

2. The first step is to convert them into a single PKCS12 file using the command: openssl pkcs12 -export -in host.crt -inkey host.key > host.p12. You will be asked for various passwords (the password to access the key (if set) and then the password for the PKCS12 file being created).

3. Then import the PKCS12 file into a keystore using the command: keytool -importkeystore -srckeystore host.p12 -destkeystore host.jks -srcstoretype pkcs12. You now have a keystore named host.jks containing the certificate/key you need.

For the sake of completeness here's the output of a full session I performe…

More fun with toys: the Ikea LILLABO Train Set

As further proof of my unsuitability to be a child minder (see previous post) I found myself playing with an Ikea LILLABO 20-piece basic set train.


The train set has 16 pieces of track (12 curves, two straight pieces and a two part bridge) and 4 pieces of train. What I wondered was... how many possible looping train tracks can be made using all 16 pieces?

The answer is... 9. Here's a picture of the 9 different layouts.


The picture was generated using a little program written in Processing. The bridge is red, the straight pieces are green and the curves are blue or magenta depending on whether they are oriented clockwise or anticlockwise. The curved pieces can be oriented in either way.

To generate those layouts I wrote a small program which runs through all the possible layouts and determines which form a loop. The program eliminates duplicate layouts (such as those that are mirror images of each other).

It outputs a list of instructions for building loops. These instructions con…