Skip to main content

GNU make insanity: finding the value of the -j parameter

The other day at work a colleague posed the seemingly innocent question "Can I find out the value of the GNU make -j parameter inside a Makefile?". Simple, right? Recall that -j (or --jobs) specifies the maximum number of jobs that GNU make can run in parallel. You've probably used it to speed up a build by typing something like make -j16.

But it turns out the actual value supplied is not available in any GNU make variable. You can verify that by doing something like make -p -j16 which will dump out pretty much everything defined by GNU make and the Makefile. You won't find any reference to the -j16 anywhere.

What follows falls into the "It's crazy, but it just might work!" category.

But it is possible to calculate its value if you are willing push GNU make hard. Here's a Makefile that gets the value given to -j (or --jobs) into a variable called JOB_COUNT

There's quite a lot of magic in that Makefile. I'll explain how it works bit by bit. First the general idea. The Makefile attempts to run up to 32 jobs in parallel (the actual value 32 can be adjusted on line 10 if you want to be able to detect -j values greater than 32). It will get GNU make to run as many jobs in parallel as possible and each job will pause for one second and then fail. The result is that exactly N jobs will run for -jN.

Where do the 32 jobs come from? They are targets named par-0, par-1, par-2, ..., par-31 and are created by the pattern rule on line 11. The pattern rule uses $(eval) to append a single x to a variable called PAR_COUNT. Each time GNU make runs one of the rules PAR_COUNT is appended.

Within the rule there's echo $(words $(PAR_COUNT)) which will echo the number of xs in PAR_COUNT when the rule runs. The result is that 1, 2, 3, ..., N is echoed for -jN.

The actual target names par-0par-1par-2, ..., par-31 are created on line 10 as prerequisites to a target called par by adding the prefix par- to the sequence of numbers 0 1 2 3 ... 31. Where that sequence comes from is interesting.

The sequence is created by the function to_n defined on line 7. To understand how to_n works consider what happens when $(call to_n,4) is executed (it should return 0 1 2 3).

Inside GNU make there's essentially a functional programming language that operates on space-separated lists. By building up a list of xs it's possible to output the numbers in sequence by using $(words) to count the number of elements in the list and by appending to the list on each recursive call to to_n. to_n finishes when the list has the required length. The equality test is done using $(filter-out) as a kind of 'string not equal function'.

So, that's par explained. To actually retrieve the number of jobs a sub-make is used on line 4. The sub-make will inherit the -jN (without the actual value being passed down) because of the way GNU make's jobserver functionality works (you can read about it here).

The output of the sub-make is written to a file called .parallel (which is always created because of the FORCE target) and then it is read at line 9 and the maximum value retrieved. Because the sub-make will actually fail (failure is used to detect the maximum parallelism available) there's an || true to hide the error from GNU make and stderr is redirected.

One last thing. To get the value into JOB_COUNT it's necessary to actually run the parallel target. I've done that using an order-only prerequisite in line 1 (notice the | before parallel?). That causes the parallel target to execute but its execution doesn't affect whether all runs or not.

The only remaining question remaining is... should you do this type of thing?

PS If you are into this sort of thing you might enjoy my new book from No Starch: The GNU Make Book.


pjb said…
This makefile is at least ten time bigger than a patch to GNU make to export a variable with the -j number!

DUH! This is open source! Send patches!
Michael said…
That's a little unreasonable. Even if you were to get a patch included in make (which is unlikely/would take a long time), most people still probably won't be able to update to a newer version with this feature.

Besides that, it's a cool hack - don't knock it because it isn't the "right way" to do this.
Emanuele said…

the right way to solve your problem is quite simple: you take the source code of GNU Make, read it, and add a #J variable.

I did it for you (feel free to offer me a beer):

And here it is in action:

It's doing exactly the thing you want.
Unknown said…
This comment has been removed by the author.
Unknown said…
What in the world, how many hours did this take?
Unknown said…
I enjoyed the post, though think the underlying mechanism is obscured by using to_n rather than just listing them explicitly.

Two problems that I can see. If -k is in use then the false doesn't stop make creating more children up to the 32 explicitly targeted. And it assumes that all the first round of children will be running before the first child finishes with failure after a second's sleep. On a busy machine, this might not happen.
There is a simpler solution do find out -j value:
malcook said…
new comment on an old thread: upgrade to version 4.2.1 and parse -j out of MAKEFLAGS

Popular posts from this blog

How to write a successful blog post

First, a quick clarification of 'successful'. In this instance, I mean a blog post that receives a large number of page views. For my, little blog the most successful post ever got almost 57,000 page views. Not a lot by some other standards, but I was pretty happy about it. Looking at the top 10 blog posts (by page views) on my site, I've tried to distill some wisdom about what made them successful. Your blog posting mileage may vary. 1. Avoid using the passive voice The Microsoft Word grammar checker has probably been telling you this for years, but the passive voice excludes the people involved in your blog post. And that includes you, the author, and the reader. By using personal pronouns like I, you and we, you will include the reader in your blog post. When I first started this blog I avoid using "I" because I thought I was being narcissistic. But we all like to read about other people, people help anchor a story in reality. Without people your bl

Your last name contains invalid characters

My last name is "Graham-Cumming". But here's a typical form response when I enter it: Does the web site have any idea how rude it is to claim that my last name contains invalid characters? Clearly not. What they actually meant is: our web site will not accept that hyphen in your last name. But do they say that? No, of course not. They decide to shove in my face the claim that there's something wrong with my name. There's nothing wrong with my name, just as there's nothing wrong with someone whose first name is Jean-Marie, or someone whose last name is O'Reilly. What is wrong is that way this is being handled. If the system can't cope with non-letters and spaces it needs to say that. How about the following error message: Our system is unable to process last names that contain non-letters, please replace them with spaces. Don't blame me for having a last name that your system doesn't like, whose fault is that? Saying "Your

The Elevator Button Problem

User interface design is hard. It's hard because people perceive apparently simple things very differently. For example, take a look at this interface to an elevator: From flickr Now imagine the following situation. You are on the third floor of this building and you wish to go to the tenth. The elevator is on the fifth floor and there's an indicator that tells you where it is. Which button do you press? Most people probably say: "press up" since they want to go up. Not long ago I watched someone do the opposite and questioned them about their behavior. They said: "well the elevator is on the fifth floor and I am on the third, so I want it to come down to me". Much can be learnt about the design of user interfaces by considering this, apparently, simple interface. If you think about the elevator button problem you'll find that something so simple has hidden depths. How do people learn about elevator calling? What's the right amount of