Macaroni for C++ uses Macaroni for C++ to generate code and then Boost Build to build itself.

I wrote Macaroni for C++ out of a desire to bring the speed at which I could code .NET and Java to C++. Unfortunately, one of the things I ran into is the classical problem that large C++ projects can quickly take forever to build if you aren't careful. From scratch, Macaroni takes between eight and ten minutes to build on my old core 2 duo laptop. On my new, top of the line i7 desktop it builds in over five minutes. On my netbook it takes an hour.

Boost Build is an incredible tool which is hurt not so much by its documentation but by the fact that its complexity is necessary to provide features that are hard to appreciate. Boost Build handles the discrepancies of building code with different compilers, operating systems, and configurations, making it possible to use a single build script that can compile an application linked to shared libraries with Visual C++ in debug mode, then a statically compiled version using MinGW, and finally build a Ubuntu version of the app in a VM.

The ability to choose on the fly the way to compile an application, rather than having to think ahead when writing the build script, is liberating. It's like using a nice multi-platform library such as Boost Filesystem instead of writing the code to access files and directories yourself from scratch for every operating system. Boost Build, along with the Boost libraries, enabled me to quickly create a Linux version of Macaroni, which makes me wonder if any person who advocates something like autotools or make on the basis that they are “universal” has bothered trying a different OS or development environment in the past ten years.

However, Boost Build has a price to pay in the form of it's learning curve. I've spent the last three years figuring it out, and at times have been frustrated to the point where I nearly dropped it. When I first discovered Boost Build, I decided to make it the required build system of Macaroni. After several months of bitter experience I realized instead I needed to find a way to represent the physical layout of the files generated by Macaroni, so that while I would mainly focus on generating files for Boost Build I could create build scripts for other tools if I or anyone else ever wanted to (this was also to generate Visual Studio projects to take advantage of VS's debugger, which is the only reason I start up Visual Studio anymore).

I feel as if I still can't claim to know Boost Build, as I'll find out new things constantly that seem so useful I wonder how I could ever get work done without them. An example of this is the “-j” flag, which I found today.

Normally when Boost Build executes, you can see it calling the compiler once for every object file (in Macaroni there is going to be a one-to-one correlation between classes and object files… I've thought of ways to customize this but so far every idea seems to add too much complexity). On a multicore beast of a machine like my new one that is clearly too boring for the CPU, which sits at 20%.

Enter the “-j” flag, which tells Boost to call multiple processes simultaneously. So “bjam -j4” will invoke four processes simultaneously. This has sped up the build times of Macaroni dramatically.

I wanted to time the differences, so I installed ptime, a useful program that spits out the time taken to invoke some process. I then tested bjam's performance without the -j flag, then with -j4 and finally -j8. Each time I first deleted the bin/msvc-11.0 directory so it would build from scratch, then I invoked it again to see if there was any speed up.

\________________________________________/
|         |   normal |     -j4 |     -j8 |
+---------+----------+---------+---------+
| build   | 310.510s | 94.594s | 69.813s |
| rebuild |   1.067s |  1.060s |  1.059s |
""""""""""""""""""""""""""""""""""""""""""

It turns out there is no speed increase for the parsing process Bjam does initially, so if nothing needs to be built the -j flags add nothing. However, moving to “-j8” made the build four and a half times faster. Its also the only process I've yet seen cause my CPU to jump to 100%.

This flag feels like some kind of unbelievable secret, like when I figured out there was a hidden ending in Link's Awakening, or discovered the #line directive which allowed compile errors to reference Macaroni source code. If it's in the Boost Build docs then it must be buried (though contributions are welcome, so maybe I should shut up and add it). When I, right now, Google specifically for “bjam -j” or “bjam -j4” I see nothing mentioning this flag on the first page of Google results.

The way I discovered the flag initially is by stumbling into an ancient page where someone had pointed out that the “-j” flag interspersed output from the commands it called in parallel. When I read it, I did a double take because I couldn't believe such a useful piece of functionality was real and being referenced so nonchalantly. After bjam seemed to work with the flag, I ran the experiments listed above to prove it to myself that “-j” did would I hoped and that it wasn't some illusion. I've since Googled “bjam -j4” and the only webpage I could find referencing this incredibly useful bit of functionality is this one).

Maybe the flag is officially unsupported? All I know is that I'm grateful it works now and has dramatically decreased the time it takes me to build from scratch.