Watching Vinnie Falco's excellent CppCon 2017 talk “Make Classes Great Again! (Using Concepts for Customization Points” inspired me to try using Boost Beast, a library for working for with HTTP and WebSocket's in C++. Unfortunately since this is C++ I spent most of my time building all of the libraries I needed in Windows, which for some reason I still use. This post is a summary of my findings for my own future reference.

Building Boost

First off, I hope you're sitting down because I must reveal to you that Boost is still a total pain in the ass to build on Windows with the latest Visual Studio 2017 release.

In the past, I considered Boost Build my friend, but now we are enemies due to reasons I keep wanting to write a blog post about; in short I realized it was a time vampire that was draining me of my precious mortality. I liked a lot about Boost Build compared to the alternatives, but unfortunately it suffers from a lack of quality which caused me to spend more time maintaining it than working on my projects. In comparison CMake is rock solid once you get it working.

Everyone calls Microsoft's C++ compiler “Visual Studio” but the compiler itself is named “Microsoft Visual C++”. Boost Build bucks these social norms by referring to it as “msvc” which I always admired. With Visual Studio 2015, the corresponding compiler version was 14.0, but with Visual Studio 2017 it's changed to 14.1 which is surprising (of course since Boost Build is built on the worse than TCL stringly-typed programming language bjam you can pass in “msvc-15.0” as the tool chain and have no idea you've done something wrong until the build totally finishes, you get linker errors, and you finally realize the libraries were built with a different compiler).

Another thing that surprised me is that Boost Build doesn't do a good job of automatically finding the Microsoft tools it needs to call, meaning you have to run the “vcvars” batch file included with Visual Studio or open up the shortcut labeled “Visual Studio 2017's Developer Command Prompt”. In the past this wasn't necessary, so I never did it. I also liked keeping my shell clean from all the strange paths MS added to it. However since both Boost and OpenSSL required me to do this I've since added CALL “C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvars64.bat” to my bash.rc-like batch file that executes for my go-to Command Prompt session. Time will tell if this creates other problems. Side note: I don't bother using mingw or any other compiler on Windows anymore, since Linux let's me do that without the stress ulcer and is run on my laptop natively and on my desktop using the Windows Subsystem for Linx (aka Bash for Ubuntu for Windows). Part of the fall out of my divorce with Boost Build also means using multiple compilers is even harder than it was before in Windows and there's no point when easier operating systems exist and most of my code is portable.

Another big change to my go-to shell was adding “C:\Program Files\7-Zip”, which gives me access to the excellent 7-zip tool from the command line (AppVeyor makes this available on it's command line which is a sign it's a good idea).

Finally, it's worth noting I have long put all of the typical unix tools (grep, sed, ls, etc) on my command prompt by installing Cygwin (which I used to need for Dreamcast development) and adding it's bin directory to my path. In the script below I use curl and also sha256sum pretty often.

Next, I realized I had an ancient site-config.jam file with all my toolchains set up (one of Boost Build's best features). Unfortunately I also had a statement in there that said “using msvc : 14.0 ;”. In the end I commented all of this out.

Manuel Gustavo has written a series of scripts for building Boost versions 1.63 and 1.64 in Windows. I didn't use them other than to crib the commands he used to build the 64-bit variants of the libraries.

First, make sure you have this stuff in your command prompt:

set PATH=%PATH%;C:\Program Files\7-Zip
set PATH=%PATH%;C:\Cygwin\bin\curl.exe
CALL "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvars64.bat"

Then run this:

cd C:\Tools\boost
curl -o boost_1_65_1.7z -L https://dl.bintray.com/boostorg/release/1.65.1/source/boost_1_65_1.7z
REM should be "df7bea4ce58076bd8e9cd63966562d8cf969b431e1c587969e6aa408e51de606"
sha256sum boost_1_65_1.7z
7z x boost_1_65_1.7z
cd boost_1_65_1
bootstrap.bat
b2.exe --toolset=msvc-14.1 --clean-all
b2.exe --toolset=msvc-14.1 architecture=x86 address-model=64 --stagedir=".\stage_x64" threading=multi --build-type=complete stage -j8
b2.exe --toolset=msvc-14.1 --clean-all
b2.exe --toolset=msvc-14.1 architecture=x86 address-model=32 --stagedir=".\stage_x86" threading=multi --build-type=complete stage -j8

Notes: change ‘-j8’ above to ‘-jn’, where ‘n’ is the number of cores on your machine.

Manuel Gustavo has more code that places the libraries into a “bin” directory, but I've skipped that.

Once this runs, you need to set the environment variable BOOST_ROOT to the location of the unzipped boost_1_65_1 stuff from above.

Building OpenSSL

If you want to make websites or service, you're going to want to use SSL. It's a huge headache but given how much the world relies on it you can't really avoid being initiated.

OpenSSL turns out to not be too difficult to build if you follow the exact necessary steps, much like every other problem faced by man. My main issues were caused by the myriad of out-dated documentation that's out there.

First off, building OpenSSL in Windows requires Perl. I used Strawberry Perl, because I'm a fan of any any programming tool or language named after food. It may feel annoying to have to install a language just to build a thing but cheer up by imagining all the totally siiiiick Perl 5 code you'll be able to write after this.

Also, there's some difference between OpenSSL distributions. 1.1.0f is the newest version, but turns out the 1.1.0 version doesn't work with Boost; see bugs like this for more info. So if you value your time on Earth stick with 1.0.2.

Also, building with the ASM stuff is probably better but adds the need to download NASM. So I just skipped it with the “no-asm” option.

mkdir C:\Tools\openssl-64
mkdir C:\Tools\openssl-src
cd C:\Tools\openssl-src
curl -o openssl-1.0.2l.tar.gz -L https://www.openssl.org/source/openssl-1.0.2l.tar.gz
7z x openssl-1.0.2l.tar.gz
7z x openssl-1.0.2l.tar
cd openssl-1.0.2l
perl Configure VC-WIN64A no-asm --prefix=C:\Tools\openssl-64 --openssldir=C:\Tools\openssl-64
ms\do_win64a
nmake -f ms\nt.mak
nmake -f ms\nt.mak install

Building some Code

To start using Boost and OpenSSL from CMake you'll need to set some environment variables so CMake knows where you put that stuff.

set BOOST_LIBRARYDIR=C:\Tools\boost\boost_1_65_1\stage_x64\lib
set OPENSSL_ROOT_DIR=C:\Tools\openssl-64
SET PATH=%PATH%;%BOOST_LIBRARYDIR%

At this point you should be able to create simple CMake projects that can find and use OpenSSL and Boost (including stuff like coroutines).

As painful as it is, the result is worth it. Here is what some asynchronous client code looks like using the Boost Beast library (it's header only so no nasty build step required).

boost::asio::spawn(ios, [&](boost::asio::yield_context yield)
    {
        tcp::resolver resolver{ios};
        ssl::stream<tcp::socket> stream{ios, ctx};

        {
            auto const begin = resolver.async_resolve(
                tcp::resolver::query{ host, port }, yield);
            tcp::resolver::iterator end;
            boost::asio::async_connect(
                stream.next_layer(), begin, end, yield);
        }

        stream.async_handshake(ssl::stream_base::client, yield);

        http::request<http::string_body> req{http::verb::get, target, version};
        req.set(http::field::host, host);
        req.set(http::field::user_agent, BOOST_BEAST_VERSION_STRING);

        http::async_write(stream, req, yield);

        boost::beast::flat_buffer b;

        http::response<http::dynamic_body> res;

        http::async_read(stream, b, res, yield);

        std::cout << res << std::endl;

        boost::system::error_code ec;
        // This is currently failing every single time due to this bug:
        //  https://svn.boost.org/trac10/ticket/12710
        stream.async_shutdown(yield[ec]);
        if(ec == boost::asio::error::eof) {
            ec.assign(0, ec.category());
        } else {
            boost::asio::detail::throw_error(ec);
        }
    });

(Note: I stole this from Vinnie Falco's example code, but I put the coroutine function into a lambda and rely on exceptions rather than error checking to make the code look nicer :)).

I think it's further proof that coroutines are the sexiest language feature ever- everyplace you see “yield” getting passed in would normally require a call to a separate function in classic Boost ASIO style, but here the coroutine yields, bringing the same advantages but without the fuss.

Originally, Mario was a normal sized man fighting monstrous turtles who stood as tall as him when on all fours. He evened the odds by becoming “Super Mario”, a giant who could go toe to toe with these adverseries and break ceilings using only a single raised fist (and possibly his cranium depending on how he was drawn).

However when Super Mario Bros 3 came out in Japan, something changed in the canonical artwork. There was no longer any distinction between Mario and Super Mario in the instruction manual's illustrations other than the scale of the images. When “Super Mario” put on a suit, he was drawn like a normal sized, pudgy man. All of the cartoons, comics and other merchandise reflected this by showing only one size of Mario. The “tiny” version went unseen except in the games themselves.

Shortly after that Super Mario Bros 2 was released in America. As everyone knows, this game was different from the Japanese SMB2 and was actually a modified version of the previously unrelated game Doki Doki Panic with the main characters changed to be Mario and his associates. As a hold over from the source game, Mario had a life bar that always started at two. As a nod to the past titles Mario shrank to a tinier version of himself with a huge head when his life bar dropped to one. However this small version was somewaht silly looking and featured an enormous head on a tiny body. This was probably done so that the game could still believably force players to duck to avoid certain obstacles.

The order had been totally reversed: now Mario was a normal sized man who, when struck or hurt, became a little person, sometimes with a huge head. Or as my nephew refers to it, a baby. This is now firmly cemented as the way things work in the Mario universe; in the opening cinematic of New Super Mario Bros Wii Mario arrives at a party at his full size but regresses into a dwarf after being struck by a flying object.

The best proof that the concept of a “Super” Mario had dissappeared was a picture from the instruction manual of 2011's Super Mario 3D Land which officially refers to the two versions as “Normal Mario” and “Small Mario.” Adding insult to injury, “Small Mario” also loses his hat in this game, as if to suggest short people have no reason to live wear hats. I think they should've also made him bald. The picture below compares the original Super Mario Bros's hierarchy of Mario states to the one created for Super Mario 3D Land.

What's stranger is the Princess, aside from her apperance in SMB2, is portrayed as always being a consistent size. This comic by KC Green from USgamer focuses on this disturbing phenomenon.

This all begs the question: is Mario's original size the bigger one, which he only reattains after eatting a mushroom, and does his smaller version represent some terrible affliction or curse? Or is the smaller Mario his true self, which he hides from even those closest to him by eatting magical fungi before attending his numerous cake themed galas?

Eating a mushroom to become a giant is empowering; but being turned into a vunerable semi-deformed version of oneself when being critically injured is weird and kind of depressing.

I'll always appreciate the original, rougher version, where Mario- who is no more than a squat little fat guy- grows into a titan to battle the forces of evil.

The ending of the 1994 Donkey Kong for Gameboy, despite being released well after SMB3, celeberates the original theme behind the mechanic. After defeating Donkey Kong, Mario's girlfriend Paulina, finds a magic mushroom which she offers as a gift. In this game Mario is always “small”, but is redrawn so that he looks more like a normal person instead of the “big head” style from SMB2, so when he powers up in the ending he seems to the player to have actually grown to a somewhat shockingly large size (he also loses most of his ponch) so it makes sense that he then proceeds to pick up the villianous Donkey Kong and hold him triumphantly over his head. Donkey Kong's son then runs after Mario implying they will fight; this is a reference to the sequel to the original arcade Donkey Kong, Donkey Kong Jr, which featured Mario playing a villian who had imprisoned the great ape (I am unsure why God never answered my prayers that such a sequel be made).

According to this article from Slate, Gorillas are “approximately six to 15 times as strong” as humans and any Gorilla that was not incapacitated by illness or drugs would be capable of savagely tearing the arms off of even the world's toughest human being in a matter of seconds were they to engage in an all-out brawl. Given this scientific fact, it makes sense that when Mario touches Donkey Kong in the Gameboy game the ape picks him up, shakes him violently, and then throws him to his death (the implication being that Mario's neck has snapped in the process). So clearly this is scientific proof that the large Mario seen in Donkey Kong's ending is indeed the “Super Mario” from the first Mario Bros who possesses extraordinary strength surprassing that of a normal human.

I also want to comment on some uncomfortable Freudian elements that may be at play in the ending. It is heavily implied that Donkey Kong ‘94 takes place before the events of the first Super Mario Bros. Mario has just traversed the world to rescue his girlfriend from a giant ape, yet here she is apropos of nothing in this, their first moment of safety and calm following what would have been a tramautic crisis, suggesting some exotic enhancement drug to Mario as if to say he wasn’t good enough for her already. This scene may show an important moment in Mario's personal history, where feelings of rejection by his far more attractive girlfriend caused him to start hiding his normal size from others by pretending his “super” size was in fact his normal stature. These same feelings of inadequacy might have also sparked Mario's subsequent promiscous behavior, where he actively sought to cheat on Paulina by seeking out relationships with multiple princesses perhaps in an effort to prove his masculinity to others.

But I digress.

A few other games have the original version of the “average person becomes big person” trope, but the one that does it best is the 1989 NES Platformer “Amagon”. Amagon tells the touching story of a weak girly man named Amagon who travels the wild jungles and is frequently murdered by birds or spiders. Amagon's only real hope is to collect a power up which transforms him into Super Amagon, a muscular, hulking version of himself capable of obliterating any birds or spiders that cross his path with a single blow of his mighty fists. Though Amagon isn't a great game, I appreciate how it most better represents the original theme of the “Super Mario” mechanic by putting the focus on Super Amagon being powerful instead of regular Amagon being weak. Normal Amagon, though he does look skinny and dies after one hit, is not drawn in the “super deformed” style of later iterations of “small” Mario. He also uses a gun to fight. Super Amagon however relies on close range physical attacks and has a real life bar. Because Super Amagon is able to sustain damage and survive, the player feels a sense of relief everytime they make it far enough to transform. The game becomes all about managing to survive as normal Amagon while waiting to transform or managing the resources to stay as Super Amagon as long as possible.

It's worth noting that the idea of “normal man becomes empowered by eating thing” was a key element of Popeye, who gains his super human abilities after eating spinach. Mario's creator Shigeru Miyamoto originally intended for the first Mario game, Donkey Kong, to be a Pop-Eye game instead. When negotiations fell through Miyamoto created Mario to stand in for Popeye, Donkey Kong to replace Bruno, and the slim Paulina to replace Olive Oil. However there was no real power up in the original Donkey Kong, aside from a mallet item that could be used temporarily. It wasn't until Super Mario Bros was released five years later that the character of “Super” Mario became tied to a premise that strongly resembled Popeye's most memorable element.

A long time ago, I realized that despite being somewhat crummy in terms of typical shell features, the Windows command prompt could do everything I usually needed a Unix shell to do. The only issue was the lack of programs and dealing with the horrible way most user guides instruct you to deal with setting environment variables. I decided to use a batch file to set all the environment variables I needed and documented how to do this here. It was a huge improvement over how I'd done things until then, where I'd simply hoped the installers had set (or not set) whichever environment variables I needed and kept a collection of lesser batch files to explicitly change variables when appropriate.

But the experience was still not optimal, and I felt this whenver I switched machines. The problem was all of the tools. So, I decided to create a “tools” directory with actual programs, such as Sublime Text, which I'd want to use on any Windows machine I moved to. This turned out to be a disaster; maintaining all those binary files led to wasted space and confusion. Eventually I settled on storing only a few select things in there, such as the all-important batch file I mention above, as well as a few important things such as my Key Pass install. For everything else I used the native installers which ended up being easier but left me with my original problem.

As any Linux user knows, this state of affairs is less than ideal. It's also fairly specific to Windows. In Linux, there are normally these amazing programs called “package managers” which let you download and install things by entering a single line of text at the command line. OSX has two of them as well- Homebrew and Macports. Windows actually has had package managers too, but the problem is there have been about a half dozen of them which I've seen and none of them contain that many packages. The most important part of a package manager is that it needs to be ubiqitious and have access to everything.

Enter Chocolatey

Yesterday I stumbled across “Chocolatey”, yet another package manager for Windows. I'm ready to say that this is the one all Windows people should be using. It is the first package manager that is good enough to make not using it seem like a mistake.

Why, in my opinion, is Chocolatey better than everything else?

First, it piggy backs off of multiple pre-existing technologies, including traditional Windows exe installers and MSIs. This means many Chocolatey packages are just simple bits of code which silently install pre-existing installers. This means if authors don't want to waste time writing Chocolatey packages in addition to their traditional installers, they don't have to- they (or someone else) just writes a tiny Chocolatey package file that defers to the installer. Adding packages like this is so easy to do it took me just a few hours to write my first one last night.

Second, it piggy backs off NuGet. This is a package manager for .NET language development that has become increasingly popular over the years for distributing libraries with or without code. NuGet recently added support for C and C++ code as well and will probably be the defacto code library manager for Windows in the future (baring any stupid ass decisions by Microsoft to fragment their own ecosystem, which they are fond of doing). What's great about backing Chocolatey with a development focused packaging tool is it makes it more approachable for exisitng developers and also allows for user-facing applications to be created from source, typical to how many Linux packages are distributed.

Third, while many Chocolatey packages do seem to put extra crap on the %PATH% environment variable (mostly because they're based off Windows installers which do the same thing), Chocolatey itself adds a single directory to the path which packages are then invited to install additional batch file redirects to. This helps to keep the %PATH% clean while allowing users to run “cinst” (the chocolatey install program) and have new programs available on their path.

Fourth, Chocolatey has great aesthetics. The name “Chocolatey” comes from a silly joke about how “everyone loves Chocolatey NuGet” which I support as a lover of inane project names that will be passive-aggressively disrespected in a “professional” envrionment (this joke is helped tremendously by the fact that the NuGet logo, when colored brown, looks like a delicious confection). I also love how Chocolatey's output looks- it isn't afraid to use colors- and how pretty it's official website is.

Fifth, installing Chocolatey is very simple- just copy and paste a single line into a command prompt! It's only serious dependency is PowerShell which makes it incompatable with Windows XP, but at this point *no one* should be running XP.

Finally, Chocolatey has already bested its competition by having a ton of packages- 1636 at this moment- and the count is continously growing.

Chocolatey is the future of Windows scripting and development. There is literally [no reason you should not visit it's site and begin using it today.

I've always appreciated quick programs, especially ones that perform well even on ancient or cheap hardware. I feel that most software running today is far slower than it needs to be, which has led to a global waste of hardware resources as everyone compensates by upgrading their gear. So of course I'd like to aspire to write quick code myself.

Most advice I've read on writing efficient code states that measurements are key and should be taken before taking any action at all. Unfortunately, everything I've read fails to mention what tools should be used for measuring code performance or cite products costing thousands of dollars. They often include a snarky comment about how if you're a serious developers you'll recoup your investment anyway. While I've slowly accrued enough hours of experience writing C++ over the years that I wouldn't laugh at myself too hard if I said I was a serious C++ developer, because the majority of work I do is at home in my spare time I just can't justify dropping several grand on code analysis tools.

So I was very happy the other day to discover Very Sleepy, a totally free code profiler that ended up helping me trim nearly 30% off the running time of Macaroni for C++.

Very Sleepy is noninvasive, meaning you don't have to recompile an application to use it (though the app must be compiled with debug symbols). How it works is by suspending the app once each millisecond and recording the stack trace. It then determines how much time is being spent in each function. Here's how I used it to get major performance gains from Macaroni in less than one hour.

Over the past year or so, I've moved from working on Macaroni for C++ in my spare time to actually using it for a game engine. The experience has made me aware of different, more practical issues in Macaroni itself. One thing I've noticed is while run times with Macaroni aren't that bad, they aren't where I'd like them to be either.

A few months back I added a feature to Macaroni which shows the time it's spent running so far next to every line output to the console. That helped me to get some solid numbers on how long projects were taking to parse and build.

Today, I was running a project which Macaroni spent 5.732 seconds simply analyzing and generating code for before it even invoked bjam. This isn't necessarily trivial work, but since Macaroni is not itself a real C++ compiler it's the kind of task I'd ultimately like to see happen in one second or less.

So, I invoked Macaroni as usual but added the “startPrompt” argument, which makes it sit around and wait for a user to press enter. This allows the program to get attached to a debugger. I then opened up Very Sleepy, and found Macaroni on it's list of running processes. I double clicked it, and then raced back to enter something into Macaroni's dummy prompt, so the time spent waiting at the prompt wouldn't skew the results (note that it's also possible to launch a process from Very Sleepy, but I was being lazy - though hopefully this serves as a proof of sorts that even casual usage can yield tangible benefits).

Since Very Sleepy was busy profiling it, Macaroni took longer to run- about 40 seconds or so- but when it stopped I got the following nice window showing all of the function calls in Macaroni and which ones had taken the most time.

One thing that's interesting is that the “% Inclusive” column shows the time spent not just running code in a function, but running all the code that function itself called. Because of this, the highest “% Inclusive” value is for the main method followed by other methods, such as “Macaroni::Environment::ProjectEnvironment::RunCommand,” which are the jumping off points for almost everything that happens in a typical invocation of Macaroni. However, it's still easy to spot functions which you know aren't called that early in the call graph.

The winning stinker of the bunch was “boost::filesystem::canonical” which is used by Macaroni's own Path class (which has different semantics in order to represent C++ source files easily, and which I wrote before I even knew of boost::filesystem… this project has taken so many years) to return an absolute path. As time has gone by, I've peppered the more recently created project / build system in Macaroni with things that need the absolute path, and it turns out the boost::filesystem function underpinning that is relatively expensive.

The fix was to simply cache the absolute path the first time it was computed, which took no time at all. This simple change cut two seconds off the run time, bringing the time until bjam was invoked from 5.732 to 3.874.

In short, Very Sleepy is an incredibly useful tool which makes spotting performance issues extremely simple. I'm grateful to it's authors for writing something so useful and giving it away for free.