People make a compelling argument that Microsoft stops innovating the moment it's locked people in, and point to Internet Explore 6 as a great example. Progress on Internet Explorer ceased for years until Firefox gained traction and lulled Microsoft out of complacency, at which point they turned the lights back on at IE HQ and got to work again.

The built in Windows console, also known as Command Prompt, makes Internet Explorer look like as innovative as a bleeding edge Linux distro. Based on the harm it does, it might be the worst piece of software pushed by Microsoft.

Why is it so bad? There are a few reasons, but my main beef is its failure to behave like every other modern computer application regarding copy and past. Copy and paste in the Windows console requires dragging over blocks of text, which is antiquated and no longer makes any sense. There is no option for selecting text line by line.

However, there are replacements.

• Power Cmd - This looks real nice, and a few years ago I bought a license. It doesn't work well on newer versions of Windows though.
  
• Console 2 - This has tabs and copy and paste. It also lets you have use background images, and looks like the terminals seen in Linux and OSX.
  
• ConEmu - Like Console 2, but seems to be the new heir to the cmd prompt throne.
  

Time for the bad news: most of these replacements don't work very well or slow down the programs they run. I wanted to write up the reasons I don't think they should be bothered with.

Let's start with PowerCmd, which I used on my laptop years ago. This was the first Cmd Prompt replacement I tried which seemed to be generally endorsed. Like all of these tools, it felt a bit wonky, but I'd accepted the quirks and switched to it for awhile until I switched back to Cmd briefly and discovered it was running programs much quicker. In other words, whenever I kicked off long running processes with PowerCmd, they ran glacially slow compared to the real Windows console. Years later, when my Netbook gave Windows XP a second life, I tried it again and found the problem to be more noticable than before.

I would have liked to write up a real speed measurement of PowerCmd, but these days it's so outdated it simply shouldn't be installed or considered. It's 32-bit, requires admin privileges, and even then failed when executing Boost Build.

Next up, there's ConEmu, described by Scott Hanselman as “The Windows Terminal/Console/Prompt we've been waiting for?” I started using this a week ago and it looked like the fully loaded real hot dog Cmd Prompt suffers had been waiting for. It offers line-by-line text selection, the ability to integrate with Cygwin, and loads upon loads of special features. I was very impressed by it, and spent half a day configuring the options to suite me.

However, when I ran the Macaroni build process with ConEmu, it went from 69 seconds to 172. What's worse is that despite the extreme slow down, the CPU stayed at 100% the entire time! I was so shocked by this I ran the tests again, this time taking care to not touch or do anything at all on my computer. But the results were the same. ConEmu strikes me as the PowerCmd successor, in that it looks pretty but cannot be used for anything serious.

Finally, I installed Console 2.

First, the good news: Console 2 runs programs at the same speed as the real Windows console. So it's worth consideration.

Personally, though, I didn't suite me. First off, it failed to move around correctly with Aero snap, and would constantly get stuck partially off screen when I moved it across monitors. These days I work in Windows, OSX, and Linux, and use some kind of “snap to grid” key bindings for all of them. Not cooperating with those bindings is a kiss of death! Second, Console 2 didn't copy and paste easily or correctly! Maybe I didn't sacrifice enough time to get that feature to work right but at this point I don't understand why Console 2 is so highly recommended given how unpleasant I found it. I guess there's no accounting for taste.

In summary, my recommendation is that you learn to deal with the Windows console's broken copy and paste. I can sympathize with you if you have to switch, in which case try Console 2 (at least the other shoe will drop quickly). Don't use ConEmu unless you want to waste your time, and don't even think of installing Power Cmd.

Creating a library in Boost Build is easy; it's doubly so when using Macaroni.

Pulling in a library like that is similarly easy - just use the “use-project” rule and reference the jamroot.jam file of the dependency.

However, the best practice of referencing a Boost library itself has confused me for awhile. There is not jamroot.jam file for each project, and referencing the main jam file at the root of the Boost distribution folder doesn't lead to happy results either. So how do you do it?

It turns out to be really simple.

First, make sure to set an environment variable named BOOST_ROOT to the absolute path of the unzipped Boost distribution directory.

Then, create a file called site-config.jam in your home directory, containing this:

using boost 
  : 1.52
  ;

For awhile, I kept thinking I needed to actually put the path to Boost in this file. Nope! That actually causes all hell to break lose in Windows and for things to generally not work (in Linux nothing bad seems to happen, but since doing it the way I'm suggesting now works in both OSes it might be better to keep things consistent).

With these two steps out of the way, it's easy to include the Boost libraries:

import boost ;
boost.use-project ;

exe recreate
    :   ../src/Main.cpp
        /boost//headers      
        /boost//system
        /boost//filesystem
    ;

This project will build Main.cpp, which uses the Boost FileSystem and System library as dependencies. It will even build these libraries if they're missing! It also adds the Boost headers to the include path while building Main.cpp. Easy, no?

On a side note, I actually knew about this method years ago but ran into trouble using it when building complex projects with Macaroni, when a library would depend on another library which depended on Boost. I think in retrospect, this was due to some problem I had created. The good news is, I recently changed Macaroni to use this method and found that it jam files which build correctly on both Windows and Linux, even with complex projects using multiple dependencies.

Awhile ago I decided to invest some time and learn how to build the Dreamcast homebrew programming tools using the latest source (C++11 for the win!). These include a customized version of the GNU compiler collection along with the KallistiOS operating system.

I've finally gotten it working, and I feel like I somewhat understand the process, so I thought I'd share what I learned.

Getting Started (Summary: Learn it by using Linux First)

The first mistake I made was trying to build it against MinGW, so I could avoid working with Cygwin. I'd seen so cool people on dcemulation.org do it that way in the past, and had previously built the tools that way by following one a tutorial. However, it quickly overwhelmed me. There is a lot of great, recent information out there on building GCC, but little of it pertains to Cygwin and far less pertains to MinGW.

I decided I'd bitten off too much to chew and instead opted to get the build working in a Ubuntu VM. I created a script to do this, which I've put on GitHub. Since then, I've been able to build on both a VM and on my Netbook, which runs Lubuntu. I can verify the script works right this minute, which means by the time you read it it will format your primary hard drive and destroy your computer.

So, first lesson: start off in the reference operating system of the tool you're building, especially if you're trying to learn it.

If you're using Ubuntu you can clone the script and run it and probably be fine (if you're using a VM, don't be a wise ass like me and try to make the directory where GCC builds a shared folder or the script will fail). The point of this post is to explain the process in general and how to run it in Cygwin.

Installing the GCC dependencies.

Like anything complex, GCC has a lot of dependencies. If you miss one or more, the build scripts will fail after running for hours and wasting a lot of your time, and the error message won't make any sense either. So its important you get all the dependencies installed right off the bat.

I kept missing things, which scuttled my ability to build GCC for hours. I finally went to the #dreamcastdev channel on FreeNode, where I begged for help. I'd like to thank kenws for providing a list of all the packages necessary in Ubuntu, along with BlueCrab for giving me some great instruction on how the KOS build scripts worked.

Cygwin doesn't have a nice command line package installer like Debian, so to install anything you need you'll need to run the setup.exe which installed Cygwin initially (just download it again if you've misplaced it) and select new packages. It might feel awkard to have to run the installer again just to add packages, but make peace with it as its the only way and doesn't hurt anything. Note that you can run the installer to add packages even as you keep other Cygwin processes open.

When the install GUI gets to “Select Packages”, you'll need to search for and select all of the packages below (this is equivalent to running the “install_gcc_prerequisites” function in the Ubuntu bash script).

• gcc
  
• make
  
• bison
  
• flex
  
• libelf-dev - In Cygwin this is called something else. I had luck installing packages called libelf0-devel and ELFIO.
  
• texinfo
  
• git
  
• wget
  
• sed
  
• lyx
  
• patchutils - This isn't in the bash script, but needs to be installed since Cygwin doesn't have this by default.
  
• libiconv - Ditto.
  

Note that “latex2html” is in the bash script but isn't necessary (it just creates documentation).

At this point running each command executed by the Ubuntu script I created should work (although you'll need to exclude sudo from a few lines). I'll spend the rest of this blog post walking through the functions though and offering some commentary on what exactly is going on.

“download_kos_source” - basically this just git clones the KallistiOS code. I put this at ~/Tools/dreamcast, which means the “tool chain” will be put at ~/Tools/dreamcast/KallistiOS/utils/dc-chain. If anything goes wrong, this is the directory you'll find yourself living in.

“prepare_gcc_source” - This runs a script written by the KOS authors which downloads and unzips the GCC source code. It also runs a script stored in the GCC code base which downloads and installs three big dependencies which the GCC docs always talk about as being necessary, which are gmp, mpfr, and mpc. I got a lot of advice to install these using a package manager, but it turns out once I had the earlier dependencies right, the “./contrib/download_prerequisites” script just worked.

Also, this function creates directories like “/opt/toolchains/dc/sh-elf/sh-elf/include” which is where the built tools eventually end up. I did this because I kept having the build process fail on me at the end because it couldn't store files there. In other OSes or environments such as Cygwin it might not be necessary but since having the build process fail after taking so long is a real drag it's probably worth your time to do what the script says and create those directories.

The last bit of the function calls a make file in the KOS dc-chain directory. According to the KallistiOS docs you can just run “make all” and everything will work from here, but what fun is that? Besides, if anything fails you'll end up confused as to where the process stopped working. So I recommend running it in the order of my bash script so you can have a fuzzy idea of whats going on.

Anyway, “make patch” actually modifies the GCC code using patch files included with the KOS source. As Marty would say, pretty heavy right?

Next up in the script is the functions “build_sh4_tools”, which just calls the KOS dc-chain make file multiple times to build GCC to target the Hitachi 200 MHz SH-4 processor, which is the main CPU the Dreamcast uses.

“make build-sh4-binutils” and “make build-sh4-gcc-pass1” seem to create some preliminary version of GCC which is then used to compile GCC. If that sounds confusing, its because it totally is. I think the name of this proto-GCC is “xgcc” or something.

BTW, if a dependency is missing, this is where fit will hit the shan. If it does, it may be tempting to proceed to the next step, because the scripts will run most of the way before failing, which will give you a brief period of hope. But that hope is an illusion! Don't fall for it! Instead back up, see if you missed a dependency, and try again.

Next up we run “make build-sh4-newlib-only”, “make fixup-sh4-newlib”, and “make build-sh4-gcc-pass2”. The last step creates the SH-4 version of GCC.

After that, it's time to build the ARM processor targetting version of GCC. Once I got the SH4 version to build the ARM version never gave me any problems. My script just runs “make build-arm-binutils” and “make build-arm-gcc”. About the whole proto-version of GCC… there's a gap in my understanding of this process, and I'm not sure why no “pass1” step is necessary like it was with the SH4.

Create an Environ.sh script

If everything else worked, you're nearly home scott-free. Now you just need an “environ.sh” script. This script sets environment variables needed to do anything useful with the Dreamcast tool chain.

I'm creating a bash script in the home directory of Cygwin to do this. Keep in mind, Cygwin maps the “home” directory, denoted by “~”, to C:\cygwin\home\%USER%, where %USER% is your Windows user name. So you can add the file that way too.

Anyway, create a file called “dcdev_environ.sh” (or whatever you want) containing this:

# KallistiOS environment variable settings
export KOS_ARCH='dreamcast'

export KOS_SUBARCH='pristine'

export KOS_BASE="$HOME/Tools/dreamcast/KallistiOS"
# Make utility
export KOS_MAKE="make"

# Load utility
export KOS_LOADER="dc-tool -x"              # dcload, preconfigured

# Genromfs utility
export KOS_GENROMFS="${KOS_BASE}/utils/genromfs/genromfs"

# Compiler prefixes
export KOS_CC_BASE="/opt/toolchains/dc/sh-elf"      # DC
export KOS_CC_PREFIX="sh-elf"

export DC_ARM_BASE="/opt/toolchains/dc/arm-eabi"
export DC_ARM_PREFIX="arm-eabi"

export PATH="${PATH}:${KOS_CC_BASE}/bin:/usr/local/dc/bin"

export KOS_INC_PATHS="-I${KOS_BASE}/../kos-ports/include"

export KOS_CFLAGS=""
export KOS_CPPFLAGS=""
export KOS_LDFLAGS=""
export KOS_AFLAGS=""

export KOS_CFLAGS="-O2 -fomit-frame-pointer"

. ${KOS_BASE}/environ_base.sh

Then, in Cygwin, run the file:

$ source ~/dcdev_environ.sh
echo $KOS_BASE
/home/Tim/Tools/dreamcast/KallistiOS

With $KOS_BASE and other environment variables defined, we're in the home stretch and can build KOS itself.

Building KallistiOS

If all is well, building KOS should be uneventful. Simply enter the directory where you cloned KallistiOS (~/Tools/dreamcast/KallistiOS, aka $KOS_BASE) and type “make”. This creates all the phenomenal Dreamcast library and boot up code needed to write apps on the Dreamcast.

Next up, enter the “kos-ports” directory. This was the second thing that we git cloned way up above. These are a bunch of tools ported to KOS (including lib SDL(!)). Again, just enter the directory (~/Tools/dreamcast/kos-ports) and type “make”.

The End…?

To make sure everything actually worked, try building some of the KOS examples. Enter $KOS_BASE/examples/dreamcast/pvr/pvrmark_strips_direct and type “make”. Do an “ls” and you should see the file “pvrmark_strips_direct.elf.”

Congratulations! You now can now use GCC 4.7.0 to compile code for the Dreamcast! Being able to actually run the code on a real Dreamcast is a topic best left for a fresh blog post.

Macaroni for C++ uses Macaroni for C++ to generate code and then Boost Build to build itself.

I wrote Macaroni for C++ out of a desire to bring the speed at which I could code .NET and Java to C++. Unfortunately, one of the things I ran into is the classical problem that large C++ projects can quickly take forever to build if you aren't careful. From scratch, Macaroni takes between eight and ten minutes to build on my old core 2 duo laptop. On my new, top of the line i7 desktop it builds in over five minutes. On my netbook it takes an hour.

Boost Build is an incredible tool which is hurt not so much by its documentation but by the fact that its complexity is necessary to provide features that are hard to appreciate. Boost Build handles the discrepancies of building code with different compilers, operating systems, and configurations, making it possible to use a single build script that can compile an application linked to shared libraries with Visual C++ in debug mode, then a statically compiled version using MinGW, and finally build a Ubuntu version of the app in a VM.

The ability to choose on the fly the way to compile an application, rather than having to think ahead when writing the build script, is liberating. It's like using a nice multi-platform library such as Boost Filesystem instead of writing the code to access files and directories yourself from scratch for every operating system. Boost Build, along with the Boost libraries, enabled me to quickly create a Linux version of Macaroni, which makes me wonder if any person who advocates something like autotools or make on the basis that they are “universal” has bothered trying a different OS or development environment in the past ten years.

However, Boost Build has a price to pay in the form of it's learning curve. I've spent the last three years figuring it out, and at times have been frustrated to the point where I nearly dropped it. When I first discovered Boost Build, I decided to make it the required build system of Macaroni. After several months of bitter experience I realized instead I needed to find a way to represent the physical layout of the files generated by Macaroni, so that while I would mainly focus on generating files for Boost Build I could create build scripts for other tools if I or anyone else ever wanted to (this was also to generate Visual Studio projects to take advantage of VS's debugger, which is the only reason I start up Visual Studio anymore).

I feel as if I still can't claim to know Boost Build, as I'll find out new things constantly that seem so useful I wonder how I could ever get work done without them. An example of this is the “-j” flag, which I found today.

Normally when Boost Build executes, you can see it calling the compiler once for every object file (in Macaroni there is going to be a one-to-one correlation between classes and object files… I've thought of ways to customize this but so far every idea seems to add too much complexity). On a multicore beast of a machine like my new one that is clearly too boring for the CPU, which sits at 20%.

Enter the “-j” flag, which tells Boost to call multiple processes simultaneously. So “bjam -j4” will invoke four processes simultaneously. This has sped up the build times of Macaroni dramatically.

I wanted to time the differences, so I installed ptime, a useful program that spits out the time taken to invoke some process. I then tested bjam's performance without the -j flag, then with -j4 and finally -j8. Each time I first deleted the bin/msvc-11.0 directory so it would build from scratch, then I invoked it again to see if there was any speed up.

\________________________________________/
|         |   normal |     -j4 |     -j8 |
+---------+----------+---------+---------+
| build   | 310.510s | 94.594s | 69.813s |
| rebuild |   1.067s |  1.060s |  1.059s |
""""""""""""""""""""""""""""""""""""""""""

It turns out there is no speed increase for the parsing process Bjam does initially, so if nothing needs to be built the -j flags add nothing. However, moving to “-j8” made the build four and a half times faster. Its also the only process I've yet seen cause my CPU to jump to 100%.

This flag feels like some kind of unbelievable secret, like when I figured out there was a hidden ending in Link's Awakening, or discovered the #line directive which allowed compile errors to reference Macaroni source code. If it's in the Boost Build docs then it must be buried (though contributions are welcome, so maybe I should shut up and add it). When I, right now, Google specifically for “bjam -j” or “bjam -j4” I see nothing mentioning this flag on the first page of Google results.

The way I discovered the flag initially is by stumbling into an ancient page where someone had pointed out that the “-j” flag interspersed output from the commands it called in parallel. When I read it, I did a double take because I couldn't believe such a useful piece of functionality was real and being referenced so nonchalantly. After bjam seemed to work with the flag, I ran the experiments listed above to prove it to myself that “-j” did would I hoped and that it wasn't some illusion. I've since Googled “bjam -j4” and the only webpage I could find referencing this incredibly useful bit of functionality is this one).

Maybe the flag is officially unsupported? All I know is that I'm grateful it works now and has dramatically decreased the time it takes me to build from scratch.