Awhile ago I decided to invest some time and learn how to build the Dreamcast homebrew programming tools using the latest source (C++11 for the win!). These include a customized version of the GNU compiler collection along with the KallistiOS operating system.

I've finally gotten it working, and I feel like I somewhat understand the process, so I thought I'd share what I learned.

Getting Started (Summary: Learn it by using Linux First)

The first mistake I made was trying to build it against MinGW, so I could avoid working with Cygwin. I'd seen so cool people on dcemulation.org do it that way in the past, and had previously built the tools that way by following one a tutorial. However, it quickly overwhelmed me. There is a lot of great, recent information out there on building GCC, but little of it pertains to Cygwin and far less pertains to MinGW.

I decided I'd bitten off too much to chew and instead opted to get the build working in a Ubuntu VM. I created a script to do this, which I've put on GitHub. Since then, I've been able to build on both a VM and on my Netbook, which runs Lubuntu. I can verify the script works right this minute, which means by the time you read it it will format your primary hard drive and destroy your computer.

So, first lesson: start off in the reference operating system of the tool you're building, especially if you're trying to learn it.

If you're using Ubuntu you can clone the script and run it and probably be fine (if you're using a VM, don't be a wise ass like me and try to make the directory where GCC builds a shared folder or the script will fail). The point of this post is to explain the process in general and how to run it in Cygwin.

Installing the GCC dependencies.

Like anything complex, GCC has a lot of dependencies. If you miss one or more, the build scripts will fail after running for hours and wasting a lot of your time, and the error message won't make any sense either. So its important you get all the dependencies installed right off the bat.

I kept missing things, which scuttled my ability to build GCC for hours. I finally went to the #dreamcastdev channel on FreeNode, where I begged for help. I'd like to thank kenws for providing a list of all the packages necessary in Ubuntu, along with BlueCrab for giving me some great instruction on how the KOS build scripts worked.

Cygwin doesn't have a nice command line package installer like Debian, so to install anything you need you'll need to run the setup.exe which installed Cygwin initially (just download it again if you've misplaced it) and select new packages. It might feel awkard to have to run the installer again just to add packages, but make peace with it as its the only way and doesn't hurt anything. Note that you can run the installer to add packages even as you keep other Cygwin processes open.

When the install GUI gets to “Select Packages”, you'll need to search for and select all of the packages below (this is equivalent to running the “install_gcc_prerequisites” function in the Ubuntu bash script).

• gcc
  
• make
  
• bison
  
• flex
  
• libelf-dev - In Cygwin this is called something else. I had luck installing packages called libelf0-devel and ELFIO.
  
• texinfo
  
• git
  
• wget
  
• sed
  
• lyx
  
• patchutils - This isn't in the bash script, but needs to be installed since Cygwin doesn't have this by default.
  
• libiconv - Ditto.
  

Note that “latex2html” is in the bash script but isn't necessary (it just creates documentation).

At this point running each command executed by the Ubuntu script I created should work (although you'll need to exclude sudo from a few lines). I'll spend the rest of this blog post walking through the functions though and offering some commentary on what exactly is going on.

“download_kos_source” - basically this just git clones the KallistiOS code. I put this at ~/Tools/dreamcast, which means the “tool chain” will be put at ~/Tools/dreamcast/KallistiOS/utils/dc-chain. If anything goes wrong, this is the directory you'll find yourself living in.

“prepare_gcc_source” - This runs a script written by the KOS authors which downloads and unzips the GCC source code. It also runs a script stored in the GCC code base which downloads and installs three big dependencies which the GCC docs always talk about as being necessary, which are gmp, mpfr, and mpc. I got a lot of advice to install these using a package manager, but it turns out once I had the earlier dependencies right, the “./contrib/download_prerequisites” script just worked.

Also, this function creates directories like “/opt/toolchains/dc/sh-elf/sh-elf/include” which is where the built tools eventually end up. I did this because I kept having the build process fail on me at the end because it couldn't store files there. In other OSes or environments such as Cygwin it might not be necessary but since having the build process fail after taking so long is a real drag it's probably worth your time to do what the script says and create those directories.

The last bit of the function calls a make file in the KOS dc-chain directory. According to the KallistiOS docs you can just run “make all” and everything will work from here, but what fun is that? Besides, if anything fails you'll end up confused as to where the process stopped working. So I recommend running it in the order of my bash script so you can have a fuzzy idea of whats going on.

Anyway, “make patch” actually modifies the GCC code using patch files included with the KOS source. As Marty would say, pretty heavy right?

Next up in the script is the functions “build_sh4_tools”, which just calls the KOS dc-chain make file multiple times to build GCC to target the Hitachi 200 MHz SH-4 processor, which is the main CPU the Dreamcast uses.

“make build-sh4-binutils” and “make build-sh4-gcc-pass1” seem to create some preliminary version of GCC which is then used to compile GCC. If that sounds confusing, its because it totally is. I think the name of this proto-GCC is “xgcc” or something.

BTW, if a dependency is missing, this is where fit will hit the shan. If it does, it may be tempting to proceed to the next step, because the scripts will run most of the way before failing, which will give you a brief period of hope. But that hope is an illusion! Don't fall for it! Instead back up, see if you missed a dependency, and try again.

Next up we run “make build-sh4-newlib-only”, “make fixup-sh4-newlib”, and “make build-sh4-gcc-pass2”. The last step creates the SH-4 version of GCC.

After that, it's time to build the ARM processor targetting version of GCC. Once I got the SH4 version to build the ARM version never gave me any problems. My script just runs “make build-arm-binutils” and “make build-arm-gcc”. About the whole proto-version of GCC… there's a gap in my understanding of this process, and I'm not sure why no “pass1” step is necessary like it was with the SH4.

Create an Environ.sh script

If everything else worked, you're nearly home scott-free. Now you just need an “environ.sh” script. This script sets environment variables needed to do anything useful with the Dreamcast tool chain.

I'm creating a bash script in the home directory of Cygwin to do this. Keep in mind, Cygwin maps the “home” directory, denoted by “~”, to C:\cygwin\home\%USER%, where %USER% is your Windows user name. So you can add the file that way too.

Anyway, create a file called “dcdev_environ.sh” (or whatever you want) containing this:

# KallistiOS environment variable settings
export KOS_ARCH='dreamcast'

export KOS_SUBARCH='pristine'

export KOS_BASE="$HOME/Tools/dreamcast/KallistiOS"
# Make utility
export KOS_MAKE="make"

# Load utility
export KOS_LOADER="dc-tool -x"              # dcload, preconfigured

# Genromfs utility
export KOS_GENROMFS="${KOS_BASE}/utils/genromfs/genromfs"

# Compiler prefixes
export KOS_CC_BASE="/opt/toolchains/dc/sh-elf"      # DC
export KOS_CC_PREFIX="sh-elf"

export DC_ARM_BASE="/opt/toolchains/dc/arm-eabi"
export DC_ARM_PREFIX="arm-eabi"

export PATH="${PATH}:${KOS_CC_BASE}/bin:/usr/local/dc/bin"

export KOS_INC_PATHS="-I${KOS_BASE}/../kos-ports/include"

export KOS_CFLAGS=""
export KOS_CPPFLAGS=""
export KOS_LDFLAGS=""
export KOS_AFLAGS=""

export KOS_CFLAGS="-O2 -fomit-frame-pointer"

. ${KOS_BASE}/environ_base.sh

Then, in Cygwin, run the file:

$ source ~/dcdev_environ.sh
echo $KOS_BASE
/home/Tim/Tools/dreamcast/KallistiOS

With $KOS_BASE and other environment variables defined, we're in the home stretch and can build KOS itself.

Building KallistiOS

If all is well, building KOS should be uneventful. Simply enter the directory where you cloned KallistiOS (~/Tools/dreamcast/KallistiOS, aka $KOS_BASE) and type “make”. This creates all the phenomenal Dreamcast library and boot up code needed to write apps on the Dreamcast.

Next up, enter the “kos-ports” directory. This was the second thing that we git cloned way up above. These are a bunch of tools ported to KOS (including lib SDL(!)). Again, just enter the directory (~/Tools/dreamcast/kos-ports) and type “make”.

The End…?

To make sure everything actually worked, try building some of the KOS examples. Enter $KOS_BASE/examples/dreamcast/pvr/pvrmark_strips_direct and type “make”. Do an “ls” and you should see the file “pvrmark_strips_direct.elf.”

Congratulations! You now can now use GCC 4.7.0 to compile code for the Dreamcast! Being able to actually run the code on a real Dreamcast is a topic best left for a fresh blog post.

Macaroni for C++ uses Macaroni for C++ to generate code and then Boost Build to build itself.

I wrote Macaroni for C++ out of a desire to bring the speed at which I could code .NET and Java to C++. Unfortunately, one of the things I ran into is the classical problem that large C++ projects can quickly take forever to build if you aren't careful. From scratch, Macaroni takes between eight and ten minutes to build on my old core 2 duo laptop. On my new, top of the line i7 desktop it builds in over five minutes. On my netbook it takes an hour.

Boost Build is an incredible tool which is hurt not so much by its documentation but by the fact that its complexity is necessary to provide features that are hard to appreciate. Boost Build handles the discrepancies of building code with different compilers, operating systems, and configurations, making it possible to use a single build script that can compile an application linked to shared libraries with Visual C++ in debug mode, then a statically compiled version using MinGW, and finally build a Ubuntu version of the app in a VM.

The ability to choose on the fly the way to compile an application, rather than having to think ahead when writing the build script, is liberating. It's like using a nice multi-platform library such as Boost Filesystem instead of writing the code to access files and directories yourself from scratch for every operating system. Boost Build, along with the Boost libraries, enabled me to quickly create a Linux version of Macaroni, which makes me wonder if any person who advocates something like autotools or make on the basis that they are “universal” has bothered trying a different OS or development environment in the past ten years.

However, Boost Build has a price to pay in the form of it's learning curve. I've spent the last three years figuring it out, and at times have been frustrated to the point where I nearly dropped it. When I first discovered Boost Build, I decided to make it the required build system of Macaroni. After several months of bitter experience I realized instead I needed to find a way to represent the physical layout of the files generated by Macaroni, so that while I would mainly focus on generating files for Boost Build I could create build scripts for other tools if I or anyone else ever wanted to (this was also to generate Visual Studio projects to take advantage of VS's debugger, which is the only reason I start up Visual Studio anymore).

I feel as if I still can't claim to know Boost Build, as I'll find out new things constantly that seem so useful I wonder how I could ever get work done without them. An example of this is the “-j” flag, which I found today.

Normally when Boost Build executes, you can see it calling the compiler once for every object file (in Macaroni there is going to be a one-to-one correlation between classes and object files… I've thought of ways to customize this but so far every idea seems to add too much complexity). On a multicore beast of a machine like my new one that is clearly too boring for the CPU, which sits at 20%.

Enter the “-j” flag, which tells Boost to call multiple processes simultaneously. So “bjam -j4” will invoke four processes simultaneously. This has sped up the build times of Macaroni dramatically.

I wanted to time the differences, so I installed ptime, a useful program that spits out the time taken to invoke some process. I then tested bjam's performance without the -j flag, then with -j4 and finally -j8. Each time I first deleted the bin/msvc-11.0 directory so it would build from scratch, then I invoked it again to see if there was any speed up.

\________________________________________/
|         |   normal |     -j4 |     -j8 |
+---------+----------+---------+---------+
| build   | 310.510s | 94.594s | 69.813s |
| rebuild |   1.067s |  1.060s |  1.059s |
""""""""""""""""""""""""""""""""""""""""""

It turns out there is no speed increase for the parsing process Bjam does initially, so if nothing needs to be built the -j flags add nothing. However, moving to “-j8” made the build four and a half times faster. Its also the only process I've yet seen cause my CPU to jump to 100%.

This flag feels like some kind of unbelievable secret, like when I figured out there was a hidden ending in Link's Awakening, or discovered the #line directive which allowed compile errors to reference Macaroni source code. If it's in the Boost Build docs then it must be buried (though contributions are welcome, so maybe I should shut up and add it). When I, right now, Google specifically for “bjam -j” or “bjam -j4” I see nothing mentioning this flag on the first page of Google results.

The way I discovered the flag initially is by stumbling into an ancient page where someone had pointed out that the “-j” flag interspersed output from the commands it called in parallel. When I read it, I did a double take because I couldn't believe such a useful piece of functionality was real and being referenced so nonchalantly. After bjam seemed to work with the flag, I ran the experiments listed above to prove it to myself that “-j” did would I hoped and that it wasn't some illusion. I've since Googled “bjam -j4” and the only webpage I could find referencing this incredibly useful bit of functionality is this one).

Maybe the flag is officially unsupported? All I know is that I'm grateful it works now and has dramatically decreased the time it takes me to build from scratch.

For years I've owned an Acer Aspire One netbook using Windows XP. While I've stayed fond of the idea of using it, for roughly half that time Windows XP and the certain programs such as web browsers have become degraded in performance to the point that using the system is no longer enjoyable (this is in spite of several reinstalls of Windows - maybe Service Patch 3 for XP has a feature which drastically decreases performance on Netbooks?).

I installed Ubuntu, but while faster, it didn't quite feel great either. For one thing, the Unity UI left-hand dock takes up an unforgivable amount of real estate on the netbook screen. For another, while Ubuntu loads fast and generally was a bit zippier than XP, there was still noticeable pauses as I switched between tasks or opened up new programs. My biggest pet peeve was the update center popping up and stealing focus from every other program a few minutes into each session despite the fact it then took a few seconds more to fully load.

Thankfully, there are other distros of Ubuntu with lighter-weight UIs meant for older hardware. I was assumed I'd be using Xubuntu, due to its important feature of having a mouse for a logo. But after reading some benchmarks, I settled on Lubuntu, which uses LXDE, a thin layer on top of X11 with little inbetween.

Lubuntu is just a rocket ship compared to everything else I've run on this netbook. It loads fast, switches apps fast, and so far everything just works. It has some failings, such as the inability to snap Windows to a grid using a keyboard shortcut, but most of these are more forgivable on a Netbook where I usually keep apps full screen and switch between them.

After that, I just had to tweak a few things to get an environment I was comfortable on. Here's a list of steps I recommend if you're planning on running Lubuntu:

• As always, install Launchy. While some bemoan where a start menu is or isn't placed, others use Launchy and forget details like that.
  
• Disable the ALT+SPACE keyboard binding, so Launchy can use it.
  
• Make Launchy start at boot. Here's a good article on how to accomplish this.
  
• Install the incredible Sublime Text 2.
  
• In order for Launchy to detect programs, such as Sublime, you'll need to add shortcuts to the desktop menu, even though you won't be using it. Here's how I did it:
  

  $ cd /usr/share/applications
  $ sudo vi sublime.desktop
  

  
  Then I added the following in VI:
  

  [Desktop Entry]
  Type=Application
  Name=Sublime Text 2
  GenericName=
  Comment=
  Icon=/home/tim/Tools/sublime/SublimeText2/Icon/48x48/sublime_text.png
  Exec=/home/tim/Tools/sublime/SublimeText2/sublime_text
  Terminal=false
  Categories=Office;IDE;
  

  
  
  
• Similar to when I'm in Windows, I like to have a bash script that runs every time I start up a terminal. I used to edit bash.bashrc directly, but now I simply add a line to bash.bashrc to source a script that lives somewhere under ~/Tools.
  
  In Windows, the main purpose of the analogous bash script was to set the path to use various command line utilities, though in Linux its better not to defy convention and instead let most programs live where the debian package says they should. However, a start up script can still be useful for certain programs that don't have packages as well as programs that do have them but which you wish to install from source in a directory of your choosing. An extra start up script can also do fun things like displaying tips fortune on startup. For example, here is my current script:
  
  

  MPATH=~/Tools/Common/U/macaroni/macaroni-0.1.0.25-ubuntu
  alias macaroni="$MPATH/macaroni generatorPath $MPATH/Generators \
      libraryRepoPath $MPATH/Libraries messagesPath $MPATH/"
  export BOOST_ROOT=~/Tools/Local/boost/boost_1_52_0
  export PATH=$PATH:$BOOST_ROOT
  fortune | cowsay
  date
  

  
  
  

I'm not being hyperbolic when I say Lubuntu has made my Netbook feel like a whole new machine. If you're in a similar situation with an older Netbook or Laptop, I highly recommend it. It's amazing to me that there's still software out there that's this fast on old hardware.

I recently got Macaroni compiled in Ubuntu and passing all the unit tests. The integration tests weren't so lucky, perhaps due to configuration issues in how I'm using Boost Build. Despite that, I've released the Ubuntu build on Macaroni's homepage. Here's a little tutorial on how to use it in Ubuntu, which I'm writing as I run Macaroni for the first time in a fresh virtual machine.

First, download Macaroni from this link. I unzipped mine to ~/Tools/macaroni, so the final directory was ~/Tools/macaroni/macaroni-0.1.0.25-ubuntu-64.

The Macaroni binary won't have the execute permission set, so fix that with:

$ chmod 700 ~/Tools/macaroni/macaroni-0.1.0.25-ubuntu-64/macaroni

Macaroni needs a settings file to be present at ~/Macaroni. This is for global settings which need to be changed on a per-machine basis, though for this project we won't need it all.

$ mkdir -p ~/Macaroni
$ echo "properties={}" > ~/Macaroni/init.lua

Unlike the Windows version, we'll need to specify each path to the various Lua scripts (the ability to avoid this might be coming to the Linux version soon, but currently it seems a little complicated). So we'll create an alias to do all this for us.

$ MPATH=~/Tools/macaroni/macaroni-0.1.0.25-ubuntu-64
$ alias mcpp="$MPATH/macaroni generatorPath $MPATH/Generators \
    libraryRepoPath $MPATH/Libraries messagesPath $MPATH/"

At this point typing “mcpp” will run Macaroni.

Create a new directory for an example project. Mine is ~/Work/mexample. In that directory, create the following file called “project.lua”:

import('Macaroni', 'ProjectTemplates', '1')
require 'SimpleProject'
GenerateCpp{group='Junk', project='Junk', version='0'};

Of course, we'll need a Macaroni source file in the “src” directory. First, create the “src” directory, and then create the following file in that directory named “Example.mcpp” (actually, it can be called anything at all so long as it ends in .mcpp).

~import std::cout;

class Class1 { public Class1() { cout << "Class1!" } };

class Class2 { public Class2() { cout << "Class2!" } };

Finally, generate C++ code by running the following:

$ mcpp generate
Macaroni for C++
Version 0.1.0.25 built at 2012-12-11-10:45:52

Creating new Environment with the following AppPaths:
Message file : /home/tim/Tools/macaroni/macaroni-0.1.0.25-ubuntu-64/
Library Paths : [/home/tim/Tools/macaroni/macaroni-0.1.0.25-ubuntu-64/Libraries/home/tim/Macaroni/Libraries]
Generator Paths : [/home/tim/Tools/macaroni/macaroni-0.1.0.25-ubuntu-64/Generators]
LoadProjectEnvironment
Parsing file project.lua...
Running /home/tim/Tools/macaroni/macaroni-0.1.0.25-ubuntu-64/Libraries/Macaroni/Project-Lua/0/src/project-init.lua...
Running /home/tim/Macaroni/init.lua...
LoadProjectEnvironment
Parsing file /home/tim/Tools/macaroni/macaroni-0.1.0.25-ubuntu-64/Libraries/Macaroni/ProjectTemplates/1/project-final.lua...
Running /home/tim/Tools/macaroni/macaroni-0.1.0.25-ubuntu-64/Libraries/Macaroni/Project-Lua/0/src/project-init.lua...
Running /home/tim/Macaroni/init.lua...

Program finished.

$ ls -la target
total 24
drwxrwxr-x 2 tim tim 4096 Dec 12 00:07 .
drwxrwxr-x 4 tim tim 4096 Dec 12 00:07 ..
-rw-rw-r-- 1 tim tim 1329 Dec 12 00:07 Class1.cpp
-rw-rw-r-- 1 tim tim  366 Dec 12 00:07 Class1.h
-rw-rw-r-- 1 tim tim 1329 Dec 12 00:07 Class2.cpp
-rw-rw-r-- 1 tim tim  366 Dec 12 00:07 Class2.h
$ cat target/Class1.h
#ifndef MACARONI_COMPILE_GUARD_Class1_H
#define MACARONI_COMPILE_GUARD_Class1_H

// This class was originally defined in Example.mcpp
// Forward declaration necessary for anything which also depend on this.
class Class1;

// Define class Class1
class  Class1
{
	public    : Class1();

}; // End of class Class1

#endif // end of MACARONI_COMPILE_GUARD_Class1_H

Hurray! You've now used Macaroni. If all you're interested in is getting rid of the duplication between header and implementation files, then you're good.

Hopefully I'll soon find time to write more blog posts soon detailing how to use Macaroni to do other things such as build projects using Boost Build, write glue code for Lua, as well as some other entries explaining the ideas behind Macaroni for anyone who's interested in writing their own code generators.