Zig's New CLI Progress Bar Explained

Sometimes, programming projects are too easy and boring. Sometimes, they're too hard, never ending or producing subpar results.

This past week I had the pleasure of completing a project that felt like maximum difficulty - only possible because I am at the top of my game, using a programming language designed for making perfect software. This problem threw everything it had at me, but I rose to the challenge and emerged victorious.

What a rush.

In this blog post I'll dig into the technical implementation as well as provide the Zig Progress Protocol Specification.

Demo

Before we take a deep dive, let's look at the final results by building my music player side project while recording with Asciinema:

Old vs New

The usage code looks basically like this:

const parent_progress_node = std.Progress.start(.{});

// ...

const progress_node = parent_progress_node.start("sub-task name", 10);
defer progress_node.end();

for (0..10) |_| progress_node.completeOne();

To include a child process's progress under a particular node, it's a single assignment before calling spawn:

child_process.progress_node = parent_progress_node;

For me the most exciting thing about this is its ability to visualize what the Zig Build System is up to after you run zig build. Before this feature was even merged into master branch, it led to discovery, diagnosis, and resolution of a subtle bug that has hidden in Zig's standard library child process spawning code for years. Not included in this blog post: a rant about how much I hate the fork() API.

Motivation

The previous implementation was more conservative. It had the design limitation that it could not assume ownership of the terminal. This meant that it had to assume another process or thread could print to stderr at any time, and it was not allowed to register a SIGWINCH signal handler to learn about when the terminal size changed. It also did not rely on knowledge of the terminal size, or spawn any threads.

This new implementation represents a more modern philosophy: when a CLI application is spawned with stderr being a terminal, then that application owns the terminal and owes its users the best possible user experience, taking advantage of all the terminal features available, working around their limitations. I'm excited for projects such as Ghostty which are expanding the user interface capabilities of terminals, and lifting those limitations.

With this new set of constraints in mind, it becomes possible to design a much more useful progress bar. We gain a new requirement: since only one process owns the terminal, child processes must therefore report their progress semantically so it can be aggregated and displayed by the terminal owner.

Implementation

The whole system is designed around the public API, which must be thread-safe, lock-free, and avoid contention as much as possible in order to prevent the progress system itself from harming performance, particularly in a multi-threaded environment.

The key insight I had here is that, since the end result must be displayed on a terminal screen, there is a reasonably small upper bound on how much memory is required, beyond which point the extra memory couldn't be utilized because it wouldn't fit on the terminal screen anyway.

By statically pre-allocating 200 nodes, the API is made infallible and non-heap-allocating. Furthermore, it makes it possible to implement a thread-safe, lock-free node allocator based on a free list.

The shared data is split into several arrays:

node_parents: [200]Node.Parent,
node_storage: [200]Node.Storage,
node_freelist: [200]Node.OptionalIndex,

The freelist is used to ensure that two racing Node.start() calls obtain different indexes to operate on, as well as a mechanism for Node.end() calls to return nodes back to the system for reuse. Meanwhile, the parents array is used to indicate which nodes are actually allocated, and to which parent node they should be attached to. Finally, the other storage contains the number of completed items, estimated total items, and task name for each node.

Each Node.Parent is 1 byte exactly. It has 2 special values, which can be represented with type safety in Zig:

const Parent = enum(u8) {
    /// Unallocated storage.
    unused = std.math.maxInt(u8) - 1,
    /// Indicates root node.
    none = std.math.maxInt(u8),
    /// Index into `node_storage`.
    _,

    fn unwrap(i: @This()) ?Index {
        return switch (i) {
            .unused, .none => return null,
            else => @enumFromInt(@intFromEnum(i)),
        };
    }
};

The data mentioned above is mutated in the implementation of the thread-safe public API.

Meanwhile, the update thread is running on a timer. After an initial delay, it wakes up at regular intervals to either draw progress to the terminal, or send progress information to another process through a pipe.

In either case, when the update thread wakes up, the first thing it does is "serialize" the shared data into a separate preallocated location. After carefully copying the shared data using atomic primitives, the copied, serialized data can then be operated on in a single-threaded manner, since it is not shared with any other threads.

The update thread performs this copy by iterating over the full 200 shared parents array, atomically loading each value and checking if it is not the special "unused" value (0xfe). In most programming projects, a linear scan to find allocated objects is undesirable, however in this case 200 bytes for node parent indexes is practically free to iterate over since it's 4 cache lines total, and the only contention comes from actual updates that must be observed. This full scan in the update thread buys us a cheap, lock-free implementation for Node.start() and Node.end() via popping and pushing the freelist, respectively.

Next, IPC nodes are expanded. More on this point later.

Once the serialization process is complete, we are left with a subset of the shared data from earlier, this time with no gaps - only used nodes are present here:

serialized_parents: [200]Node.Parent,
serialized_storage: [200]Node.Storage,
serialized_len: usize,

At this point the behavior diverges depending on whether the current process owns the terminal, or must send progress updates via a pipe. A process that fits neither of these categories would not have spawned an update thread to begin with. Any parent process that wants to track progress from children creates a pipe with O_NONBLOCK enabled, passing it to the child as if it were a fourth I/O stream after stdin, stdout, and stderr. To indicate to the process that the file descriptor is in fact a progress pipe, it sets the ZIG_PROGRESS environment variable. For example, in the Zig standard library implementation, this ends up being ZIG_PROGRESS=3.

A process that is given a progress pipe sends the serialized data over the pipe, while a process that owns the terminal draws it directly.

For example, this source code:

const std = @import("std");

pub fn main() !void {
    const root_node = std.Progress.start(.{
        .root_name = "preparing assets",
    });
    defer root_node.end();

    const sub_node = root_node.start("reticulating splines", 100);
    defer sub_node.end();

    for (0..50) |_| sub_node.completeOne();
    std.time.sleep(1000 * std.time.ns_per_ms);
    for (0..50) |_| sub_node.completeOne();
}

At the sleep() call, would display this to the terminal:

preparing assets
└─ [50/100] reticulating splines

But if it were a child process, would send this message over the pipe instead:

0000  02 00 00 00 00 00 00 00  00 70 72 65 70 61 72 69  .........prepari
00a0  6E 67 20 61 73 73 65 74  73 00 00 00 00 00 00 00  ng assets.......
00b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................
00c0  00 32 00 00 00 64 00 00  00 72 65 74 69 63 75 6C  .2...d...reticul
00d0  61 74 69 6E 67 20 73 70  6C 69 6E 65 73 00 00 00  ating splines...
00e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................
00f0  00 FF 00                                          ...

Drawing to the Terminal

Drawing to the terminal begins with the serialized data copy. This data contains only edges pointing to parents, and therefore cannot be used to walk the tree starting from the root. The first thing done here is compute sibling and children edges into preallocated buffers so that we can then walk the tree top-down.

child: [200]Node.OptionalIndex,
sibling: [200]Node.OptionalIndex,

Next, the draw buffer is computed, but not written to the terminal yet. It looks like this:

1. Start sync sequence. This makes high framerate terminals not blink rapidly.
2. Clear previous update by moving cursor up times the number of newlines outputted last time, and then a clear to end of screen escape sequence.
3. Recursively walk the tree of nodes, which which is now possible since we have computed children and sibling edges, outputting tree-drawing sequences, node names, and counting newlines.
4. End sync sequence.

This draw buffer is only computed - it is not sent to the write() syscall yet. At this point, we try to obtain the stderr lock. If we get it, then we write the buffer to the terminal. Otherwise, this update is dropped.

If any attempt to write to the terminal fails, the update thread exits so that no further attempt is made.

Inter-Process Communication

Earlier, I said "IPC nodes are expanded" without further explanation. Let's dig into that a little bit.

As a final step in the serialization process, the update thread iterates the data, looking for special nodes. Special nodes store the progress pipe file descriptor of a child process rather than completed_items and estimated_total_items. The update thread reads progress data from this pipe, and then grafts the child's sub-tree onto the parent's tree, plugging the root node of the child into the special node of the parent. The main storage data can be directly memcpy'd, but the parents array must be relocated based on the offset within the serialized data arrays.

In case of a big-endian system, the 2 integers within each node storage must be byte-swapped. Not relying on host endianness means that edge cases continue to work, such as the parent process running on an x86_64 host, with a child process running a mips program in QEMU user mode. This is a real use case when testing the Zig compiler, for example.

The parent process ignores all but the last message from the pipe fd, and references a copy of the data from the last update in case there is no message in the pipe for a particular update.

Performance

Here are some performance data points I took while working on this.

Building the Zig compiler as a sub-process. This measures the cost of the new API implementations, primarily Node.start, Node.end, and the disappearance of Node.activate. In this case, standard error is not a terminal, and thus an update thread is never spawned:

Benchmark 1 (3 runs): old zig build-exe self-hosted compiler
  measurement          mean ± σ            min … max           outliers         delta
  wall_time          51.8s  ±  138ms    51.6s  … 51.9s           0 ( 0%)        0%
  peak_rss           4.57GB ±  286KB    4.57GB … 4.57GB          0 ( 0%)        0%
  cpu_cycles          273G  ±  360M      273G  …  274G           0 ( 0%)        0%
  instructions        487G  ±  105M      487G  …  487G           0 ( 0%)        0%
  cache_references   19.0G  ± 31.8M     19.0G  … 19.1G           0 ( 0%)        0%
  cache_misses       3.87G  ± 12.0M     3.86G  … 3.88G           0 ( 0%)        0%
  branch_misses      2.22G  ± 3.44M     2.21G  … 2.22G           0 ( 0%)        0%
Benchmark 2 (3 runs): new zig build-exe self-hosted compiler
  measurement          mean ± σ            min … max           outliers         delta
  wall_time          51.5s  ±  115ms    51.4s  … 51.7s           0 ( 0%)          -  0.4% ±  0.6%
  peak_rss           4.58GB ±  190KB    4.58GB … 4.58GB          0 ( 0%)          +  0.1% ±  0.0%
  cpu_cycles          272G  ±  494M      272G  …  273G           0 ( 0%)          -  0.4% ±  0.4%
  instructions        487G  ± 63.5M      487G  …  487G           0 ( 0%)          -  0.1% ±  0.0%
  cache_references   19.1G  ± 16.9M     19.1G  … 19.1G           0 ( 0%)          +  0.3% ±  0.3%
  cache_misses       3.86G  ± 17.1M     3.84G  … 3.88G           0 ( 0%)          -  0.2% ±  0.9%
  branch_misses      2.23G  ± 5.82M     2.22G  … 2.23G           0 ( 0%)          +  0.4% ±  0.5%

Building the Zig compiler, with time zig build-exe -fno-emit-bin ... so that the progress is updating the terminal on a regular interval:

• Old:
  • 4.115s
  • 4.216s
  • 4.221s
  • 4.227s
  • 4.234s
• New (1% slower):
  • 4.231s
  • 4.240s
  • 4.271s
  • 4.339s
  • 4.340s

Building my music player application with zig build, with the project-local cache cleared. This displays a lot of progress information to the terminal. This data point accounts for many sub processes sending progress information over a pipe to the parent process for aggregation:

• Old
  • 65.74s
  • 66.39s
  • 71.09s
• New (1% faster)
  • 65.51s
  • 65.88s
  • 66.09s

Conclusion? It appears that I succeeded in making this progress-reporting system have no significant effect on the performance of the software that uses it.

The Zig Progress Protocol Specification

Any programming language can join the fun! Here is a specification so that any software project can participate in a standard way of sharing progress information between parent and child processes.

When the ZIG_PROGRESS=X environment variable is present, where X is an unsigned decimal integer in range 0...65535, a process-wide progress reporting channel is available.

The integer is a writable file descriptor opened in non-blocking mode. Subsequent messages to the stream supersede previous ones.

The stream supports exactly one kind of message that looks like this:

1. len: u8 - number of nodes, limited to 253 max, reserving 0xfe and 0xff for special meaning.
2. 48 bytes for every len:
   1. completed: u32le - how many items already done
   2. estimated_total: u32le - guessed number of items to be completed, or 0 (unknown)
   3. name: [40]u8 - task description; remaining bytes zeroed out
3. 1 byte for every len:
   1. parent: u8 - creates an edge to a parent node in the tree, or 0xff (none)

Future versions of this protocol, if necessary, will use different environment variable names.

Bonus: Using Zig Standard Library in C Code

Much of the Zig standard library is available to C programs with minimal integration pain. Here's a full, working example:

example.c

#include "zp.h"
#include <string.h>
#include <unistd.h>

int main(int argc, char **argv) {
    zp_node root_node = zp_init();

    const char *task_name = "making orange juice";
    zp_node sub_node = zp_start(root_node, task_name, strlen(task_name), 5);
    for (int i = 0; i < 5; i += 1) {
        zp_complete_one(sub_node);
        sleep(1);
    }
    zp_end(sub_node);
    zp_end(root_node);
}

zp.h

#include <stdint.h>
#include <stddef.h>

typedef uint8_t zp_node;

zp_node zp_init(void);
zp_node zp_start(zp_node parent, const char *name_ptr, size_t name_len, size_t estimated_total);
zp_node zp_end(zp_node node);
zp_node zp_complete_one(zp_node node);

zp.zig

const std = @import("std");

export fn zp_init() std.Progress.Node.OptionalIndex {
    return std.Progress.start(.{}).index;
}

export fn zp_start(
    parent: std.Progress.Node.OptionalIndex,
    name_ptr: [*]const u8,
    name_len: usize,
    estimated_total_items: usize,
) std.Progress.Node.OptionalIndex {
    const node: std.Progress.Node = .{ .index = parent };
    return node.start(name_ptr[0..name_len], estimated_total_items).index;
}

export fn zp_end(node_index: std.Progress.Node.OptionalIndex) void {
    const node: std.Progress.Node = .{ .index = node_index };
    node.end();
}

export fn zp_complete_one(node_index: std.Progress.Node.OptionalIndex) void {
    const node: std.Progress.Node = .{ .index = node_index };
    node.completeOne();
}

pub const _start = void;

Compile with zig cc -o example example.c zp.zig

Asciinema Demo

This same executable will also work correctly as a child process, reporting progress over the ZIG_PROGRESS pipe if provided!

Follow-Up Work

Big thanks to Ryan Liptak for helping me with the Windows console printing code, and Jacob Young for working on the Windows IPC logic, which isn't done yet.

Redict was originally created by Salvatore Sanfilippo under the name "Redis". Around 2018 he started losing interest in the project to pursue a science fiction career and gave stewardship of the project to Redis Labs.

I think that was an unfortunate move because their goal is mainly to extract profit from the software project rather than to uphold the ideals of Free and Open Source Software. On March 20, 2024, they changed the license to be proprietary, a widely unpopular action. You know someone is up to no good when they write "Live long and prosper 🖖" directly above a meme of Darth Vader.

What are the actual problems with these licenses?

In short summary, the licenses limit the freedoms of what one can do with the software in order for Redis Labs to be solely enriched, while asking for volunteer labor, and having already benefitted from volunteer labor, that is and was generally offered only because it enriches everybody.

SSPL is BAD

Are they allowed to do this?

All the code before the license change is available under the previous license (BSD-3), however it is perfectly legal to make further changes to the project under a different license.

This means that it is also legal to fork the project from before the license change, and continue maintaining the project without the proprietary license change. The only problem there would be, the project would be missing out on all those juicy future contributions from Redis Labs... wait a minute, isn't the project already done?

Redict is a Finished Product

Redict already works great. Lots of companies already use it in production and have been doing so for many years.

In Why We Can't Have Nice Software, I point out this pattern of needless software churn in the mindless quest for profit. This is a perfect example occurring right now. Redict has already reached its peak; it does not need any more serious software development to occur. It does not need to pivot to AI. It can be maintained for decades to come with minimal effort. It can continue to provide a high amount of value for a low amount of labor. That's the entire point of software!

Redict does not have any profit left to offer. It no longer needs a fund-raising entity behind it anymore. It just needs a good project steward.

Drew DeVault is a Good Steward

Drew is a controversial person, I think for two reasons.

One, is that he has a record of being rude to many people in the past - including myself. However, in a similar manner as Linus Torvalds, Drew has expressed what I can only interpret as sincere regret for such interactions, as well as a pattern of improved behavior. I was poking through his blog to try to find example posts of what I mean, and it's difficult to pick them out because he's such a prolific writer, but perhaps this one or maybe this one. I'm a strong believer in applying the best game theory strategy to society: people should have consequences for the harm that they do, but then they should get a chance to start cooperating again. I can certainly think of "cancelable" things I have done in the past, that I am thankful are not public, and I cringe every time I remember them.

Secondly, and I think this is actually the more important point, Drew has been an uncompromising advocate of Free and Open Source Software his entire life, walking the walk more than anyone else I can think of. It's crystal clear that this is the driving force of his core ideology that determines all of his decision making. He doesn't budge on any of these principles and it creates conflicts with people who are trying to exploit FOSS for their own gains. For example, when you call out SourceGraph you basically piss off everyone who has SourceGraph stock. Do enough of these callouts, and you've pissed off enough people that there's an entire meme subculture around hating you.

Meanwhile, Drew created and maintained wlroots and Sway, successfully appointing a successor maintainer to carry the torch, and runs a sustainable business on top of Free and Open Source Software. SourceHut has a dependency on Redict, so it naturally follows that Drew wants to keep his supply chain FOSS.

Redis is the Fork

The only thing Redis has going for it, as a software project, is the brand name. Salvatore is long gone. The active contributors who are working on it are, like I said, pivoting to AI. Seriously, here's a quote from The Future of Redis:

Making Redis the Go-To for Generative AI

we’re staying at the forefront of the GenAI wave

Meanwhile, Redict has an actual Free and Open Source Software movement behind it, spearheaded by Drew DeVault, who has a track record of effective open source project management.

In other words, Redict is the true spiritual successor to what was once Redis. The title of this blog post is not spicy or edgy; it reflects reality.

The problem with software is that it's too powerful. It creates so much wealth so fast that it's virtually impossible to not distribute it.

Think about it: sure, it takes a while to make useful software. But then you make it, and then it's done. It keeps working with no maintenance whatsoever, and just a trickle of electricity to run it.

Immediately, this poses a problem: how can a small number of people keep all that wealth for themselves, and not let it escape in the dirty, dirty fingers of the general populace?

This is a question that the music industry faced head-on, and they came up with EULAs, enforced via the state's monopoly on violence, and DRM, a way for software to act antagonistically against its own users. Software can do useful things like encode media into bits, and then copy those bits. That's dangerously useful, and it had to be stopped.

The True Cause of Bitrot

What about bitrot, you say? It takes ongoing maintenance to keep software working, doesn't it?

Let's think critically about bitrot for a moment because, as a reminder, bits don't actually rot - that's kinda the point of bits. In the best case scenario, bitrot happens due to progress - perhaps a dependency has made improvements but requires breaking API compatibility, or better hardware comes out and the software needs to be recompiled for that hardware. In this case, it's kind of a happy outcome. Some labor is needed to enhance the software in response, but then, once again, it's done; ripples disappearing from the surface of a lake hours after a stone is thrown into it.

The darker side of bitrot is due to businesses trying to make more profit than last year, and launching marketing initiatives. For example, Microsoft shipped a Windows Update that puts advertisements into the start menu, advertisements into the task bar, and changed the control panel's user interface to unify it with their business incentives - namely a superficial makeover to justify customers paying additional money for what is effectively worse software - it has new bugs and is now ridden with advertisements. This caused a bunch of churn in their own codebase, as well as other software trying to use native user interfaces on Windows.

It's all so incredibly wasteful. And that's the point, isn't it?

The programmers at Microsoft could have done less work, or worked on bug fixes instead. The UI designers could have done less work, or tweaked their existing design instead of making a new one. The managers could have done less work. Customers could have paid little to no additional money for a Windows Update. This all would have culminated in a more robust version of Windows that customers preferred, instead of one that is effectively boycotted like Vista and Windows 11.

It's actually a problem that software is too efficient and has this nasty tendency of being completed. Software offers us a glimpse into a post-scarcity society, but it is being actively sabotaged by those who seek to turn a profit.

Platform Waste

Consumers love standards. Standards allow multiple parties, perhaps even competitors, perhaps especially competitors, to have interchangeable components with each other, which gives consumers options, and negotiation power.

For-profit companies hate standards. They would rather have their own special cable, for example, that only works for their devices, and only they are allowed to manufacture them. To be more specific, underdog companies like standards because it lets them compete. The established players don't want to have to play fair.

You can see this playing out right now with the EU formally adopting a law requiring Apple to support USB-C chargers. At the time of writing there is no such law in the United States, but it is being discussed by politicians.

Standards allow software to be more efficient. By sticking with a standard for a period of time and then coordinating an upgrade to a newer one, software churn is minimized, resulting in a fixed amount of software development labor needed.

On the other hand, without a standard, for-profit companies are incentivized to fiddle with their product in a wasteful manner. For example, Apple has in the past made insignificant changes to their charging cable, making it not compatible with the one from the previous year. This resulted in more profit for Apple since consumers found their existing cables useless and had to buy new ones. Ultimately this resulted in more money being spent in the economy, increasing the country's GDP. Economists rejoice; the Earth weeps.

Think about how many messaging apps have come and go and how much programmer hours have been wasted on them. We almost had XMPP be mainstream, but then Google outgrew their "don't be evil" diaper and put on their "make profit" big boy pants. If your goal is to turn a profit, it's obviously the correct choice to invest into a platform that you own. So then we got a half-dozen buggy messaging apps from Google that didn't even work with each other, let alone Apple's platform or the other contemporary players.

The new hotness is Discord, which is already starting, predictably, to decay. I can't believe a human sat down and wasted hours of their life coding "super reactions". It's not something that really needed to happen.

Imagine if all these programmer hours spent on all these products actually centered around a proper standard, which evolved along with consumers' needs rather than these companies' ongoing need to fiddle with the knobs and sliders until profit comes out. The thing is, if this actually happened, then what would these employees spend their time on? At some point society would be pretty much done implementing messaging software. Messaging app updates would be rare, and bugs in messaging apps would be rare. We would reach peak messaging.

Peak Dishwasher

Decades ago, we already did it. We reached peak dishwasher. Dishwashers achieved perfection, and it was no longer possible to improve them. The mechanics were optimal, the user interface was ideal, and consumers had no desire for any changes.

One would, naively, think of this as an accomplishment. But how is a company supposed to make more profit than last year? By any means necessary, of course.

They invented these dishwasher detergent pods that are actually a downgrade - slightly more time consuming to use than powder, more expensive to manufacture and purchase, worse for the environment, and most offensive of all - actively sabotage the dishwasher's prewash feature making the product actually function worse than before!

And from a business perspective, it is a critical success. They found a way to make consumers spend more money on dishwashing. The line goes up, for one more year. But it's not enough. It has to go up every year. What else can we do?

I found myself in a position where I needed to buy a new dishwasher last month, and, already being aware of this problem, did my very best to buy one that worked well. I picked one that had 5/5 stars on Consumer Reports. Unfortunately, the dishwasher that I ended up with is my worst nightmare.

It takes 30 seconds to boot up, presumably because of the Bluetooth and WiFi driver in it. Many of the configuration options are hidden behind a proprietary app. The buttons are hidden and touch based instead of being visible and depressing with natural tactile feedback. I still haven't yet done the chore of going into my router and disabling it from accessing the Internet. I had to give it access to use the app to find out why it was broken. Until I do that chore, there's a chance it could auto update and have a firmware bug and stop working, or just waste my bandwidth. Who knows what it's up to?

Meanwhile, I had to call a repair technician to fix the door latch already, as well as the soap dispenser latch. Both things have since failed to work properly again and I still need to do the chore of calling the company to get a repair done a third time.

Before we moved, we had an older dishwasher that worked perfectly. No Internet, no Bluetooth, and the door latches worked flawlessly through thousands of runs.

The problem with the requirement for each year to be more profitable than the last is that once you reach the peak, once it's not possible to actually improve your product any more, you still have to change something. Since you can't change it to make it better, you therefore will change it to make it worse.

What Blockchains and LLMs Have in Common

Plenty of people roll their eyes at blockchain being the new buzzword, or about how tech is overobsessed with AI (LLMs) right now. It's easy to chock it up to it being a silly, harmless fad perpetuated by uneducated or misguided people, but in reality it's a lot more intentional than that.

Most tech workers work 40 hours per week at some company. That's a lot of collective hours spent on something. What factors go into deciding, as a whole, what that effort is spent on? Employees have some choices in the matter, but in the end those choices are limited to job offers. Job offers are created by the owners of companies who decide what they want to invest their money into.

In other words, venture capitalists decide what is the current hotness precisely by directing large amounts of labor towards whatever they want. Empirically, VCs are primarily motivated by seeking a return on investment. The goal is to turn a big sum of money into an even bigger sum of money. In theory, this is because with an even bigger sum of money, you can then start to spend that money on directing an even larger amount of labor towards whatever you want, bypassing democracy to influence the future of humanity, but in practice, most VCs get fixated on that return on investment until they die.

If you were to criticize blockchain technology from a purely technical perspective, you might point out the flaw that proof of work requires an exponentially increasing amount of computational power, and thus electricity, in order to keep the blockchain database alive over time. You might point out how inefficient of a database it is. But you would be missing the point. Blockchain technology excites investors precisely because of how wasteful it is. Even if we had fusion (!!) it would eat up all that energy and more. It's difficult to express the magnitude of how wasteful this is, and the fact that it's built into the system intentionally is sinister.

Blockchain technology is a kind of software that doesn't get completed or perfected. Rather it's the opposite; the longer it is in existence the more work it creates for everyone to do. The waste is a feature; it's how the line goes up.

Reminds me of this scene in The Fifth Element where Zorg says:

Life, which you so nobly serve, comes from destruction, disorder, and chaos. Now take this empty glass. Here it is: peaceful, serene, boring. But if it is destroyed... look at all these little things! So busy now. Notice how each one of them is useful. What a lovely ballet ensues, so full of form and color. Now, think about all those people that created them. Technicians, engineers, hundreds of people who will be able to feed their children tonight so those children can grow up big and strong and have little teeny children of their own and so on and so forth. Thus adding to the great chain of life.

You can find this scene on YouTube but I won't link it for fear of accidentally causing someone to view an advertisement.

LLMs offer an even more ideal kind of software to the investor. First of all they require an enormous amount of capital to train, and specialized hardware to run, making them suitable to offer as a service, where the amount of profit can be made to go up in a controlled manner. What a delicious idea.

More to my point, they offer a host of subjective, ill-defined tasks that are immune to being completed. They've managed to take something well-defined, well-scoped, and completable, and turn it into an untameable monster that will be sure to offer software churn for decades to come.

Have a peek at this blog post that is going around lately: The pain points of building a copilot

These people are brimming with excitement about all the new problems that LLMs are bringing to the table. Some choice quotes:

Prompt engineering is time consuming and requires considerable trial and error...As one developer said, "it's more of an art than a science".

Testing is fundamental to software development but arduous when LLMs are involved. Every test is a flaky test.

The field is moving fast, and it requires developers to "throw away everything that they've learned and rethink it."

Developers are having to learn and compare many new tools rather than focusing on the customer problem. They then have to glue these tools together.

It is still the wild, wild west ... It will be interesting to see how software engineering will evolve, either through new processes or tools, over the next several years.

LLMs are a way to make software take orders of magnitude more computational power, electricity, and human labor, while delivering a product whose extremely volatile quality is impossible to assure. The work will never be completed; it will only create the need for ever more labor.

For investors, all this churn is attractive. It's disruptive.

It's why we can't have nice software.

Conclusion

Technology, and in particular, software, offers a glimpse of magic; a perpetual motion machine; wealth created from nothing. It offers us a chance to work together on something beautiful; to achieve perfection by ratcheting improvements over time.

In the end, this opportunity is squandered in a doomed quest for endless growth.

Technology is a tool that enhances the power of the wielder.

Our tools are getting more and more powerful. Some tools, such as nuclear fission, are so powerful that they pose an existential threat to all of humanity, because human society has a history of wielding tools to maintain power over each other at any cost.

The dark desire to subjugate one another is the ultimate enemy of innovation, because the conquerors are content to have their needs met by human labor, while the conquered are too busy laboring to become scientists.

Human societies are systems. Systems are themselves tools. Free Market Capitalism is a system. It has tech specs just like any computer - features, limitations, and tradeoffs. We will not settle for a deprecated system when there is a new and improved one that can be created, discovered, and iterated upon.

Venture capital is an outdated tool. It leads to monopolies, anti-competitive practices, generational wealth, and environmental externalities. Non-donation-based non-profit organizations are an underexplored alternative, especially for tech companies where the number of users is large and size of the organization is relatively small.

Religion is an outdated tool, used to control, weaken, and exploit the masses. It causes societies to maintain unscientific practices, especially in healthcare and justice, and leads to war.

War is a means to destroy the wealth created by technology.

War turns technology against mankind.

War is opportunity cost for every scientist and engineer who works on national defense.

War makes it dangerous to research advanced technology.

As techno-optimists, we believe that we must, and we will, create more advanced human systems to prevent war forevermore. Human society will mature and overcome its barbaric history, learning to cooperate with each other so that we can create technological works of global magnitude, such as orbital rings.

We cannot meaningfully explore space while borders exist on Earth.

Every dollar spent on the military is a blight on the country's honor. We believe that military budget reductions are a cause for celebration because they symbolize progress towards the space age. We believe that the atomic bombings of Hiroshima and Nagasaki were the most horrifying, shameful thing ever done by humanity, to itself. Importantly, we believe that it will never, ever happen again.

We believe that there are irreversible destructive actions. While some amount of species extinction is a natural process, the outlier extinction event currently happening due to the emergence of homo sapiens is a tragedy. We believe that it is our duty to not only improve technology to enhance our own lives, but also enhance technology to preserve natural diversity. We believe that Free Market Capitalism is an outdated technology that has failed to prevent irreversible destruction to our home planet. However, we believe that human society will migrate to a better system without these flaws, and will prevent the ultimate irreversible action of self-destruction.

We believe in the simple, physical fact that endless growth is unsustainable, and we refuse to resolve this problem with war. Instead, we will improve efficiency and work within the limits of the system, whatever they may be. We will upgrade our economic system to one that is capable of supporting these requirements.

Therefore, finally, we believe that it is justified to spend one's life or career in pursuit of science, engineering, and technology, because we believe in humanity's ability to mature and improve to the point where it can wield ever more powerful tools responsibly.

It's been over three years since my last blog post. I think that the website code started to feel like it had bitrotted, and so making new blog posts became onerous. But I've taken the time to redo this website using Zig, so it's quite easy to add blog posts now, and plus I added dark mode.

Twitter

Over the past few years I amassed a healthy number of followers on Twitter by posting screenshots, demos, and small progress updates for the Zig programming language. It felt good to gain an audience and have my perspective and efforts weighed more heavily in the court of public opinion.

However, I always knew that it would need to end. It's not really possible to use Twitter without at least a little bit of doomscrolling, and that activity took a toll on me emotionally. I think that the core concept of a "tweet" is fundamentally unhealthy, because by design it promotes angry and extreme content. Nuance and subtlety is impossible to distinguish from dog whistling. Most users don't understand that by "dunking" on someone, they are actually promoting the content. In summary, it gave me a darker feeling about the world in general, and in exchange, did not offer much to enrich my life.

So, when the enshittification process kicked into high gear with the Acquisition of Twitter by Elon Musk, this was the perfect opportunity for me to make my exit.

It has now been a few months since I downloaded all my data and deleted my account, and in retrospect it was so, so worth it. I feel more optimistic about the world and human society in general. I interact with people around me with fewer preconceived judgements about their belief systems. And finally, I have more nuanced conversations with my wife and friends about politics and other hot topics, without worrying that I'm going to come off as dog whistling or naysaying.

One thing I do miss is to have the ability to correct misinformation that I stumble upon in the wild. For example, somebody linked me to this tweet:

I cannot stop thinking about this. Andrew created Zig and ~15 years ago asked on SO how to open a file in C++

(screenshot of this StackOverflow question)

if you take this in any other way than this is incredible and motivating that you can accomplish anything with time + effort then you nuts/sad

It's a sweet sentiment! Unfortunately it's based on a bit of a misunderstanding. If you look at the date the question was asked, it was August 11th, 2008. According to Wikipedia, StackOverflow launched on September 15th, 2008. How is this possible?

I'll tell you - if you look at the citation, it's a link to the launch announcement on Joel Spolsky's blog - a blog that I was a huge fan of when I was 20 years old. The fact is that Stack Overflow actually was annouced before that, on a podcast episode with Jeff Atwood.

Being a bored intern at Lockheed Martin, because they failed to find me anything to do besides "learn XML", I enthusiastically participated in the launch of Stack Overflow, including asking basic questions such as "How do you open a file in C++?" in order to farm karma points. And it definitely worked! I have edit powers on that website now, and you can bet your ass I use them to clean up misleading suggestions and stinky opinions whenever I see them.

Now, I don't want to poop on ThePrimeagen's parade, so I can find him some replacement lore. Here you go, enjoy! Binary files - Programmers Heaven

I don't have a clue about binary files. I'm a good programmer, have read more programming books than I can count, but still have no knowlege about binary files. Can you read them? Will they look like text? Is it good to use them? How can I print to a binary file, and then retreive from a binary file?

I was 14 years old when I wrote this cringefest. I think ThePrimeagen's take can survive transplanted from the Stack Overflow question to this forum post.

Reddit

Unlike Twitter, where I find the elemental unit of socializing problematic in and of itself, I think the Reddit formula is solid gold. This one is a case of malevolent platform stewards. I expect this from every platform run by for-profit entities, where the goal is not to serve the users, but to milk them for capital gains. The grim reaper of capitalism has come for Reddit.

Right now, venture capitalists are freaking out about "AI". Publicly-traded businesses with access to structured user speech that can be used as training data are rushing to build a moat around that data. That's why Twitter prevented viewing tweets from logged out users, and Reddit locked down its APIs despite it causing a massive protest from the moderators who do the actual labor to keep the website running.

Detecting that Reddit enshittification was reaching terminal levels, I decided that during these protests was the ideal time to close the /r/zig subreddit, which I was currently a moderator of. I closed the subreddit, downloaded all my user data, and permanently deleted my reddit account, leaving moderation to Loris Cro, VP of Community of Zig Software Foundation.

Once Loris was the sole moderator, he decided to change the subreddit to read-only mode. However, this was thwarted today when a troll (check their comment history) performed a hostile takeover in broad daylight and reopened the sub. It appears that Reddit admins are perfectly willing to forcibly replace moderators in an effort to keep the system churning out user data. Those Large Language Models are hungry!

Anyway, I don't consider this to be a problem; it is outside of my control what Reddit does with its own data. I just want to make it crystal clear that Zig, Zig Software Foundation, and myself are not in any way affiliated with /r/zig. That subreddit is now run by a third party and all signs point to a high likelihood of problematic behavior happening there which I definitely want to avoid being associated with.

Update (2023-08-30): As of today, the moderator of /r/zig is now Jens Goldberg, a well-regarded member of the Zig community.

If you want an alternative to a zig subreddit, give Ziggit a try. This is the Discourse forum software run by dude_the_builder, a friendly Zig community member, and I find it to be a healthy and rewarding way of socializing with other community members.

Moving Forward

I have a Mastodon account, but I don't love it, for the same reasons I didn't like Twitter. In addition, I think that way of consuming content is generally like watching mainstream TV or listening to radio with ads. You're letting a bunch of people who aren't really that important to you, or qualified to do the job, be the content curators for you.

Discord has been decent for a while. I suspect enshittification will commence soon, so we should be on the lookout for an alternative over the next five years.

I'm going to look into setting up an RSS reader for myself and start hunting for high quality blogs.

And finally, I will be redirecting my micro-blogging energy that was previously wasted on Twitter into actual-blogging energy here.

If you have heard of Zig before, you may know it as a promising new programming language which is ambitiously trying to overthrow C as the de-facto systems language. But did you know that it also can straight up compile C code?

This has been possible for a while, and you can see some examples of this on the home page. What's new is that the zig cc sub-command is available, and it supports the same options as Clang, which, in turn, supports the same options as GCC.

Now, I'm sure you're feeling pretty skeptical right about now, so let me hook you real quick before I get into the juicy details.

Clang and GCC cannot do this:

andy@ark ~/tmp> cat hello.c
#include <stdio.h>

int main(int argc, char **argv) {
    fprintf(stderr, "Hello, World!\n");
    return 0;
}
andy@ark ~/tmp> clang -o hello.exe hello.c -target x86_64-windows-gnu
clang-7: warning: argument unused during compilation: '--gcc-toolchain=/nix/store/ificps9si1nvz85f9xa7gjd9h6r5lzg6-gcc-9.2.0' [-Wunused-command-line-argument]
/nix/store/7bhi29ainf5rjrk7k7wyhndyskzyhsxh-binutils-2.31.1/bin/ld: unrecognised emulation mode: i386pep
Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu elf_l1om elf_k1om
clang-7: error: linker command failed with exit code 1 (use -v to see invocation)
andy@ark ~/tmp> clang -o hello hello.c -target mipsel-linux-musl
In file included from hello.c:1:
In file included from /nix/store/8pp3i3hcp7bv0f8jllzqq7gcp9dbzvp9-glibc-2.27-dev/include/stdio.h:27:
In file included from /nix/store/8pp3i3hcp7bv0f8jllzqq7gcp9dbzvp9-glibc-2.27-dev/include/bits/libc-header-start.h:33:
In file included from /nix/store/8pp3i3hcp7bv0f8jllzqq7gcp9dbzvp9-glibc-2.27-dev/include/features.h:452:
/nix/store/8pp3i3hcp7bv0f8jllzqq7gcp9dbzvp9-glibc-2.27-dev/include/gnu/stubs.h:7:11: fatal error: 
      'gnu/stubs-32.h' file not found
# include <gnu/stubs-32.h>
          ^~~~~~~~~~~~~~~~
1 error generated.
andy@ark ~/tmp> clang -o hello hello.c -target aarch64-linux-gnu
In file included from hello.c:1:
In file included from /nix/store/8pp3i3hcp7bv0f8jllzqq7gcp9dbzvp9-glibc-2.27-dev/include/stdio.h:27:
In file included from /nix/store/8pp3i3hcp7bv0f8jllzqq7gcp9dbzvp9-glibc-2.27-dev/include/bits/libc-header-start.h:33:
In file included from /nix/store/8pp3i3hcp7bv0f8jllzqq7gcp9dbzvp9-glibc-2.27-dev/include/features.h:452:
/nix/store/8pp3i3hcp7bv0f8jllzqq7gcp9dbzvp9-glibc-2.27-dev/include/gnu/stubs.h:7:11: fatal error: 
      'gnu/stubs-32.h' file not found
# include <gnu/stubs-32.h>
          ^~~~~~~~~~~~~~~~
1 error generated.

`zig cc` can:

andy@ark ~/tmp> zig cc -o hello.exe hello.c -target x86_64-windows-gnu
andy@ark ~/tmp> wine64 hello.exe
Hello, World!
andy@ark ~/tmp> zig cc -o hello hello.c -target mipsel-linux-musl
andy@ark ~/tmp> qemu-mipsel ./hello
Hello, World!
andy@ark ~/tmp> zig cc -o hello hello.c -target aarch64-linux-gnu
andy@ark ~/tmp> qemu-aarch64 -L ~/Downloads/glibc/multi-2.31/install/glibcs/aarch64-linux-gnu ./hello
Hello, World!

Features of `zig cc`

zig cc is not the main purpose of the Zig project. It merely exposes the already-existing capabilities of the Zig compiler via a small frontend layer that parses C compiler options.

Install simply by unzipping a tarball

Zig is an open source project, and of course can be built and installed from source the usual way. However, the Zig project also has tarballs available on the download page. You can download a 45 MiB tarball, unpack it, and you're done. You can even have multiple versions at the same time, no problem.

Here, rather than downloading the x86_64-linux version, which matches the computer I am currently using, I'll download the Windows version and run it in Wine to show how simple installation is:

andy@ark ~/tmp> wget --quiet https://ziglang.org/builds/zig-windows-x86_64-0.5.0+13d04f996.zip
andy@ark ~/tmp> unzip -q zig-windows-x86_64-0.5.0+13d04f996.zip 
andy@ark ~/tmp> wine64 ./zig-windows-x86_64-0.5.0+13d04f996/zig.exe cc -o hello hello.c -target x86_64-linux
andy@ark ~/tmp> ./hello
Hello, World!

Take a moment to appreciate what just happened here - I downloaded a Windows build of Zig, ran it in Wine, using it to cross compile for Linux, and then ran the binary natively. Computers are fun!

Compare this to downloading Clang, which has 380 MiB Linux-distribution-specific tarballs. Zig's Linux tarballs are fully statically linked, and therefore work correctly on all Linux distributions. The size difference here comes because the Clang tarball ships with more utilities than a C compiler, as well as pre-compiled static libraries for both LLVM and Clang. Zig does not ship with any pre-compiled libraries; instead it ships with source code, and builds what it needs on-the-fly.

Caching System

The Zig compiler uses a sophisticated caching system to avoid needlessly rebuilding artifacts. I carefully designed this caching system to make optimal use of the file system while maintaining correct semantics - which is trickier than you might think!

The caching system uses a combination of hashing inputs and checking the fstat values of file paths, while being mindful of mtime granularity. This makes it avoid needlessly hashing files, while at the same time detecting when a modified file has the same contents. It always has correct behavior, whether the file system has nanosecond mtime granularity, second granularity, always sets mtime to zero, or anything in between.

You can find a detailed description of the caching system in the 0.4.0 release notes.

zig cc makes this caching system available when compiling C code. For simple enough projects, this obviates the need for a Makefile or other build system.

andy@ark ~/tmp> cat foo.c
#include <stdio.h>

#include "another_file.c"

int main(int argc, char **argv) {
#include "printf_many_times.c"
}
andy@ark ~/tmp> cat another_file.c 
void another(void) {}
andy@ark ~/tmp> time zig cc -c foo.c
0.12
andy@ark ~/tmp> time zig cc -c foo.c
0.01
andy@ark ~/tmp> touch another_file.c 
andy@ark ~/tmp> time zig cc -c foo.c
0.01
andy@ark ~/tmp> echo "/* add a comment */" >>another_file.c
andy@ark ~/tmp> time zig cc -c foo.c
0.12
andy@ark ~/tmp> time zig cc -c foo.c
0.01

Here you can see the caching system is smart enough to find dependencies that are included with the preprocessor, and smart enough to avoid a full rebuild when the mtime of another_file.c was updated.

One last thing before I move on. I want to point out that this caching system is not some fluffy bloated feature - rather it is an absolutely critical component to making cross-compiling work in a usable manner. As we'll see below, other compilers ship with pre-compiled, target-specific binaries, while Zig ships with source code only and cross-compiles on-the-fly, caching the result.

Cross Compiling

I have carefully designed Zig since the very beginning to treat cross compilation as a first class use case. Now that the zig cc frontend is available, it brings these capabilities to C code.

I showed you above cross-compiling some simple "Hello, World!" programs. But now let's try a real-world C project.

Let's try LuaJIT!

[~/Downloads]$ git clone https://github.com/LuaJIT/LuaJIT
[~/Downloads]$ cd LuaJIT
[~/Downloads/LuaJIT]$ ls
COPYRIGHT  doc  dynasm  etc  Makefile  README  src

OK so it uses standard Makefiles. Here we go, first let's make sure it works natively with zig cc.

[~/Downloads/LuaJIT]$ export CC="zig cc"
[~/Downloads/LuaJIT]$ make CC="$CC"
==== Building LuaJIT 2.1.0-beta3 ====
make -C src
make[1]: Entering directory '/home/andy/Downloads/LuaJIT/src'
HOSTCC    host/minilua.o
HOSTLINK  host/minilua
DYNASM    host/buildvm_arch.h
HOSTCC    host/buildvm.o
HOSTCC    host/buildvm_asm.o
HOSTCC    host/buildvm_peobj.o
HOSTCC    host/buildvm_lib.o
HOSTCC    host/buildvm_fold.o
HOSTLINK  host/buildvm
BUILDVM   lj_vm.S
ASM       lj_vm.o
CC        lj_gc.o
BUILDVM   lj_ffdef.h
CC        lj_err.o
CC        lj_char.o
BUILDVM   lj_bcdef.h
CC        lj_bc.o
CC        lj_obj.o
CC        lj_buf.o
CC        lj_str.o
CC        lj_tab.o
CC        lj_func.o
CC        lj_udata.o
CC        lj_meta.o
CC        lj_debug.o
CC        lj_state.o
CC        lj_dispatch.o
CC        lj_vmevent.o
CC        lj_vmmath.o
CC        lj_strscan.o
CC        lj_strfmt.o
CC        lj_strfmt_num.o
CC        lj_api.o
CC        lj_profile.o
CC        lj_lex.o
CC        lj_parse.o
CC        lj_bcread.o
CC        lj_bcwrite.o
CC        lj_load.o
CC        lj_ir.o
CC        lj_opt_mem.o
BUILDVM   lj_folddef.h
CC        lj_opt_fold.o
CC        lj_opt_narrow.o
CC        lj_opt_dce.o
CC        lj_opt_loop.o
CC        lj_opt_split.o
CC        lj_opt_sink.o
CC        lj_mcode.o
CC        lj_snap.o
CC        lj_record.o
CC        lj_crecord.o
BUILDVM   lj_recdef.h
CC        lj_ffrecord.o
CC        lj_asm.o
CC        lj_trace.o
CC        lj_gdbjit.o
CC        lj_ctype.o
CC        lj_cdata.o
CC        lj_cconv.o
CC        lj_ccall.o
CC        lj_ccallback.o
CC        lj_carith.o
CC        lj_clib.o
CC        lj_cparse.o
CC        lj_lib.o
CC        lj_alloc.o
CC        lib_aux.o
BUILDVM   lj_libdef.h
CC        lib_base.o
CC        lib_math.o
CC        lib_bit.o
CC        lib_string.o
CC        lib_table.o
CC        lib_io.o
CC        lib_os.o
CC        lib_package.o
CC        lib_debug.o
CC        lib_jit.o
CC        lib_ffi.o
CC        lib_init.o
AR        libluajit.a
CC        luajit.o
BUILDVM   jit/vmdef.lua
DYNLINK   libluajit.so
LINK      luajit
warning: unsupported linker arg: -E
OK        Successfully built LuaJIT
make[1]: Leaving directory '/home/andy/Downloads/LuaJIT/src'
==== Successfully built LuaJIT 2.1.0-beta3 ====

[~/Downloads/LuaJIT]$ ls
COPYRIGHT  doc  dynasm  etc  Makefile  README  src

[~/Downloads/LuaJIT]$ ./src/
host/         jit/          libluajit.so  luajit        zig-cache/    

[~/Downloads/LuaJIT]$ ./src/luajit 
LuaJIT 2.1.0-beta3 -- Copyright (C) 2005-2020 Mike Pall. http://luajit.org/
JIT: ON SSE2 SSE3 SSE4.1 BMI2 fold cse dce fwd dse narrow loop abc sink fuse
> print(3 + 4)
7
> 

OK so that worked. Now for the real test - can we make it cross compile?

[~/Downloads/LuaJIT]$ git clean -xfdq
[~/Downloads/LuaJIT]$ export CC="zig cc -target aarch64-linux-gnu"
[~/Downloads/LuaJIT]$ export HOST_CC="zig cc"
[~/Downloads/LuaJIT]$ make CC="$CC" HOST_CC="$HOST_CC" TARGET_STRIP="echo"
==== Building LuaJIT 2.1.0-beta3 ====
make -C src
make[1]: Entering directory '/home/andy/Downloads/LuaJIT/src'
HOSTCC    host/minilua.o
HOSTLINK  host/minilua
DYNASM    host/buildvm_arch.h
HOSTCC    host/buildvm.o
HOSTCC    host/buildvm_asm.o
HOSTCC    host/buildvm_peobj.o
HOSTCC    host/buildvm_lib.o
HOSTCC    host/buildvm_fold.o
HOSTLINK  host/buildvm
BUILDVM   lj_vm.S
ASM       lj_vm.o
CC        lj_gc.o
BUILDVM   lj_ffdef.h
CC        lj_err.o
CC        lj_char.o
BUILDVM   lj_bcdef.h
CC        lj_bc.o
CC        lj_obj.o
CC        lj_buf.o
CC        lj_str.o
CC        lj_tab.o
CC        lj_func.o
CC        lj_udata.o
CC        lj_meta.o
CC        lj_debug.o
CC        lj_state.o
CC        lj_dispatch.o
CC        lj_vmevent.o
CC        lj_vmmath.o
CC        lj_strscan.o
CC        lj_strfmt.o
CC        lj_strfmt_num.o
CC        lj_api.o
CC        lj_profile.o
CC        lj_lex.o
CC        lj_parse.o
CC        lj_bcread.o
CC        lj_bcwrite.o
CC        lj_load.o
CC        lj_ir.o
CC        lj_opt_mem.o
BUILDVM   lj_folddef.h
CC        lj_opt_fold.o
CC        lj_opt_narrow.o
CC        lj_opt_dce.o
CC        lj_opt_loop.o
CC        lj_opt_split.o
CC        lj_opt_sink.o
CC        lj_mcode.o
CC        lj_snap.o
CC        lj_record.o
CC        lj_crecord.o
BUILDVM   lj_recdef.h
CC        lj_ffrecord.o
CC        lj_asm.o
CC        lj_trace.o
CC        lj_gdbjit.o
CC        lj_ctype.o
CC        lj_cdata.o
CC        lj_cconv.o
CC        lj_ccall.o
CC        lj_ccallback.o
CC        lj_carith.o
CC        lj_clib.o
CC        lj_cparse.o
CC        lj_lib.o
CC        lj_alloc.o
CC        lib_aux.o
BUILDVM   lj_libdef.h
CC        lib_base.o
CC        lib_math.o
CC        lib_bit.o
CC        lib_string.o
CC        lib_table.o
CC        lib_io.o
CC        lib_os.o
CC        lib_package.o
CC        lib_debug.o
CC        lib_jit.o
CC        lib_ffi.o
CC        lib_init.o
AR        libluajit.a
CC        luajit.o
BUILDVM   jit/vmdef.lua
DYNLINK   libluajit.so
libluajit.so
LINK      luajit
warning: unsupported linker arg: -E
luajit
OK        Successfully built LuaJIT
make[1]: Leaving directory '/home/andy/Downloads/LuaJIT/src'
==== Successfully built LuaJIT 2.1.0-beta3 ====

[~/Downloads/LuaJIT]$ file ./src/luajit 
./src/luajit: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 2.0.0, with debug_info, not stripped

It worked! Will it run in QEMU though?

[~/Downloads/LuaJIT]$ qemu-aarch64 -L ~/Downloads/glibc/multi-2.31/install/glibcs/aarch64-linux-gnu ./src/luajit
LuaJIT 2.1.0-beta3 -- Copyright (C) 2005-2020 Mike Pall. http://luajit.org/
JIT: ON fold cse dce fwd dse narrow loop abc sink fuse
> print(4 + 3)
7
> 

Amazing. QEMU never fails to impress me.

Before we move on, I want to show one more thing. You can see above, in order to run the foreign-architecture binary, I had to pass -L ~/Downloads/glibc/multi-2.31/install/glibcs/aarch64-linux-gnu. This is due to the binary being dynamically linked. You can confirm this with the output from file above where it says: dynamically linked, interpreter /lib/ld-linux-aarch64.so.1

Often, when cross-compiling, it is useful to make a static binary. In the case of Linux, for example, this will make the resulting binary able to run on any Linux distribution, rather than only ones with a hard-coded glibc dynamic linker path of /lib/ld-linux-aarch64.so.1.

We can accomplish this by targeting musl rather than glibc:

[~/Downloads/LuaJIT]$ git clean -qxfd
[~/Downloads/LuaJIT]$ export CC="zig cc -target aarch64-linux-musl"
[~/Downloads/LuaJIT]$ make CC="$CC" CXX="$CXX" HOST_CC="$HOST_CC" TARGET_STRIP="echo"
==== Building LuaJIT 2.1.0-beta3 ====
(same output)
==== Successfully built LuaJIT 2.1.0-beta3 ====
[~/Downloads/LuaJIT]$ file src/luajit
src/luajit: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, not stripped
[~/Downloads/LuaJIT]$ qemu-aarch64 ./src/luajit
LuaJIT 2.1.0-beta3 -- Copyright (C) 2005-2020 Mike Pall. http://luajit.org/
JIT: ON fold cse dce fwd dse narrow loop abc sink fuse
> print(11 + 22)
33

Here you can see the file command reported statically linked, and in the qemu command, the -L parameter was not needed.

Use Cases of `zig cc`

Alright, so I've given you a taste of what zig cc can do, but now I will list explicitly what I consider to be the use cases:

Experimentation

Sometimes you just want a tool that you can use to try out different things. It can quickly answer questions such as "What assembly does this code generate on MIPS vs ARM?". The widely popular Compiler Explorer serves this purpose.

zig cc provides a lightweight tool which can also answer questions such as, "What happens if I swap out glibc for musl?" and "How big is this executable when cross-compiled for Windows?". Here's me using Zig to quickly find out what the maximum UDP packet size is on Linux.

Since Zig is so easy to install - and it actually works everywhere without patches, even Linux distributions such as NixOS - it can often be a more convenient tool for running quick C test programs on your computer.

At the time of this writing, LLVM 10 was just released two hours ago. It will take days or weeks for it to become available in various system package managers. But you can already download a master branch build of Zig and play with the new features of Clang/LLVM 10. For example, improved RISC-V support!

andy@ark ~/tmp> zig cc -o hello hello.c -target riscv64-linux-musl
andy@ark ~/tmp> qemu-riscv64 ./hello
Hello, World!

Bundling a C compiler as part of a larger project

With Zig tarballs weighing in at under 45 MiB, zero system dependencies, no configuration, and MIT license, it makes for an ideal candidate when you need to bundle a C compiler along with another project.

For example, maybe you have a programming language that compiles to C. Zig is an obvious choice for what C compiler to ship with your language.

Or maybe you want to make a batteries-included IDE that ships with a compiler.

Lightweight alternative to a cross compilation environment

If you're trying to build something with a large dependency tree, you'll probably want to use a full cross compilation environment, such as mxe.cc or musl.cc.

But if you don't need such a sledgehammer, zig cc could be a useful alternative, especially if your goal is to compile for N different targets. Consider that musl.cc lists different tarballs for each architecture, each weighing in at roughly 85 MiB. Meanwhile Zig weighs in at 45 MiB and it supports all those architectures, plus glibc and Windows.

An alternative to installing MSVC on Windows

You could spend days - literally! - waiting for Microsoft Visual Studio to install, or you could install Zig and VS Code in a matter of minutes.

Under the Hood

If zig cc is built on top of Clang, why doesn't Clang just do this? What exactly is Zig doing on top of Clang to make this work?

The answer is, a lot, actually. I'll go over how it works here.

compiler-rt

compiler-rt is a library that provides "polyfill" implementations of language-supported features when the target does not have machine code instructions for it. For example, compiler-rt has the function __muldi3 to perform signed 64-bit integer multiplication on architectures that do not have a 64-bit wide integer multiplication instruction.

In the GNU world, compiler-rt is named libgcc.

Most C compilers ship with this library pre-built for the target. For example, on an Ubuntu (Bionic) system, with the build-essential package installed, you can find this at /lib/x86_64-linux-gnu/libgcc_s.so.1.

If you download clang+llvm-9.0.1-x86_64-linux-gnu-ubuntu-16.04.tar.xz and take a look around, clang actually does not even ship with compiler-rt. Instead, it relies on the system libgcc noted above. This is one reason that this tarball is Ubuntu-specific and does not work on other Linux distributions, FreeBSD's Linuxulator, or WSL, which have system files in different locations.

Zig's strategy with compiler-rt is that we have our own implementation of this library, written in Zig. Most of it is ported from LLVM's compiler-rt project, but we also have some of our own improvements on top of this.

Anyway, rather than depending on system compiler-rt being installed, or shipping a pre-compiled library, Zig ships its compiler-rt in source form, and lazily builds compiler-rt for the compilation target, and then caches the result using the caching system discussed above.

Zig's compiler-rt is not yet complete. However, completing it is a prerequisite for releasing Zig version 1.0.0.

libc

When C code calls printf, printf has to be implemented somewhere, and that somewhere is libc.

Some operating systems, such as FreeBSD and macOS, have a designated system libc, and it is the kernel syscall interface. On others, such as Windows and Linux, libc is optional, and therefore there are multiple options of which libc to use, if any.

As of the time of this writing, Zig can provide libcs for the following targets:

andy@ark ~> zig targets | jq .libc
[
  "aarch64_be-linux-gnu",
  "aarch64_be-linux-musl",
  "aarch64_be-windows-gnu",
  "aarch64-linux-gnu",
  "aarch64-linux-musl",
  "aarch64-windows-gnu",
  "armeb-linux-gnueabi",
  "armeb-linux-gnueabihf",
  "armeb-linux-musleabi",
  "armeb-linux-musleabihf",
  "armeb-windows-gnu",
  "arm-linux-gnueabi",
  "arm-linux-gnueabihf",
  "arm-linux-musleabi",
  "arm-linux-musleabihf",
  "arm-windows-gnu",
  "i386-linux-gnu",
  "i386-linux-musl",
  "i386-windows-gnu",
  "mips64el-linux-gnuabi64",
  "mips64el-linux-gnuabin32",
  "mips64el-linux-musl",
  "mips64-linux-gnuabi64",
  "mips64-linux-gnuabin32",
  "mips64-linux-musl",
  "mipsel-linux-gnu",
  "mipsel-linux-musl",
  "mips-linux-gnu",
  "mips-linux-musl",
  "powerpc64le-linux-gnu",
  "powerpc64le-linux-musl",
  "powerpc64-linux-gnu",
  "powerpc64-linux-musl",
  "powerpc-linux-gnu",
  "powerpc-linux-musl",
  "riscv64-linux-gnu",
  "riscv64-linux-musl",
  "s390x-linux-gnu",
  "s390x-linux-musl",
  "sparc-linux-gnu",
  "sparcv9-linux-gnu",
  "wasm32-freestanding-musl",
  "x86_64-linux-gnu",
  "x86_64-linux-gnux32",
  "x86_64-linux-musl",
  "x86_64-windows-gnu"
]

In order to provide libc on these targets, Zig ships with a subset of the source files for these projects:

• musl v1.2.0
• mingw-w64 v7.0.0
• glibc 2.31

For each libc, there is a process for upgrading to a new release. This process is a sort of pre-processing step. We still end up with source files, but we de-duplicate non-multi-arch source files into multi-arch source files.

glibc

glibc is the most involved. The first step is building glibc for every target that it supports, which takes upwards of 24 hours and 74 GiB of disk space.

From here, the process_headers tool inspects all the header files from all the targets, and identifies which files are the same across all targets, and which header files are target-specific. They are then sorted into the corresponding directories in Zig's source tree, in:

• lib/libc/include/generic-glibc/
• lib/libc/include/$ARCH-linux-$ABI/ (there are multiple of these directories)

Additionally, Linux header files are not included in glibc, and so the same process is applied to Linux header files, with the directories:

• lib/libc/include/any-linux-any/
• lib/libc/include/$ARCH-linux-any/

That takes care of the header files, but now we have the problem of dynamic linking against glibc, without touching any system files.

For this, we have the update_glibc tool. Given the path to the glibc source directory, it finds all the .abilist text files and uses them to produce 3 simple but crucial files:

• vers.txt - the list of all glibc versions.
• fns.txt - the list of all symbols that glibc provides, followed by the library it appears in (for example libm, libpthread, libc, librt).
• abi.txt - for each target, for each function, tells which versions of glibc, if any, it appears in.

Together, these files amount to only 192 KB (27 KB gzipped), and they allow Zig to target any version of glibc.

Yes, I did not make a typo there. Zig can target any of the 42 versions of glibc for any of the architectures listed above. I'll show you:

andy@ark ~/tmp> cat rand.zig 
const std = @import("std");

pub fn main() anyerror!void {
    var buf: [10]u8 = undefined;
    _ = std.c.getrandom(&buf, buf.len, 0);
    std.debug.warn("random bytes: {x}\n", .{buf});
}
andy@ark ~/tmp> zig build-exe rand.zig -lc -target native-native-gnu.2.25
andy@ark ~/tmp> ./rand
random bytes: e2059382afb599ea6d29
andy@ark ~/tmp> zig build-exe rand.zig -lc -target native-native-gnu.2.24
lld: error: undefined symbol: getrandom
>>> referenced by rand.zig:5 (/home/andy/tmp/rand.zig:5)
>>>               ./rand.o:(main.0)

Sure enough, if you look at the man page for getrandom, it says:

Support was added to glibc in version 2.25.

When no explicit glibc version is requested, and the target OS is the native (host) OS, Zig detects the native glibc version by inspecting the Zig executable's own dynamically linked libraries, looking for glibc, and checking the version. It turns out you can look for libc.so.6 and then readlink on that, and it will look something like libc-2.27.so. When this strategy does not work, Zig looks at /usr/bin/env, looking for the same thing. Since this file path is hard-coded into countless shebang lines, it's a pretty safe bet to find out the dynamic linker path and glibc version (if any) of the native system!

zig cc currently does not provide a way to choose a specific glibc version (because C compilers do not provide a way), and so Zig chooses the native version for compiling natively, and the default (2.17) for cross-compiling. However, I'm sure this problem can be solved, even when using zig cc. For example, maybe it could support an environment variable, or simply introduce an extra command line option that does not conflict with any Clang options.

When you request a certain version of glibc, Zig uses those text files noted above to create dummy .so files to link against, which contain exactly the correct set of symbols (with appropriate name mangling) based on the requested version. The symbols will be resolved at runtime, by the dynamic linker on the target platform.

In this way, most of libc in the glibc case resides on the target file system. But not all of it! There are still the "C runtime start files":

• Scrt1.o
• crti.o
• crtn.o

These are statically compiled into every binary that dynamically links glibc, and their ABI is therefore Very Very Stable.

And so, Zig bundles a small subset of glibc's source files needed to build these object files from source for every target. The total size of this comes out to 1.4 MiB (252 KB gzipped). I do think there is some room for improvement here, but I digress.

There are a couple of patches to this small subset of glibc source files, which simplify them to avoid including too many .h files, since the end result that we need is some bare bones object files, and not all of glibc.

And finally, we certainly do not ship the build system of glibc with Zig! I manually inspected, audited, and analyzed glibc's build system, and then by hand wrote code in the Zig compiler which hooks into Zig's caching system and performs a minimal build of only these start files, as needed.

musl

The process for preparing musl to ship with Zig is much simpler by comparison.

It still involves building musl for every target architecture that it supports, but in this case only the install-headers target has to be run, and it takes less than a minute, even to do it for all targets.

The same process_headers tool tool used for glibc headers is used on the musl headers:

• lib/libc/include/generic-musl/
• lib/libc/include/$ARCH-linux-$ABI/ (there are multiple of these directories)

Unlike glibc, musl supports building statically. Zig currently assumes a static libc when musl is chosen, and does not support dynamically linking against musl, although that could potentially be added in the future.

And so for musl, zig actually bundles most - but still not all - of musl's source files. Everything in arch, crt, compat, src, and include gets copied in.

Again much like glibc, I carefully studied musl's build system, and then hand-coded logic in the Zig compiler to build these source files. In musl's case it is simpler - just a bit of logic having to do with the file extension, and whether to override files with an architecture-specific file. The only file that needs to be patched (by hand) is version.h, which is normally generated during the configure phase in musl's build system.

I really appreciate Rich Felker's efforts to make musl simple to utilize in this way, and he has been incredibly helpful in the #musl IRC channel when I ask questions. I proudly sponsor Rich Felker for $150/month.

mingw-w64

mingw-w64 was an absolute joy to support in Zig. The beautiful thing about this project is that they have already been transitioning into having one set of header files that applies to all architectures (using #ifdefs only where needed). One set of header files is sufficient to support all four architectures: arm, aarch64, x86, and x86_64.

So for updating headers, all we have to do is build mingw-w64, then:

mv $INSTALLPREFIX/include $ZIGSRC/lib/libc/include/any-windows-any

After doing this for all 3 libcs, the libc/include directory looks like this:

aarch64_be-linux-any   i386-linux-musl           powerpc-linux-any
aarch64_be-linux-gnu   mips64el-linux-any        powerpc-linux-gnu
aarch64-linux-any      mips64el-linux-gnuabi64   powerpc-linux-musl
aarch64-linux-gnu      mips64el-linux-gnuabin32  riscv32-linux-any
aarch64-linux-musl     mips64-linux-any          riscv64-linux-any
any-linux-any          mips64-linux-gnuabi64     riscv64-linux-gnu
any-windows-any        mips64-linux-gnuabin32    riscv64-linux-musl
armeb-linux-any        mips64-linux-musl         s390x-linux-any
armeb-linux-gnueabi    mipsel-linux-any          s390x-linux-gnu
armeb-linux-gnueabihf  mipsel-linux-gnu          s390x-linux-musl
arm-linux-any          mips-linux-any            sparc-linux-gnu
arm-linux-gnueabi      mips-linux-gnu            sparcv9-linux-gnu
arm-linux-gnueabihf    mips-linux-musl           x86_64-linux-any
arm-linux-musl         powerpc64le-linux-any     x86_64-linux-gnu
generic-glibc          powerpc64le-linux-gnu     x86_64-linux-gnux32
generic-musl           powerpc64-linux-any       x86_64-linux-musl
i386-linux-any         powerpc64-linux-gnu
i386-linux-gnu         powerpc64-linux-musl

When Zig generates a C command line to send to clang, it puts the appropriate include paths using -I depending on the target. For example, if the target is aarch64-linux-musl, then the following command line parameters are appended:

• -I$LIB/libc/include/aarch64-linux-musl
• -I$LIB/libc/include/aarch64-linux-any
• -I$LIB/libc/include/generic-musl

Anyway back to mingw-w64.

Again, Zig includes a subset of source files from mingw-w64 with a few patches applied to make things compile successfully.

The Zig compiler code that builds mingw-w64 from source files emulates only the parts of the build system that are needed for this subset. This includes preprocessing .def.in files to get .def files, and then in-turn using LLD to generate .lib files from the .def files, which allows Zig to provide .lib files for any Windows DLL, such as kernel32.dll or even opengl32.dll.

Invoking Clang Without a System Dependency

Since Zig already links against Clang libraries for the translate-c feature, it was not much more cost to expose the main() entry point from Zig. So that's exactly what we do:

• llvm-project/clang/tools/driver/driver.cpp is copied to $ZIGGIT/src/zig_clang_driver.cpp
• llvm-project/clang/tools/driver/cc1_main.cpp is copied to $ZIGGIT/src/zig_clang_cc1_main.cpp
• llvm-project/clang/tools/driver/cc1as_main.cpp is copied to $ZIGGIT/src/zig_clang_cc1as_main.cpp

The following patch is applied:

--- a/src/zig_clang_driver.cpp
+++ b/src/zig_clang_driver.cpp
@@ -206,8 +205,6 @@
                     void *MainAddr);
 extern int cc1as_main(ArrayRef<const char *> Argv, const char *Argv0,
                       void *MainAddr);
-extern int cc1gen_reproducer_main(ArrayRef<const char *> Argv,
-                                  const char *Argv0, void *MainAddr);
 
 static void insertTargetAndModeArgs(const ParsedClangName &NameParts,
                                     SmallVectorImpl<const char *> &ArgVector,
@@ -330,19 +327,18 @@
   if (Tool == "-cc1as")
     return cc1as_main(makeArrayRef(ArgV).slice(2), ArgV[0],
                       GetExecutablePathVP);
-  if (Tool == "-cc1gen-reproducer")
-    return cc1gen_reproducer_main(makeArrayRef(ArgV).slice(2), ArgV[0],
-                                  GetExecutablePathVP);
   // Reject unknown tools.
   llvm::errs() << "error: unknown integrated tool '" << Tool << "'. "
                << "Valid tools include '-cc1' and '-cc1as'.\n";
   return 1;
 }
 
-int main(int argc_, const char **argv_) {
+extern "C" int ZigClang_main(int argc_, const char **argv_);
+int ZigClang_main(int argc_, const char **argv_) {
   noteBottomOfStack();
   llvm::InitLLVM X(argc_, argv_);
-  SmallVector<const char *, 256> argv(argv_, argv_ + argc_);
+  size_t argv_offset = (strcmp(argv_[1], "-cc1") == 0 || strcmp(argv_[1], "-cc1as") == 0) ? 0 : 1;
+  SmallVector<const char *, 256> argv(argv_ + argv_offset, argv_ + argc_);
 
   if (llvm::sys::Process::FixupStandardFileDescriptors())
     return 1;

This disables some cruft, and then renames main to ZigClang_main so that it can be called like any other function. Next, in Zig's actual main, it looks for clang as the first parameter, and calls it.

So, zig clang is low-level undocumented API that Zig exposes for directly invoking Clang. But zig cc is much higher level than that. When Zig needs to compile C code, it invokes itself as a child process, taking advantage of zig clang. zig cc on the other hand, has a more difficult job: it must parse Clang's command line options and map those to the Zig compiler's settings, so that ultimately zig clang can be invoked as a child process.

Parsing Clang Command Line Options

When using zig cc, Zig acts as a proxy between the user and Clang. It does not need to understand all the parameters, but it does need to understand some of them, such as the target. This means that Zig must understand when a C command line parameter expects to "consume" the next parameter on the command line.

For example, -z -target would mean to pass -target to the linker, whereas -E -target would mean that the next parameter specifies the target.

Clang has a long list of command line options and so it would be foolish to try to hard-code all of them.

Fortunately, LLVM has a file "options.td" which describes all of its command line parameter options in some obscure format. But fortunately again, LLVM comes with the llvm-tblgen tool that can dump it as JSON format.

Zig has an update_clang_options tool which processes this JSON dump and produces a big sorted list of Clang's command line options.

Combined with a list of "known options" which correspond to Zig compiler options, this is used to make an iterator API that zig cc uses to parse command line parameters and instantiate a Zig compiler instance. Any Clang options that Zig is not aware of are forwarded to Clang directly. Some parameters are handled specially.

Linking

This part is pretty straightforward. Zig depends on LLD for linking rather than shelling out to the system linker, like GCC and Clang do.

When you use -o with zig cc, Clang is not actually acting as a linker driver here. Zig is still the linker driver.

Everybody Wins

Now that I've spent this entire blog article comparing Zig and Clang as if they are competitors, let me make it absolutely clear that both of these are harmonious, mutually beneficial open-source projects. It's pretty obvious how Clang and the entire LLVM project are massively beneficial to the Zig project, since Zig builds on top of them.

But it works the other way, too.

With Zig's focus on cross-compiling, its test suite has been expanding rapidly to cover a large number of architectures and operating systems, leading to dozens of bugs reported upstream and patches sent, including, for example:

• Regression discovered in LLVM 9 release candidate
• Bug fixed in Wine's NtDll
• Directly working with RISC-V target developers
• Bug fixes in LLVM's MIPS code generation

Everybody wins.

This is still experimental!

I have only recently landed zig cc support last week, and it is still experimental. Please do not expect it to be production quality yet.

Zig's 0.6.0 release is right around the corner, scheduled for April 13th. I will be sure to provide an update on the release notes on how stable and robust you can expect zig cc to be in the 0.6.0 release.

There are some follow-up issues related to zig cc which are still open:

• improve zig cc flag integration
• using zig as a drop in replacement for msvc
• support compiling and linking c++ code
• use case: directly symlink zig binary to /usr/bin/cc

💖 Sponsor Zig 💖

Sponsor Andrew Kelley on GitHub

If you're reading this and you already sponsor me, thank you so much! I wake up every day absolutely thrilled that I get to do this for my full time job.

As Zig has been gaining popularity, demands for my time have been growing faster than funds to hire another full-time programmer. Every recurring donation helps, and if the funds keep growing then soon enough the Zig project will have two full-time programmers.

That's all folks. I hope you and your loved ones are well.

Why I'm donating $150/month (10% of my income) to the musl libc project

One year ago, I quit my day job to work on Zig full time. Since then, the project has seen a steady growth in funding. Thanks to the people donating, Zig is on track to become fully sustainable before my savings run out.

This support has allowed me to focus on steady improvements to the language and tooling. In the last year, Zig has released two versions:

• 0.3.0 (release notes)
• 0.4.0 (release notes)

These release notes serve as my accountability to people who donate, and if you have a look at them, I hope you can agree with me that they speak for themselves.

The V language drama

I know that open-source project funding has been on people's minds lately, as we've watched the Internet's reaction to the bizarre open-sourcing of the V language unfold. As to how this relates to musl libc, stick with me - let me give you the talking points of this situation:

1. The V language website was published several months ago, claiming that it could already do incredible things, such as automatically translate C++ code to V code, and compile 1.2 million lines of code per second per CPU core by directly outputting x64 machine code. These features already work today, it said, and they will be released soon. Please donate. Meanwhile, the project was closed-source. Binaries and release notes were behind a paywall.
2. Over the next few months, people saw these amazing things it was claimed that V could do, and started donating money. It even looked like the language was already available, with download links including file sizes and file names, but if you clicked the download link, it did an alert() pop-up box saying "June 20". The project climbed up to $927/month in donations.
3. On 2019-06-20, the V author released a macOS binary of the V language, and it immediately became apparent that the many fantastic claims on the V website were unsubstantiated. People were quick to point this out, including myself.
4. In response to the backlash, the V author redacted the macOS binary, and said that he would do a full source release in 2 days.
5. On 2019-06-22, he updated the V language website with "WIP" labels to point out which features of the project are not available, and within an hour after that, published the V project source code. At this point, I made the following statement:

   Now that the website has the "WIP" label to communicate which features are not available, and the source is released, I no longer consider this project to be fraudulent. The information is available to everyone, and people who donate on Patreon are making an informed choice.

   I'm genuinely glad it turned out this way. Good luck on your endeavor, and welcome to the programming languages club.

6. At this point, the Internet discovered the source release. Shitposters, trolls, and genuine people all started paying attention to the project at once. Most people were unaware that the V website had been modified to retract the false claims, which is quite understandable given that they had been there just yesterday. Now that it was crystal clear that the V project had been intentionally misleading, the backlash intensified. In 2 days, V lost $100/month in donations.
7. Finally, there was a backlash on the backlash. To some people, $927/month was "table scraps". From this perspective, people were piling on a hate-train over a paltry sum of cash. Maybe other open-source projects should take a hint from V and do a little bit more marketing. Donations to open-source projects are not a zero sum game.

One thing that is crystal clear is that the V author succeeded in creating hype. He got people excited about ambitious features - so excited that they were willing to donate some cash. This made me think of another open source project in the opposite category.

musl libc: a project with no hype but huge impact

musl libc is an alternative to GNU libc for Linux, created by Rich Felker, and with a healthy community of high-quality contributors. It's been around for years, yet making less than V in donations.

The Zig project owes a lot to musl, for many reasons:

Mentorship

Especially in the early stages of the Zig project, but still to this day, I asked question after question in the #musl IRC channel, and the musl community patiently and expertly indulged them. Sometimes my questions were not even musl-related, but just more about how the Linux kernel works.

I didn't start off as an expert systems programmer when I started Zig, but the musl community has mentored me over the years.

Zig bundles musl

Zig ships with musl source code. This allows Zig projects to cross compile for Linux. For example, on Windows, you can:

> zig.exe build-exe --c-source hello.c -target x86_64-linux-musl

Copy the resulting ELF binary to a Linux machine and it will run. (Or run that in the Windows Subsystem for Linux).

This works because Zig lazily builds musl from source on demand, for the selected target. It is in no small part thanks to musl's simplicity and well-designed nature that this is possible.

It's also useful to create static builds on Linux where the native libc is glibc.

Much of the Zig standard library is ported from musl

musl is such a high quality codebase, that most of the Zig standard library's interface to Linux is a direct port of musl code.

This has prevented countless bugs and made things "just work" in general. Without this head start, the Zig project would have had to spend more time on Linux system interface and less on everything else.

For example, thanks to Marc Tiehuis contributions, the Zig standard library has all the math functions you would expect to find in libm, and they are available at compile-time as well as runtime.

Float literal parsing adapted and ported from musl

Zig has 128-bit floating point literals, so that compile-time computations can be done in a higher level of precision, before being casted to the floating point type of choice.

This is tricky to implement in the C++ non-self-hosted compiler, however, because libc does not have a 128-bit strtod function. So I took musl's strtold function, which works with the type long double, and then ported the code into Zig, making all the #ifdefs assume that long double is 128-bits (which is only true on some architectures), and replacing all the math with SoftFloat so that it would work on any architecture, no matter what long double maps to.

Alpine Linux is based on musl

Many people are aware of Alpine Linux because it has become a popular Linux distribution to use with Docker, mainly due to its simplicity and small binary size. This is in large part thanks to the fact that the system uses musl as its libc rather than glibc.

The Zig project uses Alpine Linux to create the static Linux builds on ziglang.org/download.

And so I have decided to donate $150/month to the musl project, even though that represents 10% of my income. I'm putting my money where my mouth is. But there's more - please read on...

This is a marketing stunt

Now if I'm being honest about my motivations for this blog post, it's that I want to prove that open source funding is not a zero-sum game. If there's anything we've learned from the V language Internet drama that has unfolded over the past few days, it's that open source projects have to do marketing if they want to get financial support.

But I want to set a better example of what that might look like. I don't think you have to trick people. This blog post is a marketing stunt intended to show that you can do things to get attention and funding, without being dishonest.

Will you help me prove my point?

If enough people start pledging donations to Zig when reading this, and the $150/month is recovered, then this little stunt will have been a proof-of-concept of one open source project running a fund-raiser for another one. (To be clear my donations to musl are not conditional; regardless of the outcome I will continue to donate.)

Worth a shot, right?

Since making news headlines, several generous people have taken me up on this experiment:

• Marko Mikulicic - $4
• Tristan Hume - $5
• Christine Dodrill - $15
• komu - $1
• Dylan Baker - $3
• Dave Voutila - $6.25
• Tyler Philbrick - $3
• Ben - $1
• Tanner Schultz - $3
• Martin DeMello - $5
• Thomas Ballinger - $2
• Steven Branson - $5
• Zachary Feldman - $10
• Dan Gallagher - $10
• Quetzal Bradley - $10
• Brian Orr - $5
• YVT - $5
• Dexter Haslem - $5
• Wes - $2.40
• Aeron Avery - $10
• Carsten Dreesbach - $5
• Ville Tuulos - $5
• Curtis Fenner - $3
• Nathan Sculli - $5
• Alec Nunn - $5
• Gurpreet Singh - $5
• Haze Booth - $20
• Don Harris - $20

That's a total of 178.65, well beyond the $150 that I started donating to musl.

Thank you so much! I'm really grateful for all your support. This experiment was a success! Today, the Zig project ran a fundraiser for the musl libc project, and it worked!

Using Zig to Provide Stack Traces on Kernel Panic for a Bare Bones Operating System

Last week, I reached an exciting milestone in my career as a programmer. For the first time in my life, I ran code directly on hardware, with no Operating System sitting between the bare metal and my code.

For some context, the goal of this side project is to create a 2-4 player arcade game running directly on a Raspberry Pi 3+.

Having just gotten Hello World working, this software can do little more than send a message over the serial uART on bootup:

Hello World! ClashOS 0.0

So what's the next step? Bootloader? File system? Graphics?!

Well, when coding all of these things, I'm inevitably going to run into bugs and crashes. And when this happens, I want to quickly understand what's gone wrong. Normally, in Zig, when something goes wrong, you get a nice stack trace, like this:

This example, however, is targeting Linux, whereas this arcade game project is freestanding. The equivalent code in freestanding actually just hangs the CPU:

--- a/src/main.zig
+++ b/src/main.zig
     serial.log("Hello World! ClashOS 0.0\n");
+    var x: u8 = 255;
+    x += 1;
+    serial.log("got here\n");

When run, you'll see we never get to the "got here" message, but also we don't get any kind of error message printed or anything.

Hello World! ClashOS 0.0

To understand this, we can look at the default panic handler. In Zig, you can provide a pub fn panic in your root source file alongside pub fn main. But if you do not provide this function, the default is used:

pub fn panic(msg: []const u8, error_return_trace: ?*builtin.StackTrace) noreturn {
    @setCold(true);
    switch (builtin.os) {
        builtin.Os.freestanding => {
            while (true) {}
        },
        else => {
            const first_trace_addr = @ptrToInt(@returnAddress());
            std.debug.panicExtra(error_return_trace, first_trace_addr, "{}", msg);
        },
    }
}

Here we can see why the code earlier is hanging - the default panic handler for the freestanding target is simply while(true) {}.

So we can make an immediate improvement by creating our own panic handler:

pub fn panic(message: []const u8, stack_trace: ?*builtin.StackTrace) noreturn {
    serial.write("\n!KERNEL PANIC!\n");
    serial.write(message);
    serial.write("\n");
    while(true) {}
}

And now, the output of booting the Raspberry Pi:

Hello World! ClashOS 0.0

!KERNEL PANIC!
integer overflow

Already this is much better. We can see that an integer overflow caused a kernel panic. But an integer overflow could occur anywhere. Wouldn't it be nice to have a full stack trace printed?

Yes, yes it would. The first thing I needed to make this work is access to the DWARF debugging info from inside the kernel. But I don't even have a file system. How can that work?

Easy! Just put the DWARF info directly into the kernel's memory. I modified my linker script to do just that:

    .rodata : ALIGN(4K) {
        *(.rodata)
        __debug_info_start = .;
        KEEP(*(.debug_info))
        __debug_info_end = .;
        __debug_abbrev_start = .;
        KEEP(*(.debug_abbrev))
        __debug_abbrev_end = .;
        __debug_str_start = .;
        KEEP(*(.debug_str))
        __debug_str_end = .;
        __debug_line_start = .;
        KEEP(*(.debug_line))
        __debug_line_end = .;
        __debug_ranges_start = .;
        KEEP(*(.debug_ranges))
        __debug_ranges_end = .;
    }

And to my dismay, these error messages stared back at me:

lld: error: incompatible section flags for .rodata
>>> /home/andy/dev/clashos/zig-cache/clashos.o:(.debug_info): 0x0
>>> output section .rodata: 0x12

lld: error: incompatible section flags for .rodata
>>> <internal>:(.debug_str): 0x30
>>> output section .rodata: 0x12

lld: error: incompatible section flags for .rodata
>>> /home/andy/dev/clashos/zig-cache/clashos.o:(.debug_line): 0x0
>>> output section .rodata: 0x32

After a back and forth on a bug report, George Rimar suggested simply deleting that particular check, as it might have been an overly strict enforcement. I tried this in my LLD fork, and it worked! Debug information was now linked into my kernel images. After completing the rest of the steps in this blog post, I submitted a patch upstream, which Rui has already merged into LLD. This will be released with LLVM 8, and in the meantime Zig's LLD fork has the patch.

At this point it was a simple matter of writing the glue code between my kernel and the Zig Standard Library's stack trace facilities. In Zig, you don't have to intentionally support freestanding mode. Code which has no dependencies on a particular operating system will work in freestanding mode thanks to Zig's lazy top level declaration analysis. Because the standard library stack trace code does not call any OS API, it therefore supports freestanding mode.

The Zig std lib API for opening debug information from an ELF file looks like this:

pub fn openElfDebugInfo(
    allocator: *mem.Allocator,
    elf_seekable_stream: *DwarfSeekableStream,
    elf_in_stream: *DwarfInStream,
) !DwarfInfo

But this kernel is so bare bones, it's not even in an ELF file. It's booting directly from a binary blob. We just have the debug info sections mapped directly into memory. For that we can use a lower level API:

/// Initialize DWARF info. The caller has the responsibility to initialize most
/// the DwarfInfo fields before calling. These fields can be left undefined:
/// * abbrev_table_list
/// * compile_unit_list
pub fn openDwarfDebugInfo(di: *DwarfInfo, allocator: *mem.Allocator) !void

And DwarfInfo is defined like this:

pub const DwarfInfo = struct {
    dwarf_seekable_stream: *DwarfSeekableStream,
    dwarf_in_stream: *DwarfInStream,
    endian: builtin.Endian,
    debug_info: Section,
    debug_abbrev: Section,
    debug_str: Section,
    debug_line: Section,
    debug_ranges: ?Section,
    abbrev_table_list: ArrayList(AbbrevTableHeader),
    compile_unit_list: ArrayList(CompileUnit),
};

To hook these up, the glue code needs to initialize the fields of DwarfInfo with the offsets into a std.io.SeekableStream, which can be implemented as a simple pointer to memory. By declaring external variables and then looking at their addresses, we can find out where in memory the symbols defined in the linker script are.

For .debug_abbrev, .debug_str, and .debug_ranges, I had to set the offset to 0 to workaround LLD thinking that the sections start at 0 for some reason.

var kernel_panic_allocator_bytes: [100 * 1024]u8 = undefined;
var kernel_panic_allocator_state = std.heap.FixedBufferAllocator.init(kernel_panic_allocator_bytes[0..]);
const kernel_panic_allocator = &kernel_panic_allocator_state.allocator;

extern var __debug_info_start: u8;
extern var __debug_info_end: u8;
extern var __debug_abbrev_start: u8;
extern var __debug_abbrev_end: u8;
extern var __debug_str_start: u8;
extern var __debug_str_end: u8;
extern var __debug_line_start: u8;
extern var __debug_line_end: u8;
extern var __debug_ranges_start: u8;
extern var __debug_ranges_end: u8;

fn dwarfSectionFromSymbolAbs(start: *u8, end: *u8) std.debug.DwarfInfo.Section {
    return std.debug.DwarfInfo.Section{
        .offset = 0,
        .size = @ptrToInt(end) - @ptrToInt(start),
    };
}

fn dwarfSectionFromSymbol(start: *u8, end: *u8) std.debug.DwarfInfo.Section {
    return std.debug.DwarfInfo.Section{
        .offset = @ptrToInt(start),
        .size = @ptrToInt(end) - @ptrToInt(start),
    };
}

fn getSelfDebugInfo() !*std.debug.DwarfInfo {
    const S = struct {
        var have_self_debug_info = false;
        var self_debug_info: std.debug.DwarfInfo = undefined;

        var in_stream_state = std.io.InStream(anyerror){ .readFn = readFn };
        var in_stream_pos: usize = 0;
        const in_stream = &in_stream_state;

        fn readFn(self: *std.io.InStream(anyerror), buffer: []u8) anyerror!usize {
            const ptr = @intToPtr([*]const u8, in_stream_pos);
            @memcpy(buffer.ptr, ptr, buffer.len);
            in_stream_pos += buffer.len;
            return buffer.len;
        }

        const SeekableStream = std.io.SeekableStream(anyerror, anyerror);
        var seekable_stream_state = SeekableStream{
            .seekToFn = seekToFn,
            .seekForwardFn = seekForwardFn,

            .getPosFn = getPosFn,
            .getEndPosFn = getEndPosFn,
        };
        const seekable_stream = &seekable_stream_state;

        fn seekToFn(self: *SeekableStream, pos: usize) anyerror!void {
            in_stream_pos = pos;
        }
        fn seekForwardFn(self: *SeekableStream, pos: isize) anyerror!void {
            in_stream_pos = @bitCast(usize, @bitCast(isize, in_stream_pos) +% pos);
        }
        fn getPosFn(self: *SeekableStream) anyerror!usize {
            return in_stream_pos;
        }
        fn getEndPosFn(self: *SeekableStream) anyerror!usize {
            return @ptrToInt(&__debug_ranges_end);
        }
    };
    if (S.have_self_debug_info) return &S.self_debug_info;

    S.self_debug_info = std.debug.DwarfInfo{
        .dwarf_seekable_stream = S.seekable_stream,
        .dwarf_in_stream = S.in_stream,
        .endian = builtin.Endian.Little,
        .debug_info = dwarfSectionFromSymbol(&__debug_info_start, &__debug_info_end),
        .debug_abbrev = dwarfSectionFromSymbolAbs(&__debug_abbrev_start, &__debug_abbrev_end),
        .debug_str = dwarfSectionFromSymbolAbs(&__debug_str_start, &__debug_str_end),
        .debug_line = dwarfSectionFromSymbol(&__debug_line_start, &__debug_line_end),
        .debug_ranges = dwarfSectionFromSymbolAbs(&__debug_ranges_start, &__debug_ranges_end),
        .abbrev_table_list = undefined,
        .compile_unit_list = undefined,
    };
    try std.debug.openDwarfDebugInfo(&S.self_debug_info, kernel_panic_allocator);
    return &S.self_debug_info;
}

You can see that the Zig common practice of accepting an allocator as an argument when allocation is needed comes in extremely handy for kernel development. We simply statically allocate a 100 KiB buffer and pass that along as the debug info allocator. If this ever becomes too small, it can be adjusted.

And now it's just a matter of wiring up the panic function:

pub fn panic(message: []const u8, stack_trace: ?*builtin.StackTrace) noreturn {
    serial.log("\n!KERNEL PANIC! {}\n", message);
    const dwarf_info = getSelfDebugInfo() catch |err| {
        serial.log("unable to get debug info: {}\n", @errorName(err));
        hang();
    };
    const first_trace_addr = @ptrToInt(@returnAddress());
    var it = std.debug.StackIterator.init(first_trace_addr);
    while (it.next()) |return_address| {
        std.debug.printSourceAtAddressDwarf(
            dwarf_info,
            serial_out_stream,
            return_address,
            true, // tty color on
            printLineFromFile,
        ) catch |err| {
            serial.log("missed a stack frame: {}\n", @errorName(err));
            continue;
        };
    }
    hang();
}

fn hang() noreturn {
    while (true) {}
}

fn printLineFromFile(out_stream: var, line_info: std.debug.LineInfo) anyerror!void {
    serial.log("TODO print line from the file\n");
}

Here it is in action:

Hello World! ClashOS 0.0

!KERNEL PANIC! integer overflow
/home/andy/dev/clashos/src/main.zig:166:7: 0x15e0 in ??? (clashos)
TODO print line from the file
      ^
???:?:?: 0x1c in ??? (???)

Great progress.

That last line is coming from the startup assembly code, which has no source mapping. But we can see that it's correct, by looking at the disassembly of the kernel:

0000000000000000 <_start>:
       0:	d53800a0 	mrs	x0, mpidr_el1
       4:	d2b82001 	mov	x1, #0xc1000000
       8:	8a210000 	bic	x0, x0, x1
       c:	b4000040 	cbz	x0, 14 <master>
      10:	14000003 	b	1c <__hang>

0000000000000014 <master>:
      14:	b26503ff 	mov	sp, #0x8000000
      18:	9400054b 	bl	1544 <kernel_main>

000000000000001c <__hang>:
      1c:	d503205f 	wfe
      20:	17ffffff 	b	1c <__hang>

You can see that 0x1c is indeed the return address (the next instruction that will be executed when the function returns) of the function call to kernel_main.

Let's shuffle the code around into more files and make that trace longer.

Hello World! ClashOS 0.0

!KERNEL PANIC!
integer overflow
/home/andy/dev/clashos/src/serial.zig:42:7: 0x1b10 in ??? (clashos)
TODO print line from the file
      ^
/home/andy/dev/clashos/src/main.zig:58:16: 0x1110 in ??? (clashos)
TODO print line from the file
               ^
/home/andy/dev/clashos/src/main.zig:67:18: 0xecc in ??? (clashos)
TODO print line from the file
                 ^
???:?:?: 0x1c in ??? (???)

It's looking good. But can we add a cherry on top, and make it print the source lines?

Again we don't have a file system. How can the printLineFromFile function have access to source files?

Sometimes the simplest solution is the best one. How about the kernel just has its own source code in memory?

That's really easy to hook up in Zig:

const source_files = [][]const u8{
    "src/debug.zig",
    "src/main.zig",
    "src/mmio.zig",
    "src/serial.zig",
};
fn printLineFromFile(out_stream: var, line_info: std.debug.LineInfo) anyerror!void {
    inline for (source_files) |src_path| {
        if (std.mem.endsWith(u8, line_info.file_name, src_path)) {
            const contents = @embedFile("../" ++ src_path);
            try printLineFromBuffer(out_stream, contents[0..], line_info);
            return;
        }
    }
    try out_stream.print("(source file {} not added in std/debug.zig)\n", line_info.file_name);
}

Here we take advantage of inline for as well as @embedFile, and all the sudden we can print lines of code from our own source files. printLineFromBuffer left as an exercise for the reader.

So let's see that output again:

Hello World! ClashOS 0.0

!KERNEL PANIC!
integer overflow
/home/andy/dev/clashos/src/serial.zig:42:7: 0x1b10 in ??? (clashos)
    x += 1;
      ^
/home/andy/dev/clashos/src/main.zig:58:16: 0x1110 in ??? (clashos)
    serial.boom();
               ^
/home/andy/dev/clashos/src/main.zig:67:18: 0xecc in ??? (clashos)
    some_function();
                 ^
???:?:?: 0x1c in ??? (???)

Beautiful. Here's a picture so you can see it with color:

And now all of the protections that Zig offers against undefined behavior will result in output like this.

Conclusion

One of my big goals with Zig is to improve the embedded and OS development process. I hope you're as excited as I am about the potential here.

If this blog post captured your interest, maybe you would like to check out this Hello World x86 Kernel, which comes with a video of me live coding it. Thanks to an insightful Twitch comment, it works with only Zig code, no assembly required.

String Matching based on Compile Time Perfect Hashing in Zig

Inspired by cttrie - Compile time TRIE based string matching, I decided to see what this solution would look like in Zig.

Here's the API I came up with:

const ph = perfectHash([][]const u8{
    "a",
    "ab",
    "abc",
});
switch (ph.hash(target)) {
    ph.case("a") => std.debug.warn("handle the a case"),
    ph.case("ab") => std.debug.warn("handle the ab case"),
    ph.case("abc") => std.debug.warn("handle the abc case"),
    else => unreachable,
}

It notices at compile-time if you forget to declare one of the cases. For example, if I comment out the last item:

const ph = perfectHash([][]const u8{
    "a",
    "ab",
    //"abc",
});

When compiled this gives:

perfect-hashing.zig:147:13: error: case value 'abc' not declared
            @compileError("case value '" ++ s ++ "' not declared");
            ^
perfect-hashing.zig:18:16: note: called from here
        ph.case("abc") => std.debug.warn("handle the abc case\n"),
               ^

It also has runtime safety if you pass in a string that was not prepared for:

const std = @import("std");
const assert = std.debug.assert;

test "perfect hashing" {
    basedOnLength("zzz");
}

fn basedOnLength(target: []const u8) void {
    const ph = perfectHash([][]const u8{
        "a",
        "ab",
        "abc",
    });
    switch (ph.hash(target)) {
        ph.case("a") => @panic("wrong one a"),
        ph.case("ab") => {}, // test pass
        ph.case("abc") => @panic("wrong one abc"),
        else => unreachable,
    }
}

When run this gives:

Test 1/1 perfect hashing...attempt to perfect hash zzz which was not declared
perfect-hashing.zig:156:36: 0x205a21 in ??? (test)
                    std.debug.panic("attempt to perfect hash {} which was not declared", s);
                                   ^
perfect-hashing.zig:15:20: 0x2051bd in ??? (test)
    switch (ph.hash(target)) {
                   ^
perfect-hashing.zig:5:18: 0x205050 in ??? (test)
    basedOnLength("zzz");
                 ^
/home/andy/dev/zig/build/lib/zig/std/special/test_runner.zig:13:25: 0x2238ea in ??? (test)
        if (test_fn.func()) |_| {
                        ^
/home/andy/dev/zig/build/lib/zig/std/special/bootstrap.zig:96:22: 0x22369b in ??? (test)
            root.main() catch |err| {
                     ^
/home/andy/dev/zig/build/lib/zig/std/special/bootstrap.zig:70:20: 0x223615 in ??? (test)
    return callMain();
                   ^
/home/andy/dev/zig/build/lib/zig/std/special/bootstrap.zig:64:39: 0x223478 in ??? (test)
    std.os.posix.exit(callMainWithArgs(argc, argv, envp));
                                      ^
/home/andy/dev/zig/build/lib/zig/std/special/bootstrap.zig:37:5: 0x223330 in ??? (test)
    @noInlineCall(posixCallMainAndExit);
    ^

Tests failed. Use the following command to reproduce the failure:
/home/andy/dev/zig/build/zig-cache/test

So there's the API. How does it work?

Here's the implementation of the perfectHash function:

fn perfectHash(comptime strs: []const []const u8) type {
    const Op = union(enum) {
        /// add the length of the string
        Length,

        /// add the byte at index % len
        Index: usize,

        /// right shift then xor with constant
        XorShiftMultiply: u32,
    };
    const S = struct {
        fn hash(comptime plan: []Op, s: []const u8) u32 {
            var h: u32 = 0;
            inline for (plan) |op| {
                switch (op) {
                    Op.Length => {
                        h +%= @truncate(u32, s.len);
                    },
                    Op.Index => |index| {
                        h +%= s[index % s.len];
                    },
                    Op.XorShiftMultiply => |x| {
                        h ^= x >> 16;
                    },
                }
            }
            return h;
        }

        fn testPlan(comptime plan: []Op) bool {
            var hit = [1]bool{false} ** strs.len;
            for (strs) |s| {
                const h = hash(plan, s);
                const i = h % hit.len;
                if (hit[i]) {
                    // hit this index twice
                    return false;
                }
                hit[i] = true;
            }
            return true;
        }
    };

    var ops_buf: [10]Op = undefined;

    const plan = have_a_plan: {
        var seed: u32 = 0x45d9f3b;
        var index_i: usize = 0;
        const try_seed_count = 50;
        const try_index_count = 50;

        while (index_i < try_index_count) : (index_i += 1) {
            const bool_values = if (index_i == 0) []bool{true} else []bool{ false, true };
            for (bool_values) |try_length| {
                var seed_i: usize = 0;
                while (seed_i < try_seed_count) : (seed_i += 1) {
                    comptime var rand_state = std.rand.Xoroshiro128.init(seed + seed_i);
                    const rng = &rand_state.random;

                    var ops_index = 0;

                    if (try_length) {
                        ops_buf[ops_index] = Op.Length;
                        ops_index += 1;

                        if (S.testPlan(ops_buf[0..ops_index]))
                            break :have_a_plan ops_buf[0..ops_index];

                        ops_buf[ops_index] = Op{ .XorShiftMultiply = rng.scalar(u32) };
                        ops_index += 1;

                        if (S.testPlan(ops_buf[0..ops_index]))
                            break :have_a_plan ops_buf[0..ops_index];
                    }

                    ops_buf[ops_index] = Op{ .XorShiftMultiply = rng.scalar(u32) };
                    ops_index += 1;

                    if (S.testPlan(ops_buf[0..ops_index]))
                        break :have_a_plan ops_buf[0..ops_index];

                    const before_bytes_it_index = ops_index;

                    var byte_index = 0;
                    while (byte_index < index_i) : (byte_index += 1) {
                        ops_index = before_bytes_it_index;

                        ops_buf[ops_index] = Op{ .Index = rng.scalar(u32) % try_index_count };
                        ops_index += 1;

                        if (S.testPlan(ops_buf[0..ops_index]))
                            break :have_a_plan ops_buf[0..ops_index];

                        ops_buf[ops_index] = Op{ .XorShiftMultiply = rng.scalar(u32) };
                        ops_index += 1;

                        if (S.testPlan(ops_buf[0..ops_index]))
                            break :have_a_plan ops_buf[0..ops_index];
                    }
                }
            }
        }

        @compileError("unable to come up with perfect hash");
    };

    return struct {
        fn case(comptime s: []const u8) usize {
            inline for (strs) |str| {
                if (std.mem.eql(u8, str, s))
                    return hash(s);
            }
            @compileError("case value '" ++ s ++ "' not declared");
        }
        fn hash(s: []const u8) usize {
            if (std.debug.runtime_safety) {
                const ok = for (strs) |str| {
                    if (std.mem.eql(u8, str, s))
                        break true;
                } else false;
                if (!ok) {
                    std.debug.panic("attempt to perfect hash {} which was not declared", s);
                }
            }
            return S.hash(plan, s) % strs.len;
        }
    };
}

Here's what this is doing:

• Create the concept of a hashing operation. The operations that can happen are:
  • Length - use wraparound addition to add the length of the string to the hash value.
  • Index - use wraparound addition to add the byte from the string at the specified index. Use modulus arithmetic in case the index is out of bounds.
  • XorShiftMultiply - perform a right shift by 16 bits of the hash value and then xor with the specified value (which will come from a random number generator executed at compile time).
• Next, we're going to try a series of "plans" which is a sequence of operations strung together. We have this function testPlan which performs the hash operations on all the strings and sees if there are any collisions. If we ever find a plan that results in no collisions, we have found a perfect hashing strategy.
• First we test a plan that is simply the Length operation. If this works, then all the hash function has to do is take the length of the string mod the number of strings. Easy. You may notice this is true for the above example. Don't worry, I have a more complicated example below.
• Next we iterate over the count of how many different bytes we are willing to look at in the hash function. We start with 0. If we can xor the length with a random number and it fixes the collisions, then we're done. We try 50 seeds before giving up and deciding to inspect a random byte from the string in the hash function. If the addition of inspecting a random byte from the string to hash doesn't solve the problem, we try 50 seeds in order to choose different random byte indexes. If that still doesn't work, we look at 2 random bytes, again trying 50 different seeds with these 2 bytes. And so on until every combination of 50 seeds x 50 bytes inspected, at which point we give up and emit a compile error "unable to come up with perfect hash".

We can use @compileLog to see the plan that the function came up with:

for (plan) |op| {
    @compileLog(@TagType(Op)(op));
}

This outputs:

| @TagType(Op).Length

So this means that the hash function only has to look at the length for this example.

The nice thing about this is that the plan is all known at compile time. Indeed, you can see that the hash function uses inline for to iterate over the operations in the plan. This means that LLVM is able to fully optimize the hash function.

Here's what it gets compiled to in release mode for x86_64:

0000000000000000 <basedOnLength>:
   0:	89 f0                	mov    %esi,%eax
   2:	b9 ab aa aa aa       	mov    $0xaaaaaaab,%ecx
   7:	48 0f af c8          	imul   %rax,%rcx
   b:	48 c1 e9 21          	shr    $0x21,%rcx
   f:	8d 04 49             	lea    (%rcx,%rcx,2),%eax
  12:	29 c6                	sub    %eax,%esi
  14:	48 89 f0             	mov    %rsi,%rax
  17:	c3                   	retq   
  18:	0f 1f 84 00 00 00 00 	nopl   0x0(%rax,%rax,1)
  1f:	00 

You can see there is not even a jump instruction in there. And it will output numbers in sequential order from 0, so that the switch statement can be a jump table. Here is the LLVM IR of the switch:

 %2 = call fastcc i64 @"perfectHash((struct []const []const u8 constant))_hash"(%"[]u8"* %0), !dbg !736
  store i64 %2, i64* %1, align 8, !dbg !736
  %3 = load i64, i64* %1, align 8, !dbg !740
  switch i64 %3, label %SwitchElse [
    i64 1, label %SwitchProng
    i64 2, label %SwitchProng1
    i64 0, label %SwitchProng2
  ], !dbg !740

How about a harder example?

@setEvalBranchQuota(100000);
const ph = perfectHash([][]const u8{
    "one",
    "two",
    "three",
    "four",
    "five",
});
switch (ph.hash(target)) {
    ph.case("one") => std.debug.warn("handle the one case"),
    ph.case("two") => std.debug.warn("handle the two case"),
    ph.case("three") => std.debug.warn("handle the three case"),
    ph.case("four") => std.debug.warn("handle the four case"),
    ph.case("five") => std.debug.warn("handle the five case"),
    else => unreachable,
}

This example is interesting because there are 2 pairs of length collisions (one/two, four/five) and 2 pairs of byte collisions (two/three, four/five).

Here we have to use @setEvalBranchQuota because it takes a bit of computation to come up with the answer.

Again, the hash function comes up with a mapping to 0, 1, 2, 3, 4 (but not necessarily in the same order as specified):

 %2 = call fastcc i64 @"perfectHash((struct []const []const u8 constant))_hash.11"(%"[]u8"* %0), !dbg !749
  store i64 %2, i64* %1, align 8, !dbg !749
  %3 = load i64, i64* %1, align 8, !dbg !753
  switch i64 %3, label %SwitchElse [
    i64 4, label %SwitchProng
    i64 2, label %SwitchProng1
    i64 3, label %SwitchProng2
    i64 0, label %SwitchProng3
    i64 1, label %SwitchProng4
  ], !dbg !753

And the optimized assembly:

0000000000000020 <basedOnOtherStuff>:
  20:	48 83 fe 22          	cmp    $0x22,%rsi
  24:	77 0b                	ja     31 <basedOnOtherStuff+0x11>
  26:	b8 22 00 00 00       	mov    $0x22,%eax
  2b:	31 d2                	xor    %edx,%edx
  2d:	f7 f6                	div    %esi
  2f:	eb 05                	jmp    36 <basedOnOtherStuff+0x16>
  31:	ba 22 00 00 00       	mov    $0x22,%edx
  36:	0f b6 04 17          	movzbl (%rdi,%rdx,1),%eax
  3a:	05 ef 3d 00 00       	add    $0x3def,%eax
  3f:	b9 cd cc cc cc       	mov    $0xcccccccd,%ecx
  44:	48 0f af c8          	imul   %rax,%rcx
  48:	48 c1 e9 22          	shr    $0x22,%rcx
  4c:	8d 0c 89             	lea    (%rcx,%rcx,4),%ecx
  4f:	29 c8                	sub    %ecx,%eax
  51:	c3                   	retq   

We have a couple of jumps here, but no loop. This code looks at only 1 byte from the target string to determine the corresponding case index. We can see that more clearly with the @compileLog snippet from earlier:

| @TagType(Op).XorShiftMultiply
| @TagType(Op).Index

So it initializes the hash with a constant value of predetermined random bits, then adds the byte from a randomly chosen index of the string. Using @compileLog(plan[1].Index) I determined that it is choosing the value 34, which means that:

• For one, 34 % 3 == 1, it looks at the 'n'
• For two, 34 % 3 == 1, it looks at the 'w'
• For three, 34 % 5 == 4, it looks at the 'e'
• For four, 34 % 4 == 2, it looks at the 'u'
• For five, 34 % 4 == 2, it looks at the 'i'

So the perfect hashing exploited that these bytes are all different, and when combined with the random constant that it found, the final modulus spreads out the value across the full range of 0, 1, 2, 3, 4.

Can we do better?

There are lots of ways this can be improved - this is just a proof of concept. For example, we could start by looking for a perfect hash in the subset of bytes that are within the smallest string length. For example, if we used index 1 in the above example, we could avoid the jumps and remainder division instructions in the hash function.

Another way this can be improved, to reduce compile times, is to have the perfectHash function accept the RNG seed as a parameter. If the seed worked first try, great. Otherwise the function would find the seed that does work, and emit a compile error instructing the programmer to switch the seed argument to the good one, saving time in future compilations.

Conclusion

If you like this, you should check out Zig. Consider becoming a sponsor.

I Quit My Cushy Job at OkCupid to Live on Donations to Zig

I am fortunate to be one of those people who started tinkering with code in their teens, and then found themselves in the position where their hobby could not only pay rent, but afford a comfortable standard of living, even in New York City.

For a little over a year now, I worked on the backend team at OkCupid, maintaining a large C++ codebase that has not been reworked since 2005 when Maxwell Krohn published the original OKWS paper on which OkCupid is still based.

As satisfying as it is to delete dead code, rework abstractions to remove footguns, and migrate a monolithic codebase to Service Oriented Architecture, my coworkers were well aware that my real passion was spent on nights and weekends, where I poured all the energy I could muster into Zig - if not from my incessant comparisons between snippets of C++ code and how it could be better expressed in Zig, then from the tea mug that my lovely girlfriend Alee got me as a celebratory gift for reaching release 0.1.0:

Zig is Picking Up Steam

Ever since I gave this Software Should be Perfect talk, it seems that the Zig community has seen a steady influx of new members.

David keeps sending me selfies with this hat that he made.

The Zig community grew in size so much, that some weekends I found myself with only enough time to merge pull requests and respond to issues, and no time leftover to make progress on big roadmap changes.

I found this both exciting, and frustrating. Here I was working on legacy code 40 hours per week, while multiple Zig issues needed my attention.

And so I took the plunge. I gave up my senior backend software engineer salary, and will attempt to live on monthly donations.

Roadmap

Here are some of the bigger items that are coming up now that I have more time:

• remove more sigils from the language
• add tuples and remove var args
• self-hosted compiler
• http server and http client based on async/await
• decentralized package manager
• generate html documentation
• hot code swapping

I can't wait to make significant progress on these items. That said, I will also dedicate time each day for bug fixes and writing documentation. Every weekday except Friday, this will be my wakeup process:

1. Make tea
2. Fix a bug
3. Write some documentation
4. Deal with all open pull requests, whether that is by merging, rejecting, requesting changes, or solving the use case a different way.
5. Proceed with working on a big feature item.

On Fridays, I will spend the day working on a project other than Zig, but one that is implemented in Zig. For example:

• Rewrite Groove Basin in Zig.
• Work on my game that the raspberry pi boots directly into (this explores Zig's ability to be used for embedded and Operating System development)
• Start reworking my digital audio workstation in Zig.

I'm beyond excited to get Zig to a place where it can reasonably be used for high quality and practical software projects. My goal is to make Zig so useful and practical that people will find themselves using it without intending to.

And so it comes down to this - will you fund my efforts?

Why Donate to Zig?

Consider the basic premise of a for-profit business:

• Customers pay currency in exchange for goods or services.

If all goes well, both parties benefit from the exchange. But for-profit businesses are motivated, at the end of the day, not by customer satisfaction, but by profit.

It's fine, this is how the world works. But it's not how Zig works.

Zig is created by the open source community, for the open source community. One of our main tenets is

Together we serve end users.

The Zig project is a public service, entirely motivated by improving the technical landscape of open source tools. There is no board to please, no stock shareholders demanding higher quarterly earnings, no office politics or career progressions on the line. 100% of donations I receive go towards paying rent, buying food, and generally attempting to live a modest, but healthy life.

Will you pledge $5/month?

Zig: January 2018 in Review

One month (and a few days, sorry I'm late!) has passed since I did the December 2017 writeup, and so it's time for another month-in-review for my esteemed sponsors.

LLVM 6 Readiness

LLVM 6.0.0rc2 was just announced on the mailing list. It's scheduled to be released on February 21, and Zig is ready. I plan to have Zig release 0.2.0 one week after LLVM 6 comes out. We already have all tests passing with debug builds of LLVM 6 in the llvm6 branch, but that extra week is for a bug-stomping rampage.

After that, all the 0.3.0 milestone issues get postponed to 0.4.0, and all the 0.2.0 milestone issues get moved to 0.3.0.

0.2.0 will be an exciting release because, among many other things, it enables source-level debugging with MSVC on Windows.

Zig is once again in the release notes of LLVM, so we should see a slight increase in community size when LLVM release notes hit the tech news headlines.

Error Syntax Cleanup

One of the biggest complaints newcomers to Zig had was about its sigils regarding error handling. Given this, I made an effort to choose friendlier syntax.

• %return is replaced with try
• %defer is replaced with errdefer
• a %% b is replaced with a catch b
• %%x is removed entirely to discourage its use. You can get an equivalent effect with x catch unreachable, which has been updated to understand that it was attempting to unwrap an error union. See #545 and #510

After these changes, there is a strong pattern that only keywords can modify control flow. For example we have and and or instead of && and ||. There is one last exception, which is a ?? b. Maybe it's okay, since C# set a precedent.

An even bigger change is coming soon which I'm calling Error Sets.

Error Return Traces

I'm really excited about this one. I invented a new kind of debugging tool and integrated it into Debug and ReleaseSafe builds.

One of the concerns with removing the %% prefix operator was that it was just so gosh darn convenient to get a stack trace right at the moment where you asserted that a value did not have an error. I wanted to make it so that programmers could use try everywhere and still get the debuggability benefit when an error occurred.

Watch this:

const std = @import("std");

pub fn main() !void {
    const allocator = std.debug.global_allocator;

    const args = try std.os.argsAlloc(allocator);
    defer std.os.argsFree(allocator, args);

    const count = try parseFile(allocator, args[1]);

    if (count < 10) return error.NotEnoughItems;
}

fn parseFile(allocator: &std.mem.Allocator, file_path: []const u8) !usize {
    const contents = std.io.readFileAlloc(allocator, file_path) catch return error.UnableToReadFile;
    defer allocator.free(contents);

    return contents.len;
}

Here's a simple program with a bunch of different ways that errors could get returned from main. In our test example, we're going to open a bogus file that does not exist.

$ zig build-exe test2.zig
$ ./test2 bogus-does-not-exist.txt
error: UnableToReadFile
/home/andy/dev/zig/build/lib/zig/std/os/index.zig:301:33: 0x000000000021acd0 in ??? (test2)
                posix.ENOENT => return PosixOpenError.PathNotFound,
                                ^
/home/andy/dev/zig/build/lib/zig/std/os/file.zig:25:24: 0x00000000002096f6 in ??? (test2)
            const fd = try os.posixOpen(allocator, path, flags, 0);
                       ^
/home/andy/dev/zig/build/lib/zig/std/io.zig:267:16: 0x000000000021ebec in ??? (test2)
    var file = try File.openRead(allocator, path);
               ^
/home/andy/dev/zig/build/test2.zig:15:71: 0x000000000021ce72 in ??? (test2)
    const contents = std.io.readFileAlloc(allocator, file_path) catch return error.UnableToReadFile;
                                                                      ^
/home/andy/dev/zig/build/test2.zig:9:19: 0x000000000021c1f9 in ??? (test2)
    const count = try parseFile(allocator, args[1]);
                  ^

I'm going to include a picture of the above here, because it looks a lot better with terminal colors:

This is not a stack trace snapshot from when an error was "created". This is a return trace of all the points in the code where an error was returned from a function.

Note that, if it only told you the origin of the error that we ultimately received - UnableToReadFile - we would only see the bottom 2 items in the trace. Not only do we have this information, we have all the information about the origin of the error, right up to the fact that we received ENOENT from open.

With this in place, programmers can comfortably use try everywhere, safe in the knowledge that it will be straightforward to troubleshoot the origin of any error bubbling up through the system.

I hope you're skeptically wondering, OK, what's the tradeoff in terms of binary size, performance, and memory?

First of all, this feature is disabled in ReleaseFast mode. So the answer is, literally no cost, in this case. But what about Debug and ReleaseSafe builds?

To analyze performance cost, there are two cases:

• when no errors are returned
• when returning errors

For the case when no errors are returned, the cost is a single memory write operation, only in the first non-failable function in the call graph that calls a failable function, i.e. when a function returning void calls a function returning error. This is to initialize this struct in the stack memory:

pub const StackTrace = struct {
    index: usize,
    instruction_addresses: [N]usize,
};

Here, N is the maximum function call depth as determined by call graph analysis. Recursion is ignored and counts for 2.

A pointer to StackTrace is passed as a secret parameter to every function that can return an error, but it's always the first parameter, so it can likely sit in a register and stay there.

That's it for the path when no errors occur. It's practically free in terms of performance.

When generating the code for a function that returns an error, just before the return statement (only for the return statements that return errors), Zig generates a call to this function:

noinline fn __zig_return_error(stack_trace: &StackTrace) void {
    stack_trace.instruction_addresses[stack_trace.index] = @returnAddress();
    stack_trace.index = (stack_trace.index + 1) % N;
}

The cost is 2 math operations plus some memory reads and writes. The memory accessed is constrained and should remain cached for the duration of the error return bubbling.

As for code size cost, 1 function call before a return statement is no big deal. Even so, I have a plan to make the call to __zig_return_error a tail call, which brings the code size cost down to actually zero. What is a return statement in code without error return tracing can become a jump instruction in code with error return tracing.

There are a few ways to activate this error return tracing feature:

• Return an error from main
• An error makes its way to catch unreachable and you have not overridden the default panic handler
• Use @errorReturnTrace to access the current return trace. You can use std.debug.dumpStackTrace to print it. This function returns comptime-known null when building without error return tracing support.

Related issues: #651 #684

Documentation

Big news on the documentation front.

All the outdated docs are fixed, and we have automatic docgen tool which:

• Automatically generates the table of contents
• Validates all internal links
• Validates all code examples
• Turns terminal coloring of stack traces and compile errors into HTML

The tool is, of course, written in Zig. #465

In addition to the above, the following improvements were made to the documentation:

• Added documentation for @noInlineCall
• Added documentation for extern enum
• Improved the documentation styling
• Made the documentation a single file that has no external dependencies
• Add the documentation to appveyor build artifacts as langref.html. In other words we ship with the docs now.

Marc Tiehuis improved documentation styling for mobile devices. #729

• No overscrolling on small screens
• Font-size is reduced for more content per screen
• Tables + Code blocks scroll within a block to avoid page-widenening

There is still much more to document, before we have achieved basic documentation for everything.

Self-Hosted Compiler

The self-hosted compiler now fully successfully builds on Windows and MacOS.

The main test suite builds the self-hosted compiler.

The self-hosted build inherits the std lib file list and C header file list from the stage1 cmake build, as well as the llvm-config output. So if you get stage1 to build, stage2 will reliably build as well.

Windows 32-bit Support Status

Windows 32-bit mostly works, but there are some failing tests. The number of failing tests grew and it didn't seem fair to claim that we supported it officially.

So I removed the claims that we support Windows 32-bit from the README, and removed 32-bit Windows from the testing matrix.

We still want to support Windows 32-bit.

Syntax: Mandatory Function Return Type

-> is removed, and all functions require an explicit return type.

The purpose of this is:

• Only one way to do things
• Changing a function with void return type to return a possible error becomes a 1 character change, subtly encouraging people to use errors.

Here are some imperfect sed commands for performing this update:

remove arrow:

sed -i 's/\(\bfn\b.*\)-> /\1/g' $(find . -name "*.zig")

add void:

sed -i 's/\(\bfn\b.*\))\s*{/\1) void {/g' $(find . -name "*.zig")

Some cleanup may be necessary, but this should do the bulk of the work.

This has been a controversial change, and may be reverted.

Generating .h Files

• Now Zig emits compile errors for non-extern, non-packed struct, enum, unions in extern fn signatures.
• Zig generates .h file content for extern struct, enum, unions
• .h file generation is now tested in the main test suite.

Marc Tiehuis added array type handling:

const Foo = extern struct {
    A: [2]i32,
    B: [4]&u32,
};
export fn entry(foo: Foo, bar: [3]u8) void { }

This generates:

struct Foo {
    int32_t A[2];
    uint32_t * B[4];
};

TEST_EXPORT void entry(struct Foo foo, uint8_t bar[]);

Translating C to Zig

Jimmi Holst Christensen improved translate-c:

• output "undefined" on uninitialized variables
• correct translation of if statements on integers and floats

Crypto Additions to Zig std lib

Marc Tiehuis added a bunch of crypto functions:

• added hardware sqrt for x86_64. (See #681)
• fixed bitrotted endian swapping std lib code (See #682)

Integer Rotation Functions

/// Rotates right. Only unsigned values can be rotated.
/// Negative shift values results in shift modulo the bit count.
pub fn rotr(comptime T: type, x: T, r: var) -> T {
test "math.rotr" {
    assert(rotr(u8, 0b00000001, usize(0))  == 0b00000001);
    assert(rotr(u8, 0b00000001, usize(9))  == 0b10000000);
    assert(rotr(u8, 0b00000001, usize(8))  == 0b00000001);
    assert(rotr(u8, 0b00000001, usize(4))  == 0b00010000);
    assert(rotr(u8, 0b00000001, isize(-1)) == 0b00000010);
}
/// Rotates left. Only unsigned values can be rotated.
/// Negative shift values results in shift modulo the bit count.
pub fn rotl(comptime T: type, x: T, r: var) -> T {
test "math.rotl" {
    assert(rotl(u8, 0b00000001, usize(0))  == 0b00000001);
    assert(rotl(u8, 0b00000001, usize(9))  == 0b00000010);
    assert(rotl(u8, 0b00000001, usize(8))  == 0b00000001);
    assert(rotl(u8, 0b00000001, usize(4))  == 0b00010000);
    assert(rotl(u8, 0b00000001, isize(-1)) == 0b10000000);
}

MD5 and SHA1 Hash Functions

Marc writes:

Some performance comparisons to C.

We take the fastest time measurement taken across multiple runs.

The block hashing functions use the same md5/sha1 methods.

Cpu: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
Gcc: 7.2.1 20171224
Clang: 5.0.1
Zig: 0.1.1.304f6f1d

See https://www.nayuki.io/page/fast-md5-hash-implementation-in-x86-assembly:

gcc -O2
    661 Mb/s
clang -O2
    490 Mb/s
zig --release-fast and zig --release-safe
    570 Mb/s
zig
    50 Mb/s

See https://www.nayuki.io/page/fast-sha1-hash-implementation-in-x86-assembly :

gcc -O2
    588 Mb/s
clang -O2
    563 Mb/s
zig --release-fast and zig --release-safe
    610 Mb/s
zig
    21 Mb/s

In short, zig provides pretty useful tools for writing this sort of code. We are in the lead against clang (which uses the same LLVM backend) with us being slower only against md5 with GCC.

SHA-2 Functions

Marc writes:

We take the fastest time measurement taken across multiple runs. Tested across multiple compiler flags and the best chosen.

Cpu: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
Gcc: 7.2.1 20171224
Clang: 5.0.1
Zig: 0.1.1.304f6f1d

See https://www.nayuki.io/page/fast-sha2-hashes-in-x86-assembly.

Gcc -O2
    219 Mb/s
Clang -O2
    213 Mb/s
Zig --release-fast
    284 Mb/s
Zig --release-safe
    211 Mb/s
Zig
    6 Mb/s

Gcc -O2
    350 Mb/s
Clang -O2
    354 Mb/s
Zig --release-fast
    426 Mb/s
Zig --release-safe
    300 Mb/s
Zig
    11 Mb/s

Blake2 Hash Functions

Marc writes:

Blake performance numbers for reference:

Cpu: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz

-- Blake2s

Zig --release-fast
    485 Mb/s
Zig --release-safe
    377 Mb/s
Zig
    11 Mb/s

-- Blake2b

Zig --release-fast
    616 Mb/s
Zig --release-safe
    573 Mb/s
Zig
    18 Mb/s

Sha3 Hashing Functions

Marc writes:

These are on the slower side and could be improved. No performance optimizations yet have been done.

Cpu: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz

-- Sha3-256

Zig --release-fast
    93 Mb/s
Zig --release-safe
    99 Mb/s
Zig
    4 Mb/s

-- Sha3-512

Zig --release-fast
    49 Mb/s
Zig --release-safe
    54 Mb/s
Zig
    2 Mb/s

Interestingly, release-safe is producing slightly better code than release-fast.

Improvements

• The return type of main can now be void, noreturn, u8, or an error union. #535
• Implemented bigint div and rem. #405
• Removed coldcc keyword and added @setCold. #661
• Renamed "debug safety" to "runtime safety". #437
• Updated windows build to use llvm 5.0.1. Reported usability issue regarding diaguids.lib to llvm-dev.
• Implemented std.os.selfExePath and std.os.selfExeDirPath for windows.
• Added more test coverage.
• The same string literal codegens to the same constant. This makes it so that you can send the same string literal as a comptime slice and get the same type.

Zig now supports structs defined inside a function that reference local constants:

const assert = @import("std").debug.assert;

test "struct inside function" {
    const BlockKind = u32;

    const Block = struct {
        kind: BlockKind,
    };

    var block = Block { .kind = 1234 };

    block.kind += 1;

    assert(block.kind == 1235);
}

This fixed #672 and #552. However there is still issue #675 which is that structs inside functions get named after the function they are in:

test "struct inside function" {
    const Block = struct { kind: u32 };
    @compileLog(@typeName(Block));
}

When executed gives

| "struct inside function()"

Moving on, Zig now allows enum tag values to not be in parentheses:

const EnumWithTagValues = enum(u4) {
    A = 1 << 0,
    B = 1 << 1,
    C = 1 << 2,
    D = 1 << 3,
};

Previously this required A = (1 << 0).

Bug Fixes

• fix exp1m implementation by using @setFloatMode and using modular arithmetic
• fix compiler crash related to @alignOf
• fix null debug info for 0-length array type. #702
• fix compiler not able to rename files into place on windows if the file already existed
• fix crash when switching on enum with 1 field and no switch prongs. #712
• fix crash on union-enums with only 1 field. #713
• fix crash when align 1 field before self referential align 8 field as slice return type. #723
• fix error message mentioning unreachable instead of noreturn
• fix std.io.readFileAllocExtra incorrectly returning error.EndOfStream
• workaround for microsoft releasing windows SDK with the wrong version
• Found a bug in NewGVN. Disabled it to match clang and filed an llvm bug.
• emit compile error for @panic called at compile time. #706
• emit compile error for shifting by negative comptime integer. #698
• emit compile error for calling naked function. @ptrCast a naked function first to call it.
• emit compile error for duplicate struct, enum, union fields. #730

Thank you contributors!

• Jimmi Holst Christensen fixed bitrotted code: `std.Rand.scalar` and `std.endian.swap`. (See #674)
• Andrea Orru removed the deprecated Darwin target and added Zen OS target. (See #438)
• Andrea Orru added intrusive linked lists to the standard library. (See #680)
• Jimmi Holst Christensen fixed bigint xor with zero. (See #701)
• Jimmi Holst Christensen implemented windows versions of seekTo and getPos (See #710)
• Jeff Fowler sent a pull request to GitHub/linguist to add Zig syntax highlighting. (See github/linguist/pull/4005)
• Jeff Fowler Added Zig 0.1.1 to Homebrew. (See Homebrew/homebrew-core/pull/23189)

Thank you financial supporters!

Special thanks to those who donate monthly. We're now at $207 of the $3,000 goal.

• Lauren Chavis
• Andrea Orru
• Adrian Sinclair
• David Joseph
• jeff kelley
• Hasen Judy
• Wesley Kelley
• Harry Eakins
• Richard Ohnemus
• Brendon Scheinman
• Martin Schwaighofer
• Matthew
• Mirek Rusin
• Jordan Torbiak
• Pyry Kontio
• Thomas Ballinger
• Peter Ronnquist
• Luke McCarthy
• Robert Paul Herman
• Audun Wilhelmsen
• Marko Mikulicic
• Jimmi Holst Christensen
• Caius
• Don Poor
• Anthony J. Benik
• David Hayden
• Tanner Schultz
• Tyler Philbrick
• Eduard Nicodei
• Christopher A. Butler
• Colleen Silva-Hayden
• Jeremy Larkin
• Rasmus Rønn Nielsen
• Brian Lewis
• Tom Palmer
• Josh McDonald
• Chad Russell
• Alexandra Gillis
• david karapetyan
• Zi He Goh

Unsafe Zig is Safer than Unsafe Rust

Consider the following Rust code:

struct Foo {
    a: i32,
    b: i32,
}

fn main() {
    unsafe {
        let mut array: [u8; 1024] = [1; 1024];
        let foo = std::mem::transmute::<&mut u8, &mut Foo>(&mut array[0]);
        foo.a += 1;
    }
}

This pattern is pretty common if you are interacting with Operating System APIs. Another example.

Can you spot the problem with the code?

It's pretty subtle, but there is actually undefined behavior going on here. Let's take a look at the LLVM IR:

define internal void @_ZN4test4main17h916a53db53ad90a1E() unnamed_addr #0 {
start:
  %transmute_temp = alloca %Foo*
  %array = alloca [1024 x i8]
  %0 = getelementptr inbounds [1024 x i8], [1024 x i8]* %array, i32 0, i32 0
  call void @llvm.memset.p0i8.i64(i8* %0, i8 1, i64 1024, i32 1, i1 false)
  br label %bb1

bb1:                                              ; preds = %start
  %1 = getelementptr inbounds [1024 x i8], [1024 x i8]* %array, i64 0, i64 0
  %2 = bitcast %Foo** %transmute_temp to i8**
  store i8* %1, i8** %2, align 8
  %3 = load %Foo*, %Foo** %transmute_temp, !nonnull !1
  br label %bb2

bb2:                                              ; preds = %bb1
  %4 = getelementptr inbounds %Foo, %Foo* %3, i32 0, i32 0
  %5 = load i32, i32* %4
  %6 = call { i32, i1 } @llvm.sadd.with.overflow.i32(i32 %5, i32 1)
  %7 = extractvalue { i32, i1 } %6, 0
  %8 = extractvalue { i32, i1 } %6, 1
  %9 = call i1 @llvm.expect.i1(i1 %8, i1 false)
  br i1 %9, label %panic, label %bb3

bb3:                                              ; preds = %bb2
  %10 = getelementptr inbounds %Foo, %Foo* %3, i32 0, i32 0
  store i32 %7, i32* %10
  ret void

panic:                                            ; preds = %bb2
; call core::panicking::panic
  call void @_ZN4core9panicking5panic17hfecc01813e436969E({ %str_slice, [0 x i8], %str_slice, [0 x i8], i32, [0 x i8], i32, [0 x i8] }* noalias readonly dereferenceable(40) bitcast ({ %str_slice, %str_slice, i32, i32 }* @panic_loc.2 to { %str_slice, [0 x i8], %str_slice, [0 x i8], i32, [0 x i8], i32, [0 x i8] }*))
  unreachable
}

That's the code for the main function. This is using rustc version 1.21.0. Let's zoom in on the problematic parts:

  %array = alloca [1024 x i8]
  ; loading foo.a in order to do + 1
  %5 = load i32, i32* %4
  ; storing the result of + 1 into foo.a
  store i32 %7, i32* %10

None of these alloca, load, or store instructions have alignment attributes on them, so they use the ABI alignment of the respective types.

That means the i8 array gets alignment of 1, since the ABI alignment of i8 is 1, and the load and store instructions get alignment 4, since the ABI alignment of i32 is 4. This is undefined behavior:

the store has undefined behavior if the alignment is not set to a value which is at least the size in bytes of the pointee

the load has undefined behavior if the alignment is not set to a value which is at least the size in bytes of the pointee

It's a nasty bug, because besides being an easy mistake to make, on some architectures it will only cause mysterious slowness, while on others it can cause an illegal instruction exception on the CPU. Regardless, it's undefined behavior, and we are professionals, and so we do not accept undefined behavior.

Let's try writing the equivalent code in Zig:

const Foo = struct {
    a: i32,
    b: i32,
};

pub fn main() {
    var array = []u8{1} ** 1024;
    const foo = @ptrCast(&Foo, &array[0]);
    foo.a += 1;
}

And now we compile it:

/home/andy/tmp/test.zig:8:17: error: cast increases pointer alignment
    const foo = @ptrCast(&Foo, &array[0]);
                ^
/home/andy/tmp/test.zig:8:38: note: '&u8' has alignment 1
    const foo = @ptrCast(&Foo, &array[0]);
                                     ^
/home/andy/tmp/test.zig:8:27: note: '&Foo' has alignment 4
    const foo = @ptrCast(&Foo, &array[0]);
                          ^

Zig knows not to compile this code. Here's how to fix it:

@@ -4,7 +4,7 @@
 };
 
 pub fn main() {
-    var array = []u8{1} ** 1024;
+    var array align(@alignOf(Foo)) = []u8{1} ** 1024;
     const foo = @ptrCast(&Foo, &array[0]);
     foo.a += 1;
 }

Now it compiles fine. Let's have a look at the LLVM IR:

define internal fastcc void @main() unnamed_addr #0 !dbg !8911 {
Entry:
  %array = alloca [1024 x i8], align 4
  %foo = alloca %Foo*, align 8
  %0 = bitcast [1024 x i8]* %array to i8*, !dbg !8923
  call void @llvm.memcpy.p0i8.p0i8.i64(i8* %0, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @266, i32 0, i32 0), i64 1024, i32 4, i1 false), !dbg !8923
  call void @llvm.dbg.declare(metadata [1024 x i8]* %array, metadata !8914, metadata !529), !dbg !8923
  %1 = getelementptr inbounds [1024 x i8], [1024 x i8]* %array, i64 0, i64 0, !dbg !8924
  %2 = bitcast i8* %1 to %Foo*, !dbg !8925
  store %Foo* %2, %Foo** %foo, align 8, !dbg !8926
  call void @llvm.dbg.declare(metadata %Foo** %foo, metadata !8916, metadata !529), !dbg !8926
  %3 = load %Foo*, %Foo** %foo, align 8, !dbg !8927
  %4 = getelementptr inbounds %Foo, %Foo* %3, i32 0, i32 0, !dbg !8927
  %5 = load i32, i32* %4, align 4, !dbg !8927
  %6 = call { i32, i1 } @llvm.sadd.with.overflow.i32(i32 %5, i32 1), !dbg !8929
  %7 = extractvalue { i32, i1 } %6, 0, !dbg !8929
  %8 = extractvalue { i32, i1 } %6, 1, !dbg !8929
  br i1 %8, label %OverflowFail, label %OverflowOk, !dbg !8929

OverflowFail:                                     ; preds = %Entry
  tail call fastcc void @panic(%"[]u8"* @88, %StackTrace* null), !dbg !8929
  unreachable, !dbg !8929

OverflowOk:                                       ; preds = %Entry
  store i32 %7, i32* %4, align 4, !dbg !8929
  ret void, !dbg !8930
}

Zooming in on the relevant parts:

  %array = alloca [1024 x i8], align 4
  %5 = load i32, i32* %4, align 4, !dbg !8927
  store i32 %7, i32* %4, align 4, !dbg !8929

Notice that the alloca, load, and store all agree on the alignment.

In Zig the problem of alignment is solved completely; the compiler catches all possible alignment issues. In the situation where you need to assert to the compiler that something is more aligned than Zig thinks it is, you can use @alignCast. This inserts a cheap safety check in debug mode to make sure the alignment assertion is correct.

Zig: December 2017 in Review

I figured since I ask people to donate monthly, I will start giving a monthly progress report to provide accountability.

So here's everything that happened in Zig land in December 2017:

enum tag types

You can now specify the tag type for an enum (#305):

const Small2 = enum (u2) {
    One,
    Two,
};

If you specify the tag type for an enum, you can put it in a packed struct:

const A = enum (u3) {
    One,
    Two,
    Three,
    Four,
    One2,
    Two2,
    Three2,
    Four2,
};

const B = enum (u3) {
    One3,
    Two3,
    Three3,
    Four3,
    One23,
    Two23,
    Three23,
    Four23,
};

const C = enum (u2) {
    One4,
    Two4,
    Three4,
    Four4,
};

const BitFieldOfEnums = packed struct {
    a: A,
    b: B,
    c: C,
};

const bit_field_1 = BitFieldOfEnums {
    .a = A.Two,
    .b = B.Three3,
    .c = C.Four4,
};

You can no longer cast from an enum to an arbitrary integer. Instead you must cast to the enum tag type and vice versa:

const Small2 = enum (u2) {
    One,
    Two,
};
test "casting enum to its tag type" {
    testCastEnumToTagType(Small2.Two);
}

fn testCastEnumToTagType(value: Small2) {
    assert(u2(value) == 1);
}

enum tag values

Now you can set the tag values of enums:

const MultipleChoice = enum(u32) {
    A = 20,
    B = 40,
    C = 60,
    D = 1000,
};

Complete enum and union overhaul

Related issue: #618

Enums are now a simple mapping between a symbol and a number. They can no longer contain payloads.

Unions have been upgraded and can now accept an enum as an argument:

const TheTag = enum {A, B, C};
const TheUnion = union(TheTag) { A: i32, B: i32, C: i32 };
test "union field access gives the enum values" {
    assert(TheUnion.A == TheTag.A);
    assert(TheUnion.B == TheTag.B);
    assert(TheUnion.C == TheTag.C);
}

If you want to auto-create an enum for a union, you can use the enum keyword like this:

const TheUnion2 = union(enum) {
    Item1,
    Item2: i32,
};

You can switch on a union-enum just like you could previously with an enum:

const SwitchProngWithVarEnum = union(enum) {
    One: i32,
    Two: f32,
    Meh: void,
};
fn switchProngWithVarFn(a: &const SwitchProngWithVarEnum) {
    switch(*a) {
        SwitchProngWithVarEnum.One => |x| {
            assert(x == 13);
        },
        SwitchProngWithVarEnum.Two => |x| {
            assert(x == 13.0);
        },
        SwitchProngWithVarEnum.Meh => |x| {
            const v: void = x;
        },
    }
}

However, if you do not give an enum to a union, the tag value is not visible to the programmer:

const Payload = union {
    A: i32,
    B: f64,
    C: bool,
};
export fn entry() {
    const a = Payload { .A = 1234 };
    foo(a);
}
fn foo(a: &const Payload) {
    switch (*a) {
        Payload.A => {},
        else => unreachable,
    }
}

test.zig:11:13: error: switch on union which has no attached enum
    switch (*a) {
            ^
test.zig:1:17: note: consider 'union(enum)' here
const Payload = union {
                ^
test.zig:12:16: error: container 'Payload' has no member called 'A'
        Payload.A => {},
               ^

There is still debug safety though!

const Foo = union {
    float: f32,
    int: u32,
};

pub fn main() -> %void {
    var f = Foo { .int = 42 };
    bar(&f);
}

fn bar(f: &Foo) {
    f.float = 12.34;
}

access of inactive union field
lib/zig/std/special/panic.zig:12:35: 0x0000000000203674 in ??? (test)
        @import("std").debug.panic("{}", msg);
                                  ^
test.zig:12:6: 0x0000000000217bd7 in ??? (test)
    f.float = 12.34;
     ^
test.zig:8:8: 0x0000000000217b7c in ??? (test)
    bar(&f);
       ^
Aborted

However, if you make an extern union to be compatible with C code, there is no debug safety, just like a C union.

Other tidbits:

• @enumTagName is renamed to @tagName
• @EnumTagType is renamed to @TagType, and it works on both enums and union-enums.
• There is no longer an EnumTag type
• It is now an error for enums and unions to have 0 fields. However you can still have a struct with 0 fields.
• union values can implicitly cast to enum values when the enum type is the tag type of the union and the union value tag is comptime known to have a void field type. likewise, enum values can implicitly cast to union values. See #642.

test "cast tag type of union to union" {
    var x: Value2 = Letter2.B;
    assert(Letter2(x) == Letter2.B);
}
const Letter2 = enum { A, B, C };
const Value2 = union(Letter2) { A: i32, B, C, };

test "implicit cast union to its tag type" {
    var x: Value2 = Letter2.B;
    assert(x == Letter2.B);
    giveMeLetterB(x);
}
fn giveMeLetterB(x: Letter2) {
    assert(x == Value2.B);
}

Update LLD fork to 5.0.1rc2

We have a fork of LLD in the zig project because of several upstream issues, all of which I have filed bugs for:

• LDD calls exit after successful link. Patch to fix sent upstream and accepted.
• LDD crashes on a linker script with empty sections. Fixed upstream.
• Buggy MACH-O code causes assertion failure in simple object. We have a hacky workaround for this bug in zig's fork of LLD, but the workaround is not good enough to send upstream. See #662 for more details.
• MACH-O: Bad ASM code generated for __stub_helpers section Thanks to Patricio V. for sending the fix upstream, which has been accepted.

When LLVM 6.0.0 comes out, Zig will have to keep its fork because of the one issue, but we can drop all the other patches since they have been accepted upstream.

Self-hosted compiler progress

The self-hosted compiler effort has begun.

So far we have a tokenizer, and an incomplete parser and formatter. The code uses no recursion and therefore has compile-time known stack space usage. See #157

The self-hosted compiler works on every supported platform, is built using the zig build system, tested with zig test, links against LLVM, and can import 100% of the LLVM symbols from the LLVM C-API .h files - even the inline functions.

There is one C++ file in Zig which uses the more powerful LLVM C++ API (for example to create debug information) and exposes a C API. This file is now shared between the C++ self-hosted compiler and the self-hosted compiler. In stage1, we create a static library with this one file in it, and then use that library in both the C++ compiler and the self-hosted compiler.

Higher level arg-parsing API

It's really a shame that Windows command line parsing requires you to allocate memory. This means that to have a cross-platform API for command line arguments, even though in POSIX it can never fail, we have to handle the possibility because of Windows. This lead to a command line args API like this:

pub fn main() -> %void {
    var arg_it = os.args();
    // skip my own exe name
    _ = arg_it.skip();
    while (arg_it.next(allocator)) |err_or_arg| {
        const arg = %return err_or_arg;
        defer allocator.free(arg);
        // use the arg...
    }
}

Yikes, a bit cumbersome. I added a higher level API. Now you can call std.os.argsAlloc and get a %[]const []u8, and you just have to call std.os.argsFree when you're done with it.

pub fn main() -> %void {
    const allocator = std.heap.c_allocator;

    const args = %return os.argsAlloc(allocator);
    defer os.argsFree(allocator, args);

    var arg_i: usize = 1;
    while (arg_i < args.len) : (arg_i += 1) {
        const arg = args[arg_i];
        // do something with arg...
    }
}

Better! Single point of failure.

For now this uses the other API under the hood, but it could be reimplemented with the same API to do a single allocation.

I added a new kind of test to make sure command line argument parsing works.

Automatic C-to-Zig translation

• Translation now understands enum tag types.
• I refactored the C macro parsing, and made it understand pointer casting. Now some kinds of int-to-ptr code in .h files for embedded programming just work:

#define NRF_GPIO ((NRF_GPIO_Type *) NRF_GPIO_BASE)

Zig now understands this C macro.

std.mem

• add aligned functions to Allocator interface
• mem.Allocator initializes bytes to undefined. This does nothing in ReleaseFast mode. In Debug and ReleaseSafe modes, it initializes bytes to 0xaa which helps catch memory errors.
• add mem.FixedBufferAllocator

std.os.ChildProcess

pub fn exec(self: &Builder, argv: []const []const u8) -> []u8 {
    const max_output_size = 100 * 1024;
    const result = os.ChildProcess.exec(self.allocator, argv, null, null, max_output_size) %% |err| {
        std.debug.panic("Unable to spawn {}: {}", argv[0], @errorName(err));
    };
    switch (result.term) {
        os.ChildProcess.Term.Exited => |code| {
            if (code != 0) {
                warn("The following command exited with error code {}:\n", code);
                printCmd(null, argv);
                warn("stderr:{}\n", result.stderr);
                std.debug.panic("command failed");
            }
            return result.stdout;
        },
        else => {
            warn("The following command terminated unexpectedly:\n");
            printCmd(null, argv);
            warn("stderr:{}\n", result.stderr);
            std.debug.panic("command failed");
        },
    }
}

std.sort

Hejsil pointed out that the quicksort implementation in the standard library failed a simple test case.

There was another problem with the implementation of sort in the standard library, which is that it used O(n) stack space via recursion. This is fundamentally insecure, especially if you consider that the length of an array you might want to sort could be user input. It prevents #157 from working as well.

I had a look at Wikipedia's Comparison of Sorting Algorithms and only 1 sorting algorithm checked all the boxes:

• Best case O(n) complexity (adaptive sort)
• Average case O(n * log(n)) complexity
• Worst case O(n * log(n)) complexity
• O(1) memory
• Stable sort

And that algorithm is Block sort.

I found a high quality implementation of block sort in C, which is licensed under the public domain.

I ported the code from C to Zig, integrated it into the standard library, and it passed all tests first try. Amazing.

Surely, I thought, there must be some edge case. So I created a simple fuzz tester:

test "sort fuzz testing" {
    var rng = std.rand.Rand.init(0x12345678);
    const test_case_count = 10;
    var i: usize = 0;
    while (i < test_case_count) : (i += 1) {
        fuzzTest(&rng);
    }
}

var fixed_buffer_mem: [100 * 1024]u8 = undefined;

fn fuzzTest(rng: &std.rand.Rand) {
    const array_size = rng.range(usize, 0, 1000);
    var fixed_allocator = mem.FixedBufferAllocator.init(fixed_buffer_mem[0..]);
    var array = %%fixed_allocator.allocator.alloc(IdAndValue, array_size);
    // populate with random data
    for (array) |*item, index| {
        item.id = index;
        item.value = rng.range(i32, 0, 100);
    }
    sort(IdAndValue, array, cmpByValue);

    var index: usize = 1;
    while (index < array.len) : (index += 1) {
        if (array[index].value == array[index - 1].value) {
            assert(array[index].id > array[index - 1].id);
        } else {
            assert(array[index].value > array[index - 1].value);
        }
    }
}

This test passed as well. And so I think this problem is solved.

@export

There is now an @export builtin function which can be used in a comptime block to conditionally export a function:

const builtin = @import("builtin");

comptime {
    const strong_linkage = builtin.GlobalLinkage.Strong;
    if (builtin.link_libc) {
        @export("main", main, strong_linkage);
    } else if (builtin.os == builtin.Os.windows) {
        @export("WinMainCRTStartup", WinMainCRTStartup, strong_linkage);
    } else {
        @export("_start", _start, strong_linkage);
    }
}

It can also be used to create aliases:

const builtin = @import("builtin");
const is_test = builtin.is_test;

comptime {
    const linkage = if (is_test) builtin.GlobalLinkage.Internal else builtin.GlobalLinkage.Weak;
    const strong_linkage = if (is_test) builtin.GlobalLinkage.Internal else builtin.GlobalLinkage.Strong;

    @export("__letf2", @import("comparetf2.zig").__letf2, linkage);
    @export("__getf2", @import("comparetf2.zig").__getf2, linkage);

    if (!is_test) {
        // only create these aliases when not testing
        @export("__cmptf2", @import("comparetf2.zig").__letf2, linkage);
        @export("__eqtf2", @import("comparetf2.zig").__letf2, linkage);
        @export("__lttf2", @import("comparetf2.zig").__letf2, linkage);
        @export("__netf2", @import("comparetf2.zig").__letf2, linkage);
        @export("__gttf2", @import("comparetf2.zig").__getf2, linkage);
    }
}

Previous export syntax is still allowed. See #462 and #420.

Labeled loops, blocks, break, and continue, and R.I.P. goto

We used to have labels and goto like this:

export fn entry() {
    label:
    goto label;
}

Now this does not work, because goto is gone.

test.zig:2:10: error: expected token ';', found ':'
    label:
         ^

There are a few reasons to use goto, but all of the use cases are better served with other zig control flow features:

• cleanup pattern. Use defer and %defer instead.
• goto backward
• goto forward

goto backward

export fn entry() {
    start_over:

    while (some_condition) {
        // do something...
        goto start_over;
    }
}

Instead, use a loop!

export fn entry() {
    outer: while (true) {

        while (some_condition) {
            // do something...
            continue :outer;
        }

        break;
    }
}

goto forward

pub fn findSection(elf: &Elf, name: []const u8) -> %?&SectionHeader {
    var file_stream = io.FileInStream.init(elf.in_file);
    const in = &file_stream.stream;

    section_loop: for (elf.section_headers) |*elf_section| {
        if (elf_section.sh_type == SHT_NULL) continue;

        const name_offset = elf.string_section.offset + elf_section.name;
        %return elf.in_file.seekTo(name_offset);

        for (name) |expected_c| {
            const target_c = %return in.readByte();
            if (target_c == 0 or expected_c != target_c) goto next_section;
        }

        {
            const null_byte = %return in.readByte();
            if (null_byte == 0) return elf_section;
        }
next_section:
    }

    return null;
}

Looks like the use case is breaking out of an outer loop:

pub fn findSection(elf: &Elf, name: []const u8) -> %?&SectionHeader {
    var file_stream = io.FileInStream.init(elf.in_file);
    const in = &file_stream.stream;

    section_loop: for (elf.section_headers) |*elf_section| {
        if (elf_section.sh_type == SHT_NULL) continue;

        const name_offset = elf.string_section.offset + elf_section.name;
        %return elf.in_file.seekTo(name_offset);

        for (name) |expected_c| {
            const target_c = %return in.readByte();
            if (target_c == 0 or expected_c != target_c) continue :section_loop;
        }

        {
            const null_byte = %return in.readByte();
            if (null_byte == 0) return elf_section;
        }
    }

    return null;
}

You can also break out of arbitrary blocks:

export fn entry() {
    outer: {

        while (some_condition) {
            // do something...
            break :outer;
        }
    }
}

This can be used to return a value from a block in the same way you can return a value from a function:

export fn entry() {
    const value = init: {
        for (slice) |item| {
            if (item > 100)
                break :init item;
        }
        break :init 0;
    };
}

Omitting a semicolon no longer causes the value to be returned by the block. Instead you must use explicit block labels to return a value from a block. I'm considering a keyword such as result which defaults to the current block.

Removal of goto caused a regression in C-to-Zig translation: Switch statements no longer can be translated. However this code will be resurrected soon using labeled loops and labeled break instead of goto.

See #346, #630, and #629.

New IR pass iteration strategy

Before:

• IR basic blocks are in arbitrary order
• when doing an IR pass, when a block is encountered, code must look at all the instructions in the old basic block, determine what blocks are referenced, and queue up those old basic blocks first.
• This had a bug

while (cond) {
    if (false) { }
    break;
}

Pretty crazy right? Something as simple as this would crash the compiler.

Now:

• IR basic blocks are required to be in an order that guarantees they will be referenced by a branch, before any instructions within are referenced. ir pass1 is updated to meet this constraint.
• hen doing an IR pass, we iterate over old basic blocks in the order they appear. Blocks which have not been referenced are discarded.
• fter the pass is complete, we must iterate again to look for old basic blocks which now point to incomplete new basic blocks, due to comptime code generation.
• his last part can probably be optimized - most of the time we don't need to iterate over the basic block again.

This improvement deletes a lot of messy code:

 5 files changed, 288 insertions(+), 1243 deletions(-) 

And it also fixes comptime branches not being respected sometimes:

export fn entry() {
    while (false) {
        @compileError("bad");
    }
}

Before, this would cause a compile error. Now the while loop respects the implicit compile-time.

See #667.

Bug Fixes

• fix const and volatile qualifiers being dropped sometimes. in the expression &const a.b, the const (and/or volatile) qualifiers would be incorrectly dropped. See #655.
• fix compiler crash in a nullable if after an if in a switch prong of a switch with 2 prongs in an else. See #656.
• fix assert when wrapping zero bit type in nullable. See #659.
• fix crash when implicitly casting array of len 0 to slice. See #660.
• fix endianness of sub-byte integer fields in packed structs. In the future packed structs will require specifying endianness. See #307.
• fix std.os.path.resolve when the drive is missing.
• fix automatically C-translated functions not having debug information.
• fix crash when passing union enum with sub-byte field to const slice parameter. See #664.

Miscellaneous changes

• Rename builtin.is_big_endian to builtin.endian. This is in preparation for having endianness be a pointer property, which is related to packed structs. See #307.
• Tested Zig with LLVM debug mode and fixed some bugs that were causing LLVM assertions.
• Add @noInlineCall. See #640. This fixes a crash in --release-safe and --release-fast modes where the optimizer inlines everything into _start and clobbers the command line argument data. If we were able to verify that the user's code never reads command line args, we could leave off this "no inline" attribute. This might call for a patch to LLVM. It seems like inlining into a naked function should correctly bump the stack pointer.
• add i29 and u29 primitive types. u29 is the type of alignment, so it makes sense to be a primitive. probably in the future we'll make any i or u followed by digits into a primitive.
• add implicit cast from enum tag type of union to const ptr to the union. closes #654
• ELF stack traces support DW_AT_ranges, so sometimes when you would see "???" you now get a useful stack trace instead.
• add std.sort.min and std.sort.max functions
• std.fmt.bufPrint returns a possible error.BufferTooSmall instead of asserting that the buffer is large enough.
• Remove unnecessary inline calls in std.math.
• zig build now has a --search-prefix option. Any number of search prefixes can be specified.
• add some utf8 parsing utilities to the standard library.

Thank you contributors!

• MIURA Masahiro fixed the color of compiler messages for light-themed terminals. (See #644)
• Peter Rönnquist added format for floating point numbers. {.x} where x is the number of decimals. (See #668)

Thank you financial supporters!

Special thanks to those who donate monthly:

• Lauren Chavis
• Andrea Orru
• Adrian Sinclair
• David Joseph
• jeff kelley
• Hasan Abdul-Rahman
• Wesley Kelley
• Jordan Torbiak
• Richard Ohnemus
• Martin Schwaighofer
• Matthew
• Mirek Rusin
• Brendon Scheinman
• Pyry Kontio
• Thomas Ballinger
• Peter Ronnquist
• Robert Paul Herman
• Audun Wilhelmsen
• Marko Mikulicic
• Anthony J. Benik
• Caius
• Tyler Philbrick
• Jeremy Larkin
• Rasmus Rønn Nielsen

A Better Way to Implement Bit-Fields

One of the main use cases for the Zig Programming Language is operating system development and embedded development. So, what better way to make sure the language is suitable than to work on a project in this field?

This is why I am creating a 4-player arcade game that runs directly on the Raspberry Pi 3 hardware. The project has only just begun, but already it is revealing important fixes and features in Zig, and I am updating the compiler to incorporate these things as I work.

The next thing I'm working on is adding a USB controller driver, so that I can use a gamepad to test the game. I noticed when looking at some reference code that there are large structs full of bit-fields, and these turn out to be extremely convenient:

extern volatile struct CoreGlobalRegs {
	volatile struct {
		volatile bool sesreqscs : 1;
		volatile bool sesreq : 1;
		volatile bool vbvalidoven:1;
		volatile bool vbvalidovval:1;
		volatile bool avalidoven:1;
		volatile bool avalidovval:1;
		volatile bool bvalidoven:1;
		volatile bool bvalidovval:1;
		volatile bool hstnegscs:1;
		volatile bool hnpreq:1;
		volatile bool HostSetHnpEnable : 1;
		volatile bool devhnpen:1;
		volatile unsigned _reserved12_15:4;
		volatile bool conidsts:1;
		volatile unsigned dbnctime:1;
		volatile bool ASessionValid : 1;
		volatile bool BSessionValid : 1;
		volatile unsigned OtgVersion : 1;
		volatile unsigned _reserved21:1;
		volatile unsigned multvalidbc:5;
		volatile bool chirpen:1;
		volatile unsigned _reserved28_31:4;
	} __attribute__ ((__packed__)) OtgControl; // +0x0
	volatile struct {
		volatile unsigned _reserved0_1 : 2; // @0
		volatile bool SessionEndDetected : 1; // @2
		volatile unsigned _reserved3_7 : 5; // @3
		volatile bool SessionRequestSuccessStatusChange : 1; // @8
		volatile bool HostNegotiationSuccessStatusChange : 1; // @9
		volatile unsigned _reserved10_16 : 7; // @10
		volatile bool HostNegotiationDetected : 1; // @17
		volatile bool ADeviceTimeoutChange : 1; // @18
		volatile bool DebounceDone : 1; // @19
		volatile unsigned _reserved20_31 : 12; // @20
	} __attribute__ ((__packed__)) OtgInterrupt; // +0x4
// ...

I'll stop there, but this struct goes on for pages and pages. You can see that in C, bit-fields have special syntax. The fields have a type like normal, and then an extra colon and a number of bits.

Bit-fields in C have acquired a poor reputation in the programming community, for a few reasons:

• Poor performance due to loading more bytes than necessary
• Largely implementation-defined behavior
• Platform dependence
• Linus Torvalds: bitfields make it harder to work with combinations of flags

Many experienced developers have given up on the usefulness of bit-fields and prefer to manually wrangle their binary data.

Some of these problems I can address in Zig, by loading the minimal number of bytes necessary, defining how bit-fields are laid out in memory, and ensuring that bit-fields work the same or at least predictably on all platforms. Some of the problems are inherently tricky, such as the question of how to deal with endianness when a bit-field has more than 8 bits and crosses a byte boundary.

With these things in mind, I set out to implement bit-fields in Zig, and now I am pleased to announce that work is complete. Zig has bit-fields, and it required no syntax additions.

Zig takes advantage of the fact that types are first-class values at compile-time. There is a built-in function which returns an integer type with the specified signness and bit count, and it can be used to get access to uncommon integer types like this:

const i13 = @intType(true, 13);

These uncommon integer types work like normal integer types. Arithmetic, casting, and overflow are generalized to work with any integer type.

These integer types are one component to the way bit-fields are implemented in Zig. The other component takes advantage of the fact that Zig has 3 different kinds of struct layouts:

• Default - compiler may re-arrange fields and insert padding
• extern - compatible with the target environment C ABI
• packed - fields in exact order specified, no padding

In a packed struct, the programmer is directly in charge of the memory layout of the struct. Fields with integer types take up exactly as many bits as the integer type specifies. If a field greater than 8 bits is byte-aligned, it is represented in memory with the endianness of the host. If the field is not byte-aligned, it is represented in memory in big-endian. Boolean values are represented as 1 bit in packed structs.

So that's it. To make a bit-field, you have a packed struct with fields that are integers with the bit sizes you want.

For illustration, here is the above code translated into Zig:

const u1 = @intType(false, 1);
const u12 = @intType(false, 12);

const CoreGlobalRegs = packed struct {
    OtgControl: struct {
        sesreqscs: bool,
        sesreq: bool,
        vbvalidoven: bool,
        vbvalidovval: bool,
        avalidoven: bool,
        avalidovval: bool,
        bvalidoven: bool,
        bvalidovval: bool,
        hstnegscs: bool,
        hnpreq: bool,
        HostSetHnpEnable: bool,
        devhnpen: bool,
        _reserved12_15: u4,
        conidsts: bool,
        dbnctime: u1,
        ASessionValid: bool,
        BSessionValid: bool,
        OtgVersion: u1,
        _reserved21: u1,
        multvalidbc: u5,
        chirpen: bool,
        _reserved28_31: u4,
    }, // +0x0
    OtgInterrupt: packed struct {
        _reserved0_1: u2, // @0
        SessionEndDetected: bool, // @2
        _reserved3_7: u5, // @3
        SessionRequestSuccessStatusChange: bool, // @8
        HostNegotiationSuccessStatusChange: bool, // @9
        _reserved10_16: u7, // @10
        HostNegotiationDetected: bool, // @17
        ADeviceTimeoutChange: bool, // @18
        DebounceDone: bool, // @19
        _reserved20_31: u12, // @20
    }, // +0x4
// ...

By the way it's nice to know that the USB protocol has a bit to indicate "a valid oven". I wonder when that is used.

You may notice that we do a bit of setup before declaring the struct by creating these integer types. Currently, only integer types of size 8, 16, 32, and 64 are provided by the compiler globally. But perhaps more integer types can be provided as primitive types, or perhaps there will be a standard library file to import and get more integer types.

You may also notice that the volatile keyword is gone. This is a different issue, but Zig handles volatile at the pointer level. So instead of putting the keyword on every field, you make sure the pointer to the whole struct is volatile, and then any loads and stores done via the pointer become volatile, and any pointers derived from the volatile pointer are also volatile.

One point of comparison with C bit-fields. Take a look at this code:

struct Foo {
    unsigned a : 3;
    unsigned b : 3;
    unsigned c : 2;
} __attribute__ ((__packed__));

struct Foo foo = {1, 2, 3};

unsigned f(void) {
    unsigned *ptr = &foo.b;
    return *ptr;
}

Here we try to take the address of a bit-field, and clang doesn't like that idea so much:

test.c:10:12: error: address of bit-field requested
    return &foo.b;
           ^~~~~~

Meanwhile, in Zig, this works just fine:

const Foo = packed struct {
    a: u3,
    b: u3,
    c: u2,
};

var foo = Foo {
    .a = 1,
    .b = 2,
    .c = 3,
};

fn f() u3 {
    const ptr = &foo.b;
    return ptr.*;
}

The bit offset and length is carried in the type of the pointer, so you get an error if you try to pass such a pointer to a function expecting a normal, byte-aligned value:

const BitField = packed struct {
    a: u3,
    b: u3,
    c: u2,
};

fn foo(bit_field: *const BitField) u3 {
    return bar(&bit_field.b);
}

fn bar(x: *const u3) u3 {
    return x.*;
}

In this case the compiler catches the mistake:

./test.zig:8:26: error: expected type '*const u3', found '*:3:6 const u3'
    return bar(&bit_field.b);
                         ^

There are bound to be some edge cases and bugs as I polish this feature, but I am pleased that it turned out to integrate so cleanly into Zig's minimal design.

Zig: Already More Knowable Than C

There is a nifty article created back in 2015 that made its way onto Hacker News today: I Do Not Know C.

The author creates a delightful set of source code samples in which the reader is intended to determine the correctness, and if correct, predict the output of the provided code. If you have not done the exercises, I encourage you to take a moment to do so.

What follows are some of the examples translated into Zig for comparison. The numbers correspond to the numbers from the original post linked above, but the C code is embedded here for comparison.

1. Declaring the same variable twice

int i;
int i = 10;

In Zig:

var i = undefined;
var i = 10;

Output:

./test.zig:2:1: error: redefinition of 'i'
var i = 10;
^
./test.zig:1:1: note: previous definition is here
var i = undefined;
^

2. Null pointer

extern void bar(void);
void foo(int *x) {
    int y = *x;  /* (1) */
    if (!x) {    /* (2) */
        return;  /* (3) */
    }
    bar();
}

When this example is translated to Zig, you have to explicitly decide if the pointer is nullable. For example, if you used a bare pointer in Zig:

extern fn bar();

export fn foo(x: &c_int) {
    var y = *x;  // (1)
    if (x == null) {     // (2)
        return;    // (3)
    }
    bar();
}

Then it wouldn't even compile:

./test.zig:7:11: error: operator not allowed for type '?&c_int'
    if (x == null) {     // (2)
          ^

I think this error message can be improved, but even so it prevents a possible null related bug here.

If the code author makes the pointer nullable, then the natural way to port the C code to Zig would be:

extern fn bar();

export fn foo(x: ?&c_int) {
    var y = x ?? return;  // (1), (2), (3)
    bar();
}

This compiles to:

0000000000000000 <foo>:
   0:	48 85 ff             	test   %rdi,%rdi
   3:	74 05                	je     a <foo+0xa>
   5:	e9 00 00 00 00       	jmpq   a <foo+0xa>
   a:	c3                   	retq   

This does a null check, returns if null, and otherwise calls bar.

Perhaps a more faithful way to port the C code to Zig would be this:

extern fn bar();

fn foo(x: ?&c_int) {
    var y = ??x;  // (1)
    if (x == null) {     // (2)
        return;    // (3)
    }
    bar();
}

pub fn main() -> %void {
    foo(null);
}

The ?? operator unwraps the nullable value. It asserts that the value is not null, and returns the value. If the value is null then the behavior is undefined, just like in C.

However, in Zig, undefined behavior causes a crash in debug mode:

$ ./test
attempt to unwrap null
/home/andy/dev/zig/build/lib/zig/std/special/zigrt.zig:16:35: 0x0000000000203395 in ??? (test)
        @import("std").debug.panic("{}", message_ptr[0..message_len]);
                                  ^
/home/andy/tmp/test.zig:4:13: 0x000000000020a61d in ??? (test)
    var y = ??x;  // (1)
            ^
/home/andy/tmp/test.zig:12:8: 0x00000000002046fd in ??? (test)
    foo(null);
       ^
/home/andy/dev/zig/build/lib/zig/std/special/bootstrap.zig:60:21: 0x0000000000203697 in ??? (test)
    return root.main();
                    ^
/home/andy/dev/zig/build/lib/zig/std/special/bootstrap.zig:47:13: 0x0000000000203420 in ??? (test)
    callMain(argc, argv, envp) %% std.os.posix.exit(1);
            ^
/home/andy/dev/zig/build/lib/zig/std/special/bootstrap.zig:34:25: 0x0000000000203290 in ??? (test)
    posixCallMainAndExit()
                        ^
Aborted

This is a half-finished traceback implementation (lots of TODO items to complete before the 0.1.0 milestone), but the point is that Zig detected the undefined behavior and aborted.

In release mode, this example invokes undefined behavior just like in C. To avoid this, programmers are expected to choose one of these options:

• Test code sufficiently in debug mode to catch undefined behavior abuse.
• Utilize the safe-release option which includes the runtime undefined behavior safety checks.

5. strlen

int my_strlen(const char *x) {
    int res = 0;
    while(*x) {
        res++;
        x++;
    }
    return res;
}

In Zig, pointers generally point to single objects, while slices are used to refer to ranges of memory. So in practice you wouldn't need a strlen function, you would use some_bytes.len. But we can port this code over anyway:

export fn my_strlen(x: &const u8) -> c_int {
    var res: c_int = 0;
    while (x[res] != 0) {
        res += 1;
    }
    return res;
}

Here we must use pointer indexing because Zig does not support direct pointer arithmetic.

The compiler catches this problem:

./test.zig:3:14: error: expected type 'usize', found 'c_int'
    while (x[res] != 0) {
             ^

6. Print string of bytes backwards

#include <stdio.h>
#include <string.h>
int main() {
    const char *str = "hello";
    size_t length = strlen(str);
    size_t i;
    for(i = length - 1; i >= 0; i--) {
        putchar(str[i]);
    }
    putchar('\n');
    return 0;
}

Ported to Zig:

const c = @cImport({
    @cInclude("stdio.h");
    @cInclude("string.h");
});

export fn main(argc: c_int, argv: &&u8) -> c_int {
    const str = c"hello";
    const length: c.size_t = c.strlen(str);
    var i: c.size_t = length - 1;
    while (i >= 0) : (i -= 1) {
        _ = c.putchar(str[i]);
    }
    _ = c.putchar('\n');
    return 0;
}

It compiles fine but produces this output when run:

integer overflow
test.zig:10:25: 0x000000000020346d in ??? (test)
    while (i >= 0) : (i -= 1) {
                        ^
Aborted

8. Weirdo syntax

#include <stdio.h>
int main() {
    int array[] = { 0, 1, 2 };
    printf("%d %d %d\n", 10, (5, array[1, 2]), 10);
}

There is no way to express this code in Zig. Good riddance.

9. Unsigned overflow

unsigned int add(unsigned int a, unsigned int b) {
    return a + b;
}

In Zig:

const io = @import("std").io;

export fn add(a: c_uint, b: c_uint) -> c_uint {
    return a + b;
}

pub fn main() -> %void {
    %%io.stdout.printf("{}\n", add(@maxValue(c_uint), 1));
}

Output:

$ ./test
integer overflow
test.zig:4:14: 0x00000000002032b4 in ??? (test)
    return a + b;
             ^
test.zig:8:35: 0x0000000000204747 in ??? (test)
    %%io.stdout.printf("{}\n", add(@maxValue(c_uint), 1));
                                  ^
lib/zig/std/special/bootstrap.zig:60:21: 0x00000000002036d7 in ??? (test)
    return root.main();
                    ^
lib/zig/std/special/bootstrap.zig:47:13: 0x0000000000203460 in ??? (test)
    callMain(argc, argv, envp) %% std.os.posix.exit(1);
            ^
lib/zig/std/special/bootstrap.zig:34:25: 0x00000000002032d0 in ??? (test)
    posixCallMainAndExit()
                        ^
Aborted

The + operator asserts that there will be no overflow. If you want twos complement wraparound behavior, that is possible with the +% operator instead:

export fn add(a: c_uint, b: c_uint) -> c_uint {
    return a +% b;
}

Now the output is:

$ ./test 
0

10. Signed overflow

int add(int a, int b) {
    return a + b;
}

In C signed and unsigned integer overflow work differently. In Zig, they work the same. + asserts that no overflow occurs, and +% performs twos complement wraparound behavior.

11. Negation overflow

int neg(int a) {
    return -a;
}

By now you can probably predict how this works in Zig.

const io = @import("std").io;

export fn neg(a: c_int) -> c_int {
    return -a;
}

pub fn main() -> %void {
    %%io.stdout.printf("{}\n", neg(@minValue(c_int)));
}

Output:

$ ./test
integer overflow
test.zig:4:12: 0x00000000002032b0 in ??? (test)
    return -a;
           ^
test.zig:8:35: 0x0000000000204742 in ??? (test)
    %%io.stdout.printf("{}\n", neg(@minValue(c_int)));
                                  ^
lib/zig/std/special/bootstrap.zig:60:21: 0x00000000002036d7 in ??? (test)
    return root.main();
                    ^
lib/zig/std/special/bootstrap.zig:47:13: 0x0000000000203460 in ??? (test)
    callMain(argc, argv, envp) %% std.os.posix.exit(1);
            ^
lib/zig/std/special/bootstrap.zig:34:25: 0x00000000002032d0 in ??? (test)
    posixCallMainAndExit()
                        ^
Aborted

The -% wraparound variant of the negation operator works here too:

export fn neg(a: c_int) -> c_int {
    return -%a;
}

Output:

$ ./test 
-2147483648

12. Division overflow

int div(int a, int b) {
    assert(b != 0);
    return a / b;
}

Different operation, same deal.

const io = @import("std").io;

fn div(a: i32, b: i32) -> i32 {
    return a / b;
}

pub fn main() -> %void {
    %%io.stdout.printf("{}\n", div(@minValue(i32), -1));
}

First of all, Zig doesn't let us do this operation because it's unclear whether we want floored division or truncated division:

test.zig:4:14: error: division with 'i32' and 'i32': signed integers must use @divTrunc, @divFloor, or @divExact
    return a / b;
             ^

Some languages use truncation division (C) while others (Python) use floored division. Zig makes the programmer choose explicitly.

fn div(a: i32, b: i32) -> i32 {
    return @divTrunc(a, b);
}

Output:

$ ./test
integer overflow
test.zig:4:12: 0x000000000020a683 in ??? (test)
    return @divTrunc(a, b);
           ^
test.zig:8:35: 0x0000000000204707 in ??? (test)
    %%io.stdout.printf("{}\n", div(@minValue(i32), -1));
                                  ^
lib/zig/std/special/bootstrap.zig:60:21: 0x0000000000203697 in ??? (test)
    return root.main();
                    ^
lib/zig/std/special/bootstrap.zig:47:13: 0x0000000000203420 in ??? (test)
    callMain(argc, argv, envp) %% std.os.posix.exit(1);
            ^
lib/zig/std/special/bootstrap.zig:34:25: 0x0000000000203290 in ??? (test)
    posixCallMainAndExit()
                        ^
Aborted

Notably, if you execute the division operation at compile time, the overflow becomes a compile error (same for the other operations):

    %%io.stdout.printf("{}\n", comptime div(@minValue(i32), -1));

Output:

./test.zig:4:14: error: operation caused overflow
    return a / b;
             ^
./test.zig:8:44: note: called from here
    %%io.stdout.printf("{}\n", comptime div(@minValue(i32), -1));
                                           ^

Conclusion

Zig is on track to boot out C as the simple, straightforward way to write system code.

Zig Programming Language Blurs the Line Between Compile-Time and Run-Time

Zig places importance on the concept of whether an expression is known at compile-time. There are a few different places this concept is used, and these building blocks are used to keep the language small, readable, and powerful.

• Introducing the Compile-Time Concept
• Compile-time parameters
• Compile-time variables
• Compile-time expressions
• Generic Data Structures
• Case Study: printf in C, Rust, and Zig
• Conclusion

Introducing the Compile-Time Concept

Compile-Time Parameters

Compile-time parameters is how Zig implements generics. It is compile-time duck typing and it works mostly the same way that C++ template parameters work. Example:

fn max(comptime T: type, a: T, b: T) -> T {
    if (a > b) a else b
}
fn gimmeTheBiggerFloat(a: f32, b: f32) -> f32 {
    max(f32, a, b)
}
fn gimmeTheBiggerInteger(a: u64, b: u64) -> u64 {
    max(u64, a, b)
}

In Zig, types are first-class citizens. They can be assigned to variables, passed as parameters to functions, and returned from functions. However, they can only be used in expressions which are known at compile-time, which is why the parameter T in the above snippet must be marked with comptime.

A comptime parameter means that:

• At the callsite, the value must be known at compile-time, or it is a compile error.
• In the function definition, the value is known at compile-time.

For example, if we were to introduce another function to the above snippet:

fn max(comptime T: type, a: T, b: T) -> T {
    if (a > b) a else b
}
fn letsTryToPassARuntimeType(condition: bool) {
    const result = max(
        if (condition) f32 else u64,
        1234,
        5678);
}

Then we get this result from the compiler:

./test.zig:6:9: error: unable to evaluate constant expression
        if (condition) f32 else u64,
        ^

This is an error because the programmer attempted to pass a value only known at run-time to a function which expects a value known at compile-time.

Another way to get an error is if we pass a type that violates the type checker when the function is analyzed. This is what it means to have compile-time duck typing.

For example:

fn max(comptime T: type, a: T, b: T) -> T {
    if (a > b) a else b
}
fn letsTryToCompareBools(a: bool, b: bool) -> bool {
    max(bool, a, b)
}

The code produces this error message:

./test.zig:2:11: error: operator not allowed for type 'bool'
    if (a > b) a else b
          ^
./test.zig:5:8: note: called from here
    max(bool, a, b)
       ^

On the flip side, inside the function definition with the comptime parameter, the value is known at compile-time. This means that we actually could make this work for the bool type if we wanted to:

fn max(comptime T: type, a: T, b: T) -> T {
    if (T == bool) {
        return a or b;
    } else if (a > b) {
        return a;
    } else {
        return b;
    }
}
fn letsTryToCompareBools(a: bool, b: bool) -> bool {
    max(bool, a, b)
}

This works because Zig implicitly inlines if expressions when the condition is known at compile-time, and the compiler guarantees that it will skip analysis of the branch not taken.

This means that the actual function generated for max in this situation looks like this:

fn max(a: bool, b: bool) -> bool {
    return a or b;
}

All the code that dealt with compile-time known values is eliminated and we are left with only the necessary run-time code to accomplish the task.

This works the same way for switch expressions - they are implicitly inlined when the target expression is compile-time known.

Compile-Time Variables

In Zig, the programmer can label variables as comptime. This guarantees to the compiler that every load and store of the variable is performed at compile-time. Any violation of this results in a compile error.

This combined with the fact that we can inline loops allows us to write a function which is partially evaluated at compile-time and partially at run-time.

For example:

const CmdFn = struct {
    name: []const u8,
    func: fn(i32) -> i32,
};

const cmd_fns = []CmdFn{
    CmdFn {.name = "one", .func = one},
    CmdFn {.name = "two", .func = two},
    CmdFn {.name = "three", .func = three},
};
fn one(value: i32) -> i32 { value + 1 }
fn two(value: i32) -> i32 { value + 2 }
fn three(value: i32) -> i32 { value + 3 }

fn performFn(comptime prefix_char: u8, start_value: i32) -> i32 {
    var result: i32 = start_value;
    comptime var i = 0;
    inline while (i < cmd_fns.len) : (i += 1) {
        if (cmd_fns[i].name[0] == prefix_char) {
            result = cmd_fns[i].func(result);
        }
    }
    return result;
}

fn testPerformFn() {
    @setFnTest(this);

    assert(performFn('t', 1) == 6);
    assert(performFn('o', 0) == 1);
    assert(performFn('w', 99) == 99);
}

fn assert(ok: bool) {
    if (!ok) unreachable;
}

This example is a bit contrived, because the compile-time evaluation component is unnecessary; this code would work fine if it was all done at run-time. But it does end up generating different code. In this example, the function performFn is generated three different times, for the different values of prefix_char provided:

// From the line:
// assert(performFn('t', 1) == 6);
fn performFn(start_value: i32) -> i32 {
    var result: i32 = start_value;
    result = two(result);
    result = three(result);
    return result;
}

// From the line:
// assert(performFn('o', 0) == 1);
fn performFn(start_value: i32) -> i32 {
    var result: i32 = start_value;
    result = one(result);
    return result;
}

// From the line:
// assert(performFn('w', 99) == 99);
fn performFn(start_value: i32) -> i32 {
    var result: i32 = start_value;
    return result;
}

Note that this happens even in a debug build; in a release build these generated functions still pass through rigorous LLVM optimizations. The important thing to note, however, is not that this is a way to write more optimized code, but that it is a way to make sure that what should happen at compile-time, does happen at compile-time. This catches more errors and as demonstrated later in this article, allows expressiveness that in other languages requires using macros, generated code, or a preprocessor to accomplish.

Compile-Time Expressions

In Zig, it matters whether a given expression is known at compile-time or run-time. A programmer can use a comptime expression to guarantee that the expression will be evaluated at compile-time. If this cannot be accomplished, the compiler will emit an error. For example:

extern fn exit() -> unreachable;

fn foo() {
    comptime {
        exit();
    }
}

./test.zig:5:9: error: unable to evaluate constant expression
        exit();
        ^

It doesn't make sense that a program could call exit() (or any other external function) at compile-time, so this is a compile error. However, a comptime expression does much more than sometimes cause a compile error.

Within a comptime expression:

• All variables are comptime variables.
• All if, while, for, switch, and goto expressions are evaluated at compile-time, or emit a compile error if this is not possible.
• All function calls cause the compiler to interpret the function at compile-time, emitting a compile error if the function tries to do something that has global run-time side effects.

This means that a programmer can create a function which is called both at compile-time and run-time, with no modification to the function required.

Let's look at an example:

fn fibonacci(index: u32) -> u32 {
    if (index < 2) return index;
    return fibonacci(index - 1) + fibonacci(index - 2);
}

fn testFibonacci() {
    @setFnTest(this);

    // test fibonacci at run-time
    assert(fibonacci(7) == 13);

    // test fibonacci at compile-time
    comptime {
        assert(fibonacci(7) == 13);
    }
}

fn assert(ok: bool) {
    if (!ok) unreachable;
}

$ zig test test.zig
Test 1/1 testFibonacci...OK

Imagine if we had forgotten the base case of the recursive function and tried to run the tests:

fn fibonacci(index: u32) -> u32 {
    //if (index < 2) return index;
    return fibonacci(index - 1) + fibonacci(index - 2);
}

fn testFibonacci() {
    @setFnTest(this);

    comptime {
        assert(fibonacci(7) == 13);
    }
}

fn assert(ok: bool) {
    if (!ok) unreachable;
}

$ zig test test.zig
./test.zig:3:28: error: operation caused overflow
    return fibonacci(index - 1) + fibonacci(index - 2);
                           ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:14:25: note: called from here
        assert(fibonacci(7) == 13);
                        ^

The compiler produces an error which is a stack trace from trying to evaluate the function at compile-time.

Luckily, we used an unsigned integer, and so when we tried to subtract 1 from 0, it triggered undefined behavior, which is always a compile error if the compiler knows it happened. But what would have happened if we used a signed integer?

fn fibonacci(index: i32) -> i32 {
    //if (index < 2) return index;
    return fibonacci(index - 1) + fibonacci(index - 2);
}

fn testFibonacci() {
    @setFnTest(this);

    comptime {
        assert(fibonacci(7) == 13);
    }
}

fn assert(ok: bool) {
    if (!ok) unreachable;
}

./test.zig:3:21: error: evaluation exceeded 1000 backwards branches
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^

The compiler noticed that evaluating this function at compile-time took a long time, and thus emitted a compile error and gave up. If the programmer wants to increase the budget for compile-time computation, they can use a built-in function called @setEvalBranchQuota to change the default number 1000 to something else.

What if we fix the base case, but put the wrong value in the assert line?

comptime {
    assert(fibonacci(7) == 99999);
}

./test.zig:15:14: error: unable to evaluate constant expression
    if (!ok) unreachable;
             ^
./test.zig:10:15: note: called from here
        assert(fibonacci(7) == 99999);
              ^

What happened is Zig started interpreting the assert function with the parameter ok set to false. When the interpreter hit unreachable it emitted a compile error, because reaching unreachable code is undefined behavior, and undefined behavior causes a compile error if it is detected at compile-time.

In the global scope (outside of any function), all expressions are implicitly comptime expressions. This means that we can use functions to initialize complex static data. For example:

const first_25_primes = firstNPrimes(25);
const sum_of_first_25_primes = sum(first_25_primes);

fn firstNPrimes(comptime n: usize) -> [n]i32 {
    var prime_list: [n]i32 = undefined;
    var next_index: usize = 0;
    var test_number: i32 = 2;
    while (next_index < prime_list.len) : (test_number += 1) {
        var test_prime_index: usize = 0;
        var is_prime = true;
        while (test_prime_index < next_index) : (test_prime_index += 1) {
            if (test_number % prime_list[test_prime_index] == 0) {
                is_prime = false;
                break;
            }
        }
        if (is_prime) {
            prime_list[next_index] = test_number;
            next_index += 1;
        }
    }
    return prime_list;
}

fn sum(numbers: []i32) -> i32 {
    var result: i32 = 0;
    for (numbers) |x| {
        result += x;
    }
    return result;
}

When we compile this program, Zig generates the constants with the answer pre-computed. Here are the lines from the generated LLVM IR:

@0 = internal unnamed_addr constant [25 x i32] [i32 2, i32 3, i32 5, i32 7, i32 11, i32 13, i32 17, i32 19, i32 23, i32 29, i32 31, i32 37, i32 41, i32 43, i32 47, i32 53, i32 59, i32 61, i32 67, i32 71, i32 73, i32 79, i32 83, i32 89, i32 97]
@1 = internal unnamed_addr constant i32 1060

Note that we did not have to do anything special with the syntax of these functions. For example, we could call the sum function as is with a slice of numbers whose length and values were only known at run-time.

Generic Data Structures

Zig uses these capabilities to implement generic data structures without introducing any special-case syntax. If you followed along so far, you may already know how to create a generic data structure.

Here is an example of a generic List data structure, that we will instantiate with the type i32. Whereas in C++ or Rust we would refer to the instantiated type as List<i32>, in Zig we refer to the type as List(i32).

fn List(comptime T: type) -> type {
    struct {
        items: []T,
        len: usize,
    }
}

That's it. It's a function that returns an anonymous struct. For the purposes of error messages and debugging, Zig infers the name "List(i32)" from the function name and parameters invoked when creating the anonymous struct.

To keep the language small and uniform, all aggregate types in Zig are anonymous. To give a type a name, we assign it to a constant:

const Node = struct {
    next: &Node,
    name: []u8,
};

This works because all top level declarations are order-independent, and as long as there isn't an actual infinite regression, values can refer to themselves, directly or indirectly. In this case, Node refers to itself as a pointer, which is not actually an infinite regression, so it works fine.

Case Study: printf in C, Rust, and Zig

Putting all of this together, let's compare how printf works in C, Rust, and Zig.

Here's how printf work in C:

#include <stdio.h>

static const int a_number = 1234;
static const char * a_string = "foobar";

int main(int argc, char **argv) {
    fprintf(stderr, "here is a string: '%s' here is a number: %d\n", a_string, a_number);
    return 0;
}

here is a string: 'foobar' here is a number: 1234

What happens here is the printf implementation iterates over the format string at run-time, and when it encounters a format specifier such as %d it looks at the next argument which is passed in an architecture-specific way, interprets the argument as a type depending on the format specifier, and attempts to print it. If the types are incorrect or not enough arguments are passed, undefined behavior occurs - it may crash, print garbage data, or access invalid memory.

Luckily, the compiler defines an attribute that you can use like this:

__attribute__ ((format (printf, x, y)));

Where x and y are the 1-based indexes of the argument parameters that correspond to the format string and the first var args parameter, respectively.

This attribute adds type checking to the function it decorates, to prevent the above problems, and the printf function from stdio.h has this attribute on it, so these problems are solved.

But what if you want to invent your own format string syntax and have the compiler check it for you?

You can't.

That's how it works in C. It is hard-coded into the compiler. If you wanted to write your own format string printing code and have it checked by the compiler, you would have to use the preprocessor or metaprogramming - generate C code as output from some other code.

Zig is a programming language which is intended to replace C. We can do better than this.

Here's the equivalent program in Zig:

const io = @import("std").io;

const a_number: i32 = 1234;
const a_string = "foobar";

pub fn main(args: [][]u8) -> %void {
    %%io.stderr.printf("here is a string: '{}' here is a number: {}\n", a_string, a_number);
}

here is a string: 'foobar' here is a number: 1234

Let's crack open the implementation of this and see how it works:

/// Calls print and then flushes the buffer.
pub fn printf(self: &OutStream, comptime format: []const u8, args: ...) -> %void {
    const State = enum {
        Start,
        OpenBrace,
        CloseBrace,
    };

    comptime var start_index: usize = 0;
    comptime var state = State.Start;
    comptime var next_arg: usize = 0;

    inline for (format) |c, i| {
        switch (state) {
            State.Start => switch (c) {
                '{' => {
                    if (start_index < i) %return self.write(format[start_index...i]);
                    state = State.OpenBrace;
                },
                '}' => {
                    if (start_index < i) %return self.write(format[start_index...i]);
                    state = State.CloseBrace;
                },
                else => {},
            },
            State.OpenBrace => switch (c) {
                '{' => {
                    state = State.Start;
                    start_index = i;
                },
                '}' => {
                    %return self.printValue(args[next_arg]);
                    next_arg += 1;
                    state = State.Start;
                    start_index = i + 1;
                },
                else => @compileError("Unknown format character: " ++ c),
            },
            State.CloseBrace => switch (c) {
                '}' => {
                    state = State.Start;
                    start_index = i;
                },
                else => @compileError("Single '}' encountered in format string"),
            },
        }
    }
    comptime {
        if (args.len != next_arg) {
            @compileError("Unused arguments");
        }
        if (state != State.Start) {
            @compileError("Incomplete format string: " ++ format);
        }
    }
    if (start_index < format.len) {
        %return self.write(format[start_index...format.len]);
    }
    %return self.flush();
}

This is a proof of concept implementation; it will gain more formatting capabilities before Zig reaches its first release.

Note that this is not hard-coded into the Zig compiler; this userland code in the standard library.

When this function is analyzed from our example code above, Zig partially evaluates the function and emits a function that actually looks like this:

pub fn printf(self: &OutStream, arg0: i32, arg1: []const u8) -> %void {
    %return self.write("here is a string: '");
    %return self.printValue(arg0);
    %return self.write("' here is a number: ");
    %return self.printValue(arg1);
    %return self.write("\n");
    %return self.flush();
}

printValue is a function that takes a parameter of any type, and does different things depending on the type:

pub fn printValue(self: &OutStream, value: var) -> %void {
    const T = @typeOf(value);
    if (@isInteger(T)) {
        return self.printInt(T, value);
    } else if (@isFloat(T)) {
        return self.printFloat(T, value);
    } else if (@canImplicitCast([]const u8, value)) {
        const casted_value = ([]const u8)(value);
        return self.write(casted_value);
    } else {
        @compileError("Unable to print type '" ++ @typeName(T) ++ "'");
    }
}

And now, what happens if we give too many arguments to printf?

%%io.stdout.printf("here is a string: '{}' here is a number: {}\n",
        a_string, a_number, a_number);

.../std/io.zig:147:17: error: Unused arguments
                @compileError("Unused arguments");
                ^
./test.zig:7:23: note: called from here
    %%io.stdout.printf("here is a number: {} and here is a string: {}\n",
                      ^

Zig gives programmers the tools needed to protect themselves against their own mistakes.

Let's take a look at how Rust handles this problem. Here's the equivalent program:

const A_NUMBER: i32 = 1234;
const A_STRING: &'static str = "foobar";

fn main() {
    print!("here is a string: '{}' here is a number: {}\n",
        A_STRING, A_NUMBER);
}

here is a string: 'foobar' here is a number: 1234

print!, as evidenced by the exclamation point, is a macro. Here is the definition:

#[macro_export]
#[stable(feature = "rust1", since = "1.0.0")]
#[allow_internal_unstable]
macro_rules! print {
    ($($arg:tt)*) => ($crate::io::_print(format_args!($($arg)*)));
}
#[stable(feature = "rust1", since = "1.0.0")]
#[macro_export]
macro_rules! format_args { ($fmt:expr, $($args:tt)*) => ({
/* compiler built-in */
}) }

Rust accomplishes the syntax that one would want from a var args print implementation, but it requires using a macro to do so.

Macros have some limitations. For example, in this case, if you move the format string to a global variable, the Rust example can no longer compile:

const A_NUMBER: i32 = 1234;
const A_STRING: &'static str = "foobar";
const FMT: &'static str = "here is a string: '{}' here is a number: {}\n";

fn main() {
    print!(FMT, A_STRING, A_NUMBER);
}

error: format argument must be a string literal.
 --> test.rs:6:12
  |
6 |     print!(FMT, A_STRING, A_NUMBER);
  |            ^^^

On the other hand, Zig doesn't care whether the format argument is a string literal, only that it is a compile-time known value that is implicitly castable to a []const u8:

const io = @import("std").io;

const a_number: i32 = 1234;
const a_string = "foobar";
const fmt = "here is a string: '{}' here is a number: {}\n";

pub fn main(args: [][]u8) -> %void {
    %%io.stderr.printf(fmt, a_string, a_number);
}

This works fine.

A macro is a reasonable solution to this problem, but it comes at the cost of readability. From Rust's own documentation:

The drawback is that macro-based code can be harder to understand, because fewer of the built-in rules apply. Like an ordinary function, a well-behaved macro can be used without understanding its implementation. However, it can be difficult to design a well-behaved macro! Additionally, compiler errors in macro code are harder to interpret, because they describe problems in the expanded code, not the source-level form that developers use.

These drawbacks make macros something of a "feature of last resort". That’s not to say that macros are bad; they are part of Rust because sometimes they’re needed for truly concise, well-abstracted code. Just keep this tradeoff in mind.

One of the goals of Zig is to avoid these drawbacks while still providing enough of the power that macros provide in order to make them unnecessary.

There is another thing I noticed, and I hope someone from the Rust community can correct me if I'm wrong, but it looks like Rust also special cased format_args! in the compiler by making it a built-in. If my understanding is correct, this would make Zig stand out as the only language of the three mentioned here which does not special case string formatting in the compiler and instead exposes enough power to accomplish this task in userland.

But more importantly, it does so without introducing another language on top of Zig, such as a macro language or a preprocessor language. It's Zig all the way down.

Conclusion

Thank you for following along and checking out what I've been working on lately.

As always, I welcome discussion, criticism, and users. Please keep in mind that this is alpha software; I am working toward a first beta release, but the project is not there yet.

Troubleshooting a Zig Regression with apitrace

The past three months I have spent rewriting Zig internals.

Previously, the compiler looked like this:

Source → Tokenize → Abstract Syntax Tree → Semantic Analysis → LLVM Codegen → Binary

Now the compiler looks like this:

Source → Tokenize → Abstract Syntax Tree → Intermediate Representation Code → Evaluation and Analysis → Intermediate Representation Code → LLVM Codegen → Binary

This was a significant amount of work:

92 files changed, 22307 insertions(+), 15357 deletions(-)

It took a while to get all 361 tests passing before I could merge back into master.

Part of my testing process is making sure this Tetris game continues to work. Here's a screenshot of it working, from master branch:

Unfortunately, after my changes to Zig, the game looked like this:

So, I ran both versions of the game with apitrace. This resulted in a handy diff:

Here, it looks like both programs are issuing the same OpenGL commands except for different values to glUniform4fv. Aha! Let's go see what's going on there.

After investigating this, it turned out that the glUniform4fv was simply for the piece color and since the game uses a random number for each piece, the two instances of the game started with different pieces.

So, I made a small change to the Random Number Generator code...

Before:

fn getRandomSeed() -> %u32 {
    var seed : u32 = undefined;
    const seed_bytes = (&u8)(&seed)[0...4];
    %return std.os.getRandomBytes(seed_bytes);
    return seed;
 }

After:

fn getRandomSeed() -> %u32 {
    return 4;
 }

After this change, the glUniform4fv commands were sending the same data. Therefore, the difference must be in the "data blob" parameters sent in initialization.

This led me to scrutinize this code:

const rect_2d_vertexes = [][3]c.GLfloat {
    []c.GLfloat{0.0, 0.0, 0.0},
    []c.GLfloat{0.0, 1.0, 0.0},
    []c.GLfloat{1.0, 0.0, 0.0},
    []c.GLfloat{1.0, 1.0, 0.0},
};
c.glGenBuffers(1, &sg.rect_2d_vertex_buffer);
c.glBindBuffer(c.GL_ARRAY_BUFFER, sg.rect_2d_vertex_buffer);
c.glBufferData(c.GL_ARRAY_BUFFER, 4 * 3 * @sizeOf(c.GLfloat), (&c_void)(&rect_2d_vertexes[0][0]), c.GL_STATIC_DRAW);

I discovered the problem was &rect_2d_vertexes[0][0]. The compiler noticed that rect_2d_vertexes was a compile-time constant and therefore generated the 2D array data structure as static data. It therefore evaluated &rect_2d_vertexes[0][0] as a compile-time known expression as well.

The problem was that each element in the rect_2d_vertexes referenced another array. The compile-time constant generation code emitted an independent array for the inner arrays, whereas we are expecting the pointer to point to a 2D array that contains all the data contiguously.

So I updated the data structure of constant arrays to refer to their parents, added a test case to cover the change, and now the tetris game works again. Huzzah!

Introduction to the Zig Programming Language

The past few months I took a break from working on Genesis Digital Audio Workstation to work, instead, on creating a new programming language.

I am nothing if not ambitious, and my goal is to create a new programming language that is more pragmatic than C. This is like to trying to be more evil than the devil himself.

So, in order, these are the priorities of Zig:

1. Pragmatic: At the end of the day, all that really matters is whether the language helped you do what you were trying to do better than any other language.
2. Optimal: The most natural way to write a program should result in top-of-the-line runtime performance, equivalent to or better than C. In places where performance is king, the optimal code should be clearly expressible.
3. Safe: Optimality may be sitting in the driver's seat, but safety is sitting in the passenger's seat, wearing its seatbelt, and asking nicely for the other passengers to do the same.
4. Readable: Zig prioritizes reading code over writing it. Avoid complicated syntax. Generally there should be a canonical way to do everything.

Table of Contents

1. Table of Contents
2. Design Decisions
   1. Widely Diverging Debug and Release Builds
   2. C ABI Compatibility
   3. Maybe Type Instead of Null Pointer
   4. The Error Type
   5. Alternate Standard Library
   6. Alternatives to the Preprocessor
3. Milestone: Tetris Implemented in Zig
4. Resources

Design Decisions

Widely Diverging Debug and Release Builds

Zig has the concept of a debug build vs a release build. Here is a comparison of priorities for debug mode vs release mode:

Note: Since this blog post, Zig has gained two more release modes:

• Release Safe
• Release Small

Complete C ABI Compatibility

Part of being pragmatic is recognizing C's existing success. Interop with C is crucial. Zig embraces C like the mean older brother who you are a little afraid of but you still want to like you and be your friend.

In Zig, functions look like this:

fn doSomething() {
    // ...
}

The compiler is free to inline this function, change its parameters, and otherwise do whatever it wants, since this is an internal function. However if you decide to export it:

export fn doSomething() {
    // ...
}

Now this function has the C ABI, and the name shows up in the symbol table verbatim. Likewise, you can declare an external function prototype:

extern fn puts(s: [*]const u8) c_int;

In Zig, like in C, you typically do not create a "wrapper" or "bindings" to a library, you just use it. But if you had to type out or generate all the extern function prototypes, this would be a binding. That is why Zig has the ability to parse .h files:

use @cImport({
    @cInclude("stdio.h");
});

This exposes all the symbols in stdio.h - including the #define statements - to the zig program, and then you can call puts or printf just like you would in C.

One of Zig's use cases is slowly transitioning a large C project to Zig. Zig can produce simple .o files for linking against other .o files, and it can also generate .h files based on what you export. So you could write part of your application in C and part in Zig, link all the .o files together and everything plays nicely with each other.

Optional Type Instead of Null Pointer

One area that Zig provides safety without compromising efficiency or readability is with the optional type.

The question mark symbolizes the optional type. You can convert a type to an optional type by putting a question mark in front of it, like this:

// normal integer
const normal_int: i32 = 1234;

// optional integer
const optional_int: ?i32 = 5678;

Now the variable optional_int could be an i32, or null.

Instead of integers, let's talk about pointers. Null references are the source of many runtime exceptions, and even stand accused of being the worst mistake of computer science.

Zig does not have them.

Instead, you can use an optional pointer. This secretly compiles down to a normal pointer, since we know we can use 0 as the null value for the maybe type. But the compiler can check your work and make sure you don't assign null to something that can't be null.

Typically the downside of not having null is that it makes the code more verbose to write. But, let's compare some equivalent C code and Zig code.

Task: call malloc, if the result is null, return null.

C code

// malloc prototype included for reference
void *malloc(size_t size);

struct Foo *do_a_thing(void) {
    char *ptr = malloc(1234);
    if (!ptr) return NULL;
    // ...
}

Zig code

// malloc prototype included for reference
extern fn malloc(size: size_t) ?[*]u8;

fn doAThing() ?*Foo {
    const ptr = malloc(1234) orelse return null;
    // ...
}

Here, Zig is at least as convenient, if not more, than C. And, the type of "ptr" is [*]u8 not ?[*]u8. The orelse operator unwrapped the maybe type and therefore ptr is guaranteed to be non-null everywhere it is used in the function.

The other form of checking against NULL you might see looks like this:

void do_a_thing(struct Foo *foo) {
    // do some stuff

    if (foo) {
        do_something_with_foo(foo);
    }

    // do some stuff
}

In Zig you can accomplish the same thing:

fn doAThing(optional_foo: ?*Foo) {
    // do some stuff

    if (optional_foo) |foo| {
      doSomethingWithFoo(foo);
    }

    // do some stuff
}

Once again, the notable thing here is that inside the if block, foo is no longer an optional pointer, it is a pointer, which cannot be null.

One benefit to this is that functions which take pointers as arguments can be annotated with the "nonnull" attribute - __attribute__((nonnull)) in GCC. The optimizer can sometimes make better decisions knowing that pointer arguments cannot be null.

Note: when this blog post was written, Zig did not distinguish between Single Item Pointers and Unknown Length Pointers. You can read about this in the documentation.

Errors

One of the distinguishing features of Zig is its exception handling strategy.

Zig introduces two primitive types:

• Error Sets
• Error Unions

An error set can be declared like this:

const FileOpenError = error {
  FileNotFound,
  OutOfMemory,
  UnexpectedToken,
};

An error set is a lot like an enum, except errors from different error sets which share a name, are defined to have the same numerical value. So each error name has a globally unique integer associated with it. The integer value 0 is reserved.

You can refer to these error values with field access syntax, such as FileOpenError.FileNotFound. There is syntactic sugar for creating an ad-hoc error set and referring to one of its errors: error.SomethingBroke. This is equivalent to error{SomethingBroke}.SomethingBroke.

In the same way that pointers cannot be null, an error set value is always an error.

const err = error.FileNotFound;

Most of the time you will not find yourself using an error set type. Instead, likely you will be using the error union type. Error unions are created with the binary operator !, with the error set on the left and any other type on the right: ErrorSet!OtherType.

Here is a function to parse a string into a 64-bit integer:

const ParseError = error {
    InvalidChar,
    Overflow,
};

pub fn parseU64(buf: []const u8, radix: u8) ParseError!u64 {
    var x: u64 = 0;

    for (buf) |c| {
        const digit = charToDigit(c);

        if (digit >= radix) {
            return error.InvalidChar;
        }

        // x *= radix
        if (@mulWithOverflow(u64, x, radix, &x)) {
            return error.Overflow;
        }

        // x += digit
        if (@addWithOverflow(u64, x, digit, &x)) {
            return error.Overflow;
        }
    }

    return x;
}

Notice the return type is ParseError!u64. This means that the function either returns an unsigned 64 bit integer, or one of the ParseError errors.

Within the function definition, you can see some return statements that return an error set value, and at the bottom a return statement that returns a u64. Both types implicitly cast to ParseError!u64.

Note: this blog post was written before Zig had the concept of Error Sets vs anyerror, and before Zig had Error Set Inference. Most functions in Zig can rely on error set inference, which would make the prototype of parseU64 look like this:

pub fn parseU64(buf: []const u8, radix: u8) !u64 {
    ...

What it looks like to use this function varies depending on what you're trying to do. One of the following:

• You want to provide a default value if it returned an error.
• If it returned an error then you want to return the same error.
• You know with complete certainty it will not return an error, so want to unconditionally unwrap it.
• You want to take a different action for each possible error.

If you want to provide a default value, you can use the catch expression:

fn doAThing(str: []u8) void {
    const number = parseU64(str, 10) catch 13;
    // ...
}

In this code, number will be equal to the successfully parsed string, or a default value of 13. The type of the right hand side of the catch expression must match the unwrapped error union type, or of type noreturn.

Let's say you wanted to return the error if you got one, otherwise continue with the function logic:

fn doAThing(str: []u8) !void {
    const number = parseU64(str, 10) catch |err| return err;
    // ...
}

There is a shortcut for this. The try expression:

fn doAThing(str: []u8) !void {
    const number = try parseU64(str, 10);
    // ...
}

try evaluates an error union expression. If it is an error, it returns from the current function with the same error. Otherwise, the expression results in the unwrapped value.

Maybe you know with complete certainty that an expression will never be an error. In this case you can do this:

const number = parseU64("1234", 10) catch unreachable;

Here we know for sure that "1234" will parse successfully. So we put the unreachable keyword on the right hand side. unreachable generates a panic in debug mode and undefined behavior in release mode. So, while we're debugging the application, if there was a surprise error here, the application would crash appropriately.

There is no syntactic shortcut for catch unreachable. This encourages programmers to think carefully before using it.

Finally, you may want to take a different action for every situation. For that, we have if combined with switch:

fn doAThing(str: []u8) {
    if (parseU64(str, 10)) |number| {
        doSomethingWithNumber(number);
    } else |err| switch (err) {
        error.Overflow => {
            // handle overflow...
        },
        // we promise that InvalidChar won't happen (or crash in debug mode if it does)
        error.InvalidChar => unreachable,
    }
}

The important thing to note here is that if parseU64 is modified to return a different set of errors, Zig will emit compile errors for handling impossible error codes, and for not handling possible error codes.

The other component to error handling is defer statements. In addition to an unconditional defer, Zig has errdefer, which evaluates the deferred expression on block exit path if and only if the function returned with an error from the block.

Example:

fn createFoo(param: i32) !Foo {
    const foo = try tryToAllocateFoo();
    // now we have allocated foo. we need to free it if the function fails.
    // but we want to return it if the function succeeds.
    errdefer deallocateFoo(foo);

    const tmp_buf = allocateTmpBuffer() orelse return error.OutOfMemory;
    // tmp_buf is truly a temporary resource, and we for sure want to clean it up
    // before this block leaves scope
    defer deallocateTmpBuffer(tmp_buf);

    if (param > 1337) return error.InvalidParam;

    // here the errdefer will not run since we're returning success from the function.
    // but the defer will run!
    return foo;
}

The neat thing about this is that you get robust error handling without the verbosity and cognitive overhead of trying to make sure every exit path is covered. The deallocation code is always directly following the allocation code.

A couple of other tidbits about error handling:

• These primitives give enough expressiveness that it's completely practical that failing to check for an error is a compile error. If you really want to ignore the error, you can use catch unreachable and get the added benefit of crashing in debug mode if your assumption was wrong.
• Since Zig understands error types, it can pre-weight branches in favor of errors not occuring. Just a small optimization benefit that is not available in other languages.
• There are no C++ style exceptions or stack unwinding or anything fancy like that. Zig simply makes it convenient to pass error codes around.

Alternate Standard Library

Part of the Zig project is providing an alternative to libc.

libc has a lot of useful stuff in it, but it also has cruft. Since we're starting fresh here, we can create a new API without some of the mistakes of the 70s still haunting us, and with our 20-20 hindsight.

Further, calling dynamically linked functions is slow. Zig's philosophy is that compiling against the standard library in source form is worth it. In C this would be called Link Time Optimization - where you generate Intermediate Representation instead of machine code and then do another compile step at link time. In Zig, we skip the middle man, and create a single compilation unit with everything in it, then run the optimizations.

So, you can choose to link against libc and take advantage of it, or you can choose to ignore it and use the Zig standard library instead. Note, however, that virtually every C library you depend on probably also depends on libc, which drags libc as a dependency into your project. Using libc is still a first class use case for Zig.

Alternatives to the Preprocessor

The C preprocessor is extremely powerful. Maybe a little too powerful.

The problem with the preprocessor is that it turns one language into two languages that don't know about each other.

Here are some examples of where the preprocessor messes things up:

• The compiler cannot catch even simple syntax errors in code that is excluded via #ifdef.
• IDEs cannot implement a function, variable, or field renaming feature that works correctly. Among other mistakes, it will miss renaming things that are in code excluded via #ifdef.
• Preprocessor defines do not show up in debug symbols by default.
• #include is the single biggest contributor to slow compile times in both C and C++.
• Preprocessor defines are problematic for bindings generators for other languages.

Regardless of the flaws, C programmers find ourselves using the preprocessor because it provides necessary features, such as conditional compilation, a constant that can be used for array sizes, and generics.

Zig plans to provide better alternatives to solve these problems. For example, the constant expression evaluator of Zig allows you to do this:

const array_len = 10 * 2 + 1;
const Foo = struct {
    array: [array_len]i32,
};

This is not an amazing concept, but it eliminates one use case for #define.

Next, conditional compilation. In Zig, compilation variables are available via @import("builtin").

The declarations available in this import evaluate to constant expressions. You can write normal code using these constants:

const builtin = @import("builtin");
fn doSomething() {
    if (builtin.mode == builtin.Mode.ReleaseFast) {
        // do the release behavior
    } else {
        // do the debug behavior
    }
}

This is guaranteed to leave out the if statement when the code is generated.

One use case for conditional compilation is demonstrated in libsoundio:

static const enum SoundIoBackend available_backends[] = {
#ifdef SOUNDIO_HAVE_JACK
    SoundIoBackendJack,
#endif
#ifdef SOUNDIO_HAVE_PULSEAUDIO
    SoundIoBackendPulseAudio,
#endif
#ifdef SOUNDIO_HAVE_ALSA
    SoundIoBackendAlsa,
#endif
#ifdef SOUNDIO_HAVE_COREAUDIO
    SoundIoBackendCoreAudio,
#endif
#ifdef SOUNDIO_HAVE_WASAPI
    SoundIoBackendWasapi,
#endif
    SoundIoBackendDummy,
};

Here, we want a statically sized array to have different contents depending on whether we have certain libraries present.

In Zig, it would look something like this:

const opts = @import("build_options");
const available_backends =
    (if (opts.have_jack)
        []SoundIoBackend{SoundIoBackend.Jack}
    else
        []SoundIoBackend{})
    ++
    (if (opts.have_pulse_audio)
        []SoundIoBackend{SoundIoBackend.PulseAudio}
    else
        []SoundIoBackend{})
    ++
    (if (opts.have_alsa)
        []SoundIoBackend{SoundIoBackend.Alsa}
    else
        []SoundIoBackend{})
    ++
    (if (opts.have_core_audio)
        []SoundIoBackend{SoundIoBackend.CoreAudio}
    else
        []SoundIoBackend{})
    ++
    (if (opts.have_wasapi)
        []SoundIoBackend{SoundIoBackend.Wasapi}
    else
        []SoundIoBackend{})
    ++
    []SoundIoBackend{SoundIoBackend.Dummy};

Here we take advantage of the compile-time array concatenation operator, ++. It's a bit more verbose than the C equivalent, but the important thing is that it's one language, not two.

Finally, generics. Zig implements generics by allowing programmers to mark parameters to functions as known at compile-time.

Milestone: Tetris Implemented in Zig

This past week I achieved a fun milestone: a fully playable Tetris clone implemented in Zig, with the help of libc, GLFW, and libpng.

If you're using Linux on the x86_64 architecture, which is currently the only supported target, you could download a Zig build and then build this Tetris game.

Otherwise, here's a video of me demoing it:

Resources

If you are interested in the language, feel free to participate.

• Home Page
• Source code and issue tracker: https://github.com/ziglang/zig
• IRC channel: #zig on Freenode
• Financial Support: Become a sponsor
• Official Documentation

A few months ago I published My Quest to Build the Ultimate Music Player, where I described some of the trials and tribulations that led to Groove Basin, an open-source music player server that I've been building off and on for almost 4 years.

It ships with a web-based client, which looks like this:

You can also tinker with the live demo version.

If you install this on a Raspberry Pi, you can attach speakers to it and use it as a music player which you can control remotely, and you can remotely listen to your music by pressing the "stream" button in the browser.

Before I get into it, however, I would like to point out that if you're deciding whether or not to get a Raspberry Pi, the answer is no. The Raspberry Pi is overhyped - this is why I'm writing this guide - and there are much better alternatives. I'll mention one here, the Beagle Bone Black. Update: another good one: Hummingboard

Why you should get this instead:

• Faster CPU and Memory
  • 1GHz processor instead of 700MHz
  • DDR3 instead of SDRAM

• It can run Debian or Ubuntu armhf directly instead of having to run something like Raspian. It's silly that Raspbian exists when there is already an armhf port of Debian. If you just install normal armhf Ubuntu on the Beagle Bone Black then this entire guide is unnecessary and you can just do

  # apt-add-repository ppa:andrewrk/libgroove
  # apt-get update
  # apt-get install libgroove-dev

  And presto, you're done.

  In fact, libgroove is in Debian Testing and Ubuntu Utopic Unicorn, so in a year or so when these distributions are updated, you won't even need to add the extra PPA.

• Debian officially recommends against the Raspberry Pi, notably because there is non-free software required to run it. Debian specifically endorses the Beagle Bone Black.

If you are like me, and you unfortunately purchased a Raspberry Pi before you became educated about better options, then you'll have to jump through some hoops to get this working. This article will hold your hand and guide you through all the hoops so that you don't have to waste time figuring it out yourself.

I'll start this guide at the point where you have a fresh Raspberry Pi and don't even have an operating system yet. If you're past this point then skip ahead.

Table of Contents

1. Table of Contents
2. Install Raspbian and get SSH Access
3. Install Groove Basin

Install Raspbian and get SSH Access

Head over to the Raspberry Pi downloads page and grab the Raspbian Debian Wheezy torrent (or download directly if you're not l33t).

Unzip to get the .img file out and flash it to the biggest SD card you have. You'll want lots of room for music!

I'm on Ubuntu - all I had to do was right-click on the .img file in nautilus, Open With, Disk Image Writer:

I'm sure there are plenty of ways to get the job done, this was easiest for me.

Once that's done, find a keyboard, monitor, and HDMI cable so that you can see what you're doing. Our goal is to get SSH access going as soon as possible so that we can work on the Pi without plugging things into it other than the power and network cables.

Once the Pi boots up for the first time it gives you a menu of things you can do. Here's what I did:

• Expand the file system to fit the full SD card.
• Set pi user password.
• Advanced Options, enable SSH server.

Now let's set it up so that it always binds to the same IP address when it boots up. These are the settings I used, obviously you should tweak them to your network's needs:

sudo vi /etc/network/interfaces

Replace iface eth0 inet dhcp with:

auto eth0
iface eth0 inet static
    address 192.168.1.99
    netmask 255.255.255.0
    gateway 192.168.1.1

Now we can unplug the TV and keyboard, we won't be needing this junk anymore. Plug that Raspberry Pi into your network and power it on!

On your normal computer that you're used to using, you can now ssh to the Pi, something like this:

ssh pi@192.1.68.1.99

I like to put an entry in my ~/.ssh/config file like this:

host pi
hostname 192.168.1.99
user pi

It makes you type in your password, but we can fix that:

ssh-copy-id pi

Now connecting to the Pi is as simple as ssh pi.

The first thing to do here is update all the outdated packages to the latest.

$ sudo apt-get update
$ sudo apt-get dist-upgrade

Hmm, what's that I see there?

...
Unpacking replacement raspberrypi-bootloader
...

Bootloader replaced huh? Better reboot to make sure that still works.

Alright, at this point we are able to ssh into our Raspberry Pi and all the packages that come installed are fully updated.

Install Groove Basin

First let's install some packages:

$ sudo apt-get install htop vim git cmake screen

I recommend that you do this work in something like screen or tmux so that if the connection is dropped, the commands we're running will continue. Also this allows us to disconnect and go do something else while the Pi crunches numbers.

I'm going to explain how to do this one step at a time for clarity. However, note that there are essentially 3 compilations that will take a very long time, so if you want to start those 3 in parallel and then walk away from the computer for 8 hours or so, you can skip around this guide and start them all in parallel. Those 3 things are the make steps of:

• SDL2
• libav
• Node.js

libgroove, Part 1

Get the libgroove source code and create a build folder inside of the source:

$ cd
$ git clone https://github.com/andrewrk/libgroove
$ cd libgroove
$ mkdir build
$ cd build

Let's build in debug mode so that if we happen upon any errors we can get a useful stack trace.

Note, you can skip the following step - it takes a minute or two to complete, this particular command is just for your information, and I have reproduced the output below:

$ cmake .. -DCMAKE_BUILD_TYPE=Debug

Installation Summary
--------------------
* Install Directory            : /usr/local
* Build libgroove              : missing dependencies
* Build libgrooveplayer        : missing dependencies
* Build libgrooveloudness      : missing dependencies
* Build libgroovefingerprinter : yes

Bundled Dependencies
--------------------
* SDL2                         : ready to build
* libav                        : missing dependencies, see below
* libebur128                   : ready to build

System Dependencies
-------------------
* C99 Compiler                 : OK
* threads                      : OK
* SDL2                         : not found - will use bundled version
* ebur128                      : not found - will use bundled version
* chromaprint                  : not found
* libavformat                  : not found - will use bundled version
* libavcodec                   : not found - will use bundled version
* libavfilter                  : not found - will use bundled version
* libavutil                    : not found - will use bundled version
* yasm                         : not found
* bzip2                        : not found
* mp3lame                      : not found
* zlib                         : OK

It's missing these libraries:

• chromaprint
• libebur128
• SDL2
• libav

We could let libgroove install with the bundled dependencies, but it will be easier to just install those dependencies on the system first. Let's do that.

chromaprint

Luckily chromaprint is in the repository already:

$ sudo apt-get install libchromaprint-dev

libebur128

Next we compile the easy one, libebur128.

$ cd
$ git clone https://github.com/jiixyj/libebur128
$ cd libebur128
$ mkdir build
$ cd build
$ cmake .. -DCMAKE_BUILD_TYPE=Debug

Oops, looks like we're missing a dependency:

-- checking for module 'speexdsp'
--   package 'speexdsp' not found

Better install that.

$ sudo apt-get install libspeexdsp-dev

Let's try that configure line again:

$ cmake .. -DCMAKE_BUILD_TYPE=Debug

-- checking for module 'speexdsp'
--   found speexdsp, version 1.2rc1
-- speexdsp library dirs: 
-- speexdsp cflags: 
-- speexdsp include dirs: 
-- speexdsp libraries: speexdsp
-- speexdsp ldflags: 
-- status          found / disabled --
-- queue.h:        yes     using system copy of queue.h
-- speexdsp:       yes     no 
-- not building tests, set ENABLE_TESTS to ON to enable
-- Configuring done
-- Generating done
-- Build files have been written to: /home/pi/libebur128/build

That's better.

Now compile and then install the code:

$ make
$ sudo make install

[ 50%] Built target ebur128
[100%] Built target ebur128_static
Install the project...
-- Install configuration: "Debug"
-- Up-to-date: /usr/local/include/ebur128.h
-- Installing: /usr/local/lib/arm-linux-gnueabihf/libebur128.so.1.0.1
-- Installing: /usr/local/lib/arm-linux-gnueabihf/libebur128.so.1
-- Installing: /usr/local/lib/arm-linux-gnueabihf/libebur128.so
-- Installing: /usr/local/lib/arm-linux-gnueabihf/libebur128.a

Argh, it put the library files in /usr/local/lib/arm-linux-gnueabihf/ due to a bug in the Debian cmake package.

Let's hack around that:

$ sudo mv /usr/local/lib/arm-linux-gnueabihf/* /usr/local/lib/
$ sudo rmdir /usr/local/lib/arm-linux-gnueabihf
$ sudo ldconfig

SDL2

Next let's get SDL2 going. What's wrong with the SDL that comes with the Raspberry Pi? Well it's version 1.2 and libgroove depends on version 2.

Find the URL for the source code of the latest SDL2 version on the SDL download page.

$ cd
$ wget https://www.libsdl.org/release/SDL2-2.0.3.tar.gz
$ tar xvf SDL2-2.0.3.tar.gz 
$ cd SDL2-2.0.3/

SDL needs an audio backend to work, so we install that now, before the configure command:

$ sudo apt-get install libasound2-dev

We only need the audio features of SDL2 and in fact some of the video stuff can cause compilation problems. So we'll disable all the features we don't need when we configure.

$ ./configure --enable-audio --disable-video --disable-render --disable-events --disable-joystick --disable-haptic --disable-power --disable-file --disable-timers --disable-loadso --disable-cpuinfo

This could take a while, but when it's done you should see a line like this:

Audio drivers   : disk dummy oss alsa(dynamic)

It's important that you have an audio driver other than disk, dummy, and oss.

$ make

Find something to do, this is going to take upwards of an hour to complete. Or if you want to slap your poor Raspberry Pi into submission, this would be the time to skip around in this guide and get libav compiling at the same time. But again, I'm going to pretend that you're doing this sequentially and let you deal with thinking about how to skip around the article.

So at this point that long compilation process succeeded and we're ready to install SDL2:

$ sudo make install

libav

Let's go back to your home folder (or wherever you decided to do this):

$ cd

Grab the URL to the latest libav 10 release from the libav downloads page.

$ wget http://www.libav.org/releases/libav-10.1.tar.gz
$ tar xvf libav-10.1.tar.gz 
$ cd libav-10.1/

Let's get some prerequisites out of the way and then start configuring:

$ sudo apt-get install libmp3lame-dev libvorbis-dev
$ ./configure --enable-shared --enable-debug --disable-static --enable-gpl --enable-libmp3lame --enable-libvorbis

Again the Pi is going to have to work really hard to complete this configure command, especially if you're simultaneously compiling SDL2, so don't worry if it takes a minute or two.

$ make

This is going to take upwards of 8 hours. Seriously, I'd start this one and then go to bed. If you're trying to start all the compilations simultaneously, you might also want to start Node.js compiling as well.

After libav compilation succeeds:

$ sudo make install

Now we've finally finished installing libgroove's dependencies and we can finally move on to installing libgroove itself.

libgroove, Part 2

So at this point, you've waited a very long time and the Pi has successfully finished compiling libav and SDL2, and you have installed both of them. If this is not true, then you need to figure out why and fix it before progressing with this guide.

$ sudo ldconfig
$ cd ~/libgroove/build/
$ cmake .. -DCMAKE_BUILD_TYPE=Debug

Installation Summary
--------------------
* Install Directory            : /usr/local
* Build libgroove              : yes
* Build libgrooveplayer        : yes
* Build libgrooveloudness      : yes
* Build libgroovefingerprinter : yes

Bundled Dependencies
--------------------
* SDL2                         : using system library
* libav                        : using system libraries
* libebur128                   : using system library

System Dependencies
-------------------
* C99 Compiler                 : OK
* threads                      : OK
* SDL2                         : OK
* ebur128                      : OK
* chromaprint                  : OK
* libavformat                  : OK
* libavcodec                   : OK
* libavfilter                  : OK
* libavutil                    : OK

Ah that output looks much better than before.

$ make

This make should be relatively quick.

$ sudo make install
$ sudo ldconfig

At this point we have the necessary libraries installed:

$ ls /usr/local/lib/

libavcodec.so           libebur128.a                     libgroove.so.4
libavcodec.so.55        libebur128.so                    libgroove.so.4.1.0
libavcodec.so.55.34.1   libebur128.so.1                  libSDL2-2.0.so.0
libavdevice.so          libebur128.so.1.0.1              libSDL2-2.0.so.0.2.1
libavdevice.so.54       libgroove.a                      libSDL2.a
libavdevice.so.54.0.0   libgroovefingerprinter.a         libSDL2.la
libavfilter.so          libgroovefingerprinter.so        libSDL2main.a
libavfilter.so.4        libgroovefingerprinter.so.4      libSDL2.so
libavfilter.so.4.2.0    libgroovefingerprinter.so.4.1.0  libSDL2_test.a
libavformat.so          libgrooveloudness.a              libswscale.so
libavformat.so.55       libgrooveloudness.so             libswscale.so.2
libavformat.so.55.12.0  libgrooveloudness.so.4           libswscale.so.2.1.2
libavresample.so        libgrooveloudness.so.4.1.0       pkgconfig
libavresample.so.1      libgrooveplayer.a                python2.7
libavresample.so.1.1.0  libgrooveplayer.so               python3.2
libavutil.so            libgrooveplayer.so.4             site_ruby
libavutil.so.53         libgrooveplayer.so.4.1.0
libavutil.so.53.3.0     libgroove.so

Node.js

Now we need Node.js. Get the latest stable source code from the downloads page.

$ cd
$ wget http://nodejs.org/dist/v0.10.29/node-v0.10.29.tar.gz
$ tar xvf node-v0.10.29.tar.gz 
$ cd node-v0.10.29/
$ ./configure
$ make

This compilation process will take several hours.

Once it's done:

$ sudo make install

Groove Basin

Now it is time to start Groove Basin, the music player server.

$ cd
$ git clone https://github.com/andrewrk/groovebasin
$ cd groovebasin/
$ npm run build

This step can take several minutes - it downloads and compiles Groove Basin dependencies.

Let's make the music directory if we don't already have one.

$ mkdir ~/music/

Copy all your music there at this point if you have any.

$ node lib/server.js

Now you should be up and running. If you want to change configuration options, kill the server with Ctrl+C and edit config.js.

Enjoy! Feel free to follow Groove Basin on GitHub, file a bug report or feature request, or join #libgroove on irc.freenode.org to discuss or get help troubleshooting.

Pull requests are welcome, especially ones that make groovebasin.com look nicer.

I used to have a Dell Inspiron XPS 13 Developer Edition. It was fine, but I needed something with a better graphics card because I wanted to learn some of the new OpenGL features by prototyping a 3D space flight simulator game.

I decided to buy the Bonobo Extreme laptop which comes with Ubuntu pre-installed.

This thing is a monster. It's huge, bulky, and heavy. Check out the power brick:

Yes, that is a standard IEC 60320 C13 power supply cord going into the laptop brick.

Here are the peripherals:

Compliments

• Plenty fast and performant. Handles any game I've thrown at it, no problem. make -j8 like a boss.
• The Ubuntu logo on the super key instead of Windows logo is pretty fun.
• Drivers and everything work out of the box. This is huge.
• Battery life is surprisingly good for this beast of a machine. I can get upwards of 4 hours easily. Of course this goes down when I run the CPU or GPU hot.
• The keyboard button to turn off the LCD display is surprisingly handy.
• The built-in speakers are loud and sound decent for a laptop.

Criticism

Driver/Hardware

• Dead pixels after 1 month of use. Now I'm 8 months in and I have an entire column of about 20 dead pixels in a row. Based on reading the support forum this is normal.
• Sometimes it tries to connect to a wired network when there is no network cable plugged in. This may be an Ubuntu bug.
• The power cable is not a perfect fit. It feels like you must jam it in there and it's not clear when it is "in".
• Every two weeks or so, the system has a complete hardware failure and powers off. It's not due to heat, it's a weird hardware problem. And it's not only me; my room mate has the same laptop and it happens to him too.

Keyboard

• It's not possible to push Shift+Home or Shift+End, which is an extremely common and handy shortcut when text editing. I'd rather give up the entire numpad than give up Shift+Home and Shift+End.
• It took me a long time to not accidentally press spacebar with my left thumb when I wanted to press Alt. Spacebar goes pretty far to the left.
• The key between Spacebar and Right Alt is completely useless. It displays a pipe and backslash on it ("| \") but it actually inserts a "<" or a ">" depending on whether I press shift. None of these 4 things are more convenient than the canonical position of the respective key it replaces. I'd rather have the traditional key that pops the context menu for the widget that is in focus.
• There is a button to put the computer to sleep which is between Mute and Volume Down. I press it on accident sometimes.

Touchpad

• It is difficult to click without moving the mouse.
• The difference between left click and right click is subtle. It's very easy to do the wrong one.

Curiosity

I noticed that the HDD LED flashes whenever I toggle numlock. I wonder why that is. It doesn't do that for caps lock.

Conclusion

Lots of room for improvement. I think it's overpriced given the problems that it has. Getting a powerful machine for which you know for sure all the drivers will work with Linux is nothing to sneeze at. However, it's probably worth it to spend your money on a system for which the hardware has undergone more quality assurance.

Over the past few years, I have been slowly but surely building my own music player. It's been a wild ride. The codebase has radically changed several times, but is always converging on a better music listening experience.

In this article my goal is to take you along for the ride.

See the project on GitHub

Table of Contents

1. Table of Contents
2. I <3 Amarok 1.4
3. A Short Explanation of Loudness Compensation
4. Shortcomings of Amarok 1.4
5. My Laundry List of Music Player Features
6. "PartyBeat"
7. Fumbling Around with Technology
8. Growing Pains
9. Rejecting MPD
10. Building a Music Player Backend
11. Packaging
12. Contributing to libav
13. Conclusion

I <3 Amarok 1.4

Back in 2009, my music player of choice was Amarok 1.4. This was by far the best music player I had ever used on Windows, Mac, or Linux, especially when combined with the wonderful ReplayGain plugin. Here's a screenshot:

One way you can tell how much people loved this music player is by looking at the comments on the release blog articles for Amarok 2.0, which rebuilt the player from scratch and took it in a completely different direction. Arguably, they should have picked a different project name and logo. Look at some of these comments, how angry and vitriolic they are: 2.0 2.1 2.2

Even now, 4 years later, the project is at version 2.8 and the release name is titled "Return To The Origin":

Amarok 1.8 is titled "Return To The Origin" as we are bringing back the polish that many users loved from the original 1.x series!

That should give you an idea of how much respect Amarok 1.4 commanded.

Even so, it was not perfect. Notably, the ReplayGain plugin I mentioned above had several shortcomings. Before I get into that, however, let me take a detour and explain what ReplayGain, or more generally, loudness compensation, is and some of its implications.

A Short Explanation of Loudness Compensation

Have you ever seen this 2-minute video explaining the Loudness War?

The video demonstrates a trend in digital audio mastering where songs are highly compressed to sound louder, and how this can compromise the integrity of the music.

While thinking about building a music player, we're not going to make moral judgments about whether or not compression is ruining music for everybody. If users want to listen to highly compressed music, that's a valid use case. So we have to consider a music library which contains both compressed songs and dynamic songs.

Here is a song called The Happiest Days of Our Lives by Pink Floyd, mastered in 1979:

Here is a song called Saying Sorry by Hawthorne Heights, mastered in 2006:

It is immediately obvious that the second one is much louder than the other. So what happens when they are played one after the other in a music player?

When the quieter song comes on first, the user reaches for the volume knob to turn it up so they can hear. Oops. When the next song begins, a surge of adrenaline shoots through the user's body as they scramble to turn the volume down. This goes beyond poor usability; this problem can cause hearing loss.

The solution is to analyze each song before playing it to figure out how "loud" it sounds to humans. Then the music player adjusts the playback volume of each track to compensate for the perceived loudness. This way, the user does not have to adjust the volume for each track that comes on.

The idea is simple enough, but it poses a few subtle challenges.

For one, the loudness of an individual track might be different than the loudness of the album as a whole. A complete loudness compensation solution has to take this into account, both during scanning and playback.

An even trickier problem is avoiding clipping. Music is composed of samples which have a fixed range. For example in floating point format, samples can be between 0.0 and 1.0. Even quiet songs usually have some samples which peak at 1.0, for example on the drums. But we need to turn the volume up on these quiet songs to make them sound as loud as the highly compressed ones.

If we naïvely increased the volume on such a song, we would end up with something like this:

The grey bars above the red lines represent clipping. This causes distortion and generally sounds awful.

The solution is not to increase the volume of the quiet song, but to decrease the volume of the loud song. In order to do this, we introduce an amount called pre-gain. All songs are turned down by this amount, which gives us the headroom we need to turn the quieter ones back up.

It's not a perfect solution though.

The lower the pre-gain, the more the music player will sound quieter than other applications on the computer. The higher the pre-gain, the more likely that there is not enough headroom to increase the volume of a quiet song enough.

The ReplayGain 1.0 Specification outlines this in more detail.

In 2010, the European Broadcasting Union introduced a new standard called R128. This standard outlines a strategy for analyzing media and determining how loud it is. There is a motion to make ReplayGain 2.0 use this standard.

I recommend this excellent Introduction to EBU R128 by Florian Camerer:

Shortcomings of Amarok 1.4

As much as I loved Amarok 1.4, it did not even attempt to address these loudness issues. There is no built-in loudness compensation.

The ReplayGain plugin I mentioned earlier was great, but it was limited in usefulness:

• It had to scan every time the playlist updated; it didn't cache the data.
• Each format that you wanted to scan had a different command-line utility which had to be installed. This means that the set of songs that Amarok 1.4 could play was completely different than the set of songs that it could scan.
• It applied the volume changes on a gradient instead of instantly, and timing was not precise. This means that it might erroneously turn up the loudness far too high in the transition time to the next track. This behavior was distracting and sometimes ear-piercingly painful.
• You had to manually decide between track and album mode. This is a pointless chore that the music player should do automatically. Here's a simple algorithm:
  • If the previous item in the playlist is the previous item from the same album, or the next item in the playlist is the next item from the same album, use the album ReplayGain information.
  • Otherwise, use the track ReplayGain information.

Aside from the loudness compensation, I had a couple other nits to pick:

• Dynamic Mode was a useful feature that could continually play random songs from the library. But the random selection was too random; it would often queue the same song within a short period of time.
• If the duration tag was incorrect in a song, or if in was a variable rate MP3, the song would seemingly end when the song had not yet gotten to the end. Or in other words, the reported duration was incorrect and seeking would be broken.

I've spent some time criticizing, now let me be more constructive and actually specify some features that I think music players should have.

My Laundry List of Music Player Features

Loudness Compensation using the same scanner as decoder

This is absolutely crucial. If you want to solve the loudness compensation problem, the set of songs which you can decode and play back must be the same set of songs which you can scan for loudness. I should never have to manually adjust the volume because a different song or album came on.

Ideally, loudness scanning should occur lazily when items are added to the play queue and then the generated values should be saved so that the loudness scanning would not have to be repeated.

Do not trust duration tags

A music player already must scan songs to determine loudness compensation values. At the same time, it should determine the true duration of the file and use that information instead of a tag which could be wrong.

If my friends come over, they can control the music playback

Friends should be able to upload and download music, as well as queue, skip, pause, and play.

Ability to listen to my music library even when I'm not home

I should be able to run the music player on my home computer and listen to a real-time stream from work, for example.

Gapless Playback

Many albums are created in order to be a listening experience that transcends tracks. When listening to an album, songs should play seamlessly and without volume changes at the seams. This means that loudness scanning must automatically take into account albums.

Robust codec support

You know how when you need to play some obscure video format, you can always rely on VLC to play it? That must be true for the ultimate music player as well. A music player must be able to play music. If you don't have a wide range of codecs supported, you don't have a music player.

Keyboard Shortcuts for Everything

I should be allowed to never touch the mouse when operating the music player.

Clean up my messy files

One thing that Amarok 1.4 got right is library organization. It offered a powerful way to specify the canonical location for a music file, and then it had an option to organize properly tagged music files into the correct file location.

I don't remember the exact format, but you could specify a format something like this:

%artist/%album/%track %title%extension

Filter Search

There should be a text box where I can type search terms and instantly see the search results live. And it should ignore diacritics. For example, I could type "jonsi ik" and match the song Boy Lilikoi by Jónsi.

Playlist Mode that Automatically Queues Songs

Some names for this feature are:

• Dynamic Mode
• Party Mode
• DJ Mode

The idea is that it automatically queues songs - kind of like a real-time shuffle - so that you don't have to manually decide what to listen to.

One common flaw found in many players is using a truly random algorithm. With true randomness, it will frequently occur that a song which has recently been randomly chosen will be randomly chosen again.

A more sophisticated algorithm weights songs by how long it has been since they have been queued. So any song would be possible to be queued, but songs that have not been queued recently are much more likely to be queued. Queue date is chosen rather than play date because if a song is queued and the user skips the song, this should still count in the weight against it being chosen again.

"PartyBeat"

It would be a long time before my wishlist of features would become a reality. Meanwhile, back in college my buddy made a fun little project which served as a music player that multiple people could control at the same time with a web interface. He installed it in my and my roommate's apartment, and the three of us used it in our apartment as a shared jukebox; anarchy deciding what we would listen to while we worked on our respective jobs, homework, or projects. We dubbed it "PartyBeat".

Here's a screenshot:

You might recognize that UI scheme - it is the Trontastic JQuery UI theme.

This project used Django and xmms2 and had a bug-ridden, barren, and clunky web-based user interface. It was so lacking compared to my usual Amarok 1.4 experience, yet somehow I could not give up the "shared jukebox" aspect. It was simply too fun to listen to music together, regardless of the interface.

So finally I decided to build the ultimate music player. It would still have a web-based interface, but it would behave like a native application - both in feel and in responsiveness. It should be nice enough that even when you want to listen alone it would still be your go-to player of choice.

Fumbling Around with Technology

In early 2011 I started investigating what technology to use to build this thing. I knew that I wanted a backend which could decode many audio formats, do gapless playback, and provide some kind of interface for a web server to control it.

I tinkered a bit with Qt and the Phonon framework, but I didn't get as far as having a web interface controlling it.

Eventually I stumbled upon Music Player Daemon. At the time this seemed like a perfect fit, especially since the XMMS2 wiki admitted that if they had known that MPD existed when they started the project, they would probably have just used it. MPD is a service - it has a config file which tells it, among other things, the location of your music library, and then it runs in the background, listening on a port (typically 6600), where you can issue commands via the protocol telling it to pause, play, skip, queue, unqueue, and all that jazz.

The first iteration of "PartyBeat2" was a small Python 3 server which was merely a proxy between the client-side JavaScript code and MPD, as well as a file server to serve the client-side HTML and JavaScript.

At this point I had a basic proof-of-concept. However, progress slowed for a few months as I embarked on a 12-day hiking trip followed immediately by the first day of work at Amazon, my first out-of-college job.

After a short hiatus, I revisited the project. This was right when socket.io was getting a lot of hype, and it seemed like the perfect fit for my design. Also I had just given Coffee-Script a real chance after snubbing it initially. So I ported over the proxy/file server to Node.js and got a prototype working:

I even did some design drawings on paper:

A week of iterating later, I had the basics of a user interface, and a name:

I named it Groove Basin, after the Sonic the Hedgehog 3 Azure Lake Remix by Rayza. As homage to the original project, I picked a JQuery UI theme for the UI, except this time I chose Dot Luv.

Growing Pains

Progress continued off and on over the period of about a year. As the feature set grew larger and more solidified, the server which used to only be a proxy and file server took on more and more responsibilities.

Dynamic Mode required that the server watch the main playlist and add songs to the end and remove them from the beginning as playback continued from one track to the next.

Last.fm scrobbling required that the server send scrobbles even when the client was not connected.

As the server took on more responsibilities, it began to make less and less sense for it to directly proxy the MPD protocol for the client.

Meanwhile, I became unsatisfied with Coffee-Script. Perhaps digressing too much, some of the issues I took with it were:

Error-prone variable scoping

If you accidentally name a local variable the same as one in an outer scope, you mutate the value of the outer one rather than shadowing it. For example if you path = require('path') and then later innocently use path as the name of a local variable, you're in for a wild ride:

path = require('path')

# ...

makePathName = (song) ->
  path = song.title
  if song.artist
    path = song.artist + '/' + path
  return path

# ...

basename = makePathName(title: 'hello')
path.join(musicDir, basename) # TypeError: Object  has no method 'join'

Messy and inefficient output code

arr = [1, 2, 3]
f = ->
  for x in arr
    console.log x
  for x in arr
    console.log x
  for x in arr
    console.log x

Produces:

(function() {
  var arr, f;

  arr = [1, 2, 3];

  f = function() {
    var x, _i, _j, _k, _len, _len1, _len2, _results;
    for (_i = 0, _len = arr.length; _i < _len; _i++) {
      x = arr[_i];
      console.log(x);
    }
    for (_j = 0, _len1 = arr.length; _j < _len1; _j++) {
      x = arr[_j];
      console.log(x);
    }
    _results = [];
    for (_k = 0, _len2 = arr.length; _k < _len2; _k++) {
      x = arr[_k];
      _results.push(console.log(x));
    }
    return _results;
  };

}).call(this);

If you look closely at that output JavaScript code, you'll notice something even more annoying. Every function returns a value unless you explicitly put a return statement at the end. This can have surprising side effects. In our example code, Coffee-Script decided to put the output of console.log into an array. This is a controversial feature, but it is not going to change.

Inability to declare functions

In Coffee-Script you cannot declare a function; you can only assign an anonymous function to a variable. This makes it impossible to do what I consider to be the cleanest organization of callback code.

Playing with coco

What I should have done at this time is gone back to plain old JavaScript. But I was still seduced by the features that compile-to-js languages bring to the table. So instead I switched the codebase over to coco. This project solved some of Coffee-Script's problems including all the ones I listed above, and satyr seemed to have a better understanding of compiler design than jashkenas given that coco ran twice as fast.

coco lasted about a year before I removed it. satyr started taking coco in a pretty wild direction, adding things like c = a-b compiling to c = aB rather than c = a - b. The syntax became so complicated that if you made a typo in the source code, you had more of a chance of introducing a subtle bug than of introducing a syntax error.

In the end, though, the biggest factor, and this goes for Coffee-Script as well as coco as well as any other compile-to-js language, is that it alienates possible contributors.

As I gained more experience with Node.js, I realized that most developers used JavaScript directly instead of a compile-to-js language. By using a language that significantly fewer people were familiar with, I made Groove Basin a less attractive project to contribute to.

Rejecting MPD

All this compile-to-js stuff was meta work; it had no fundamental effect on how well the music player performed. Meanwhile there lurked a more substantial problem with the way Groove Basin was designed.

At first, MPD seemed like a great choice. It is in most popular Linux distributions' package managers, including the Raspberry Pi. It had been around for long enough that there are multiple free iPhone and Android apps available to act as a controller. It can play most audio formats. But in the end, there are some critical issues that prevent it from being the right choice for Groove Basin.

Demands control of your music library

The only way to play a song is to add it to the library, and then queue it. If you want to implement your own music database and use MPD as a simple playlist for playback, you still have to keep MPD's library up to date too.

ReplayGain support is laughable

MPD supports only APEv2 ReplayGain tags. This must be some kind of joke, because obviously not every format supports APEv2 tags, and most ReplayGain scanners actually write to ID3v2 tags. But more importantly, MPD misses the entire point. Relying on external ReplayGain scanning makes the set of songs you can play different from the set of songs you can scan. This leads to an inconsistent and rather unpleasant listening experience. Further, relying on tags makes it impossible to store ReplayGain data for songs in the library in a container format which does not support tags. And finally... what the hell? Is the user supposed to set up their own cron job to scan their music collection? How about the music player app does it for the user, silently, in the background, no questions asked?

No tag editing

After demanding control of your music library, MPD provides no way to edit tags of a song.

If you're going to read and write audio tags, the same library should be in charge of both. Otherwise, like the ReplayGain problem I outlined earlier, you end up with discrepancies in what you can read and write.

Protocol is severely limited and poorly designed

MPD is controlled via the MPD Protocol that can be used to control playback and query information. For the most part, all the information that you need is there. However the protocol is massively inefficient.

For one example, consider the use case of a client which wants to keep an index of the music database. That is, it wants to have an updated copy of all the music metadata, so that it can do things such as queuing a random track, or allowing the user to quickly search for a song. In this use case, the mpd client would have to request a copy of the music database index, and then subscribe to a notification when the database changes. Once that notification is sent, the client would then have to re-request the entire index again, an operation which can be upwards of 3MB of data for a music library with 9000 songs. A better behavior would be if the notification included a delta of exactly what changed, so that the client could keep their copy updated without that massive payload.

This same problem exists with the main playlist - you have to request the entire playlist instead of receiving a delta of what changed.

Another problem with the MPD protocol is that although it is intended to support multiple concurrent users controlling the same server, it is riddled with race conditions.

For example, it packages multiple state updates into one. Consider the status update message:

status
volume: 100
repeat: 0
random: 0
single: 0
consume: 1
playlist: 0
playlistlength: 23
xfade: 0
mixrampdb: 0.000000
mixrampdelay: nan
state: play
song: 10
songid: 0
nextsong: 11
nextsongid: 1
time: 69:224
elapsed: 68.985
bitrate: 192
audio: 44100:24:2
OK

Now imagine that one user adjusts volume while another user toggles the repeat state. Because volume and repeat state are sent in the same message, at least one user will receive a status message with incorrect information before receiving a new one with correct information. In practice, this means that the UI on clients will momentarily display bogus state when things change which makes clients feel "glitchy".

As another example, audio files are indexed by filename. This means that if you rename a file, every client which had a handle on that file now has an invalid handle. Even after you download the entire new music library index, there is no way to tell which file got renamed.

One final example. Consider the use case where the user presses the volume up button several times quickly.

The problem is that you receive an event saying that the status has changed. The only reasonable response to this is to ask what the new status is. The new status tells us that the volume is at 3. Now, there are 2 more messages that will arrive soon telling us that the new volume is 4 and then 5. But before they arrive, the UI is updated, the user sees an invalid value, and when they press volume up again, it is from the invalid base position of 3 instead of 5. Consider a simpler alternative which solves this problem. When the client sends a volume update, the server accepts the message and then only notifies other clients that the volume changed.

Various audio playback glitches

MPD supports something called "stickers". These are simple pieces of data MPD clients can add to tracks to use for their own purposes. Groove Basin took advantage of stickers to store "last queued date" on each track in order to implement its random song selection which favors songs you haven't heard recently. Josh discovered that MPD apparently makes two stupid decisions with regard to stickers:

1. It uses the same thread for audio playback as it uses for updating sticker information.
2. It somehow is so inefficient at updating sticker information that it causes audio playback to skip a few tenths of a second if you try to do it for upwards of 8 songs at once.

The end result here was that audio playback would glitch if you queued an album.

In addition there were some basic audio playback issues. Sometimes after unpausing, audio playback would stutter quickly until the song ended. Sometimes the HTTP stream would not send audio data quickly enough, so the HTTP stream would have to stop to buffer repeatedly, and the behavior had to be fixed by disabling and re-enabling the audio output in MPD.

Now it is possible to submit patches to MPD to get these bugs fixed. But if I'm going to work on that layer of the problem, why not put that effort toward a project which better fulfills Groove Basin's goals?

We want more control over the HTTP audio stream

In the best case scenario, the music player server would know which of the connected browser clients are actually streaming music. With MPD in control of the HTTP audio stream, we have no access to or control over the HTTP request. We can only provide the URL. Also it requires running on a separate port from the main web interface which is its own can of worms.

We also want more control over the audio stream. When a song is skipped, for example, the best thing to do is flush the audio buffer so that the buffer can start filling up with data from the new song. This is precisely what happens in media players that play locally on your computer. However, with MPD this is not possible. When you skip a song you still have to wait for the audio stream to catch up.

Having direct control over the HTTP audio stream also enables us to experiment with some more creative ideas. For example, recently I updated the HTTP stream so that when a client connects, it receives a burst of 200 KB of encoded audio, followed by a steady stream of exactly as many bytes per second as the encoded audio contains. This gives clients enough data to begin playback immediately. You may have noticed this behavior when you watch a YouTube video - the buffering bar loads very quickly at first, but then slows down to a crawl once playback begins.

In practice, I have observed the delay between connecting to the stream URL and playback beginning to be anywhere from seemingly instantaneous to 300ms with this method, depending on latency and bandwidth. Meanwhile, if you use MPD's HTTP audio stream, clients will take upwards of 10 seconds to buffer audio before playback begins.

Building a Music Player Backend

So, how hard could it be to build my own music player backend? Seems like it would be a matter of solving these things:

• Use a robust library for audio decoding. How about the same one that VLC uses?
• Support adding and removing entries on a playlist for gapless playback.
• Support pause, play, and seek.
• Per-playlist-item gain adjustment so that perfect loudness compensation can be implemented.
• Support loudness scanning to make it easy to implement for example ReplayGain.
• Support playback to a sound device chosen at runtime.
• Support transcoding audio into another format so a player can implement, for example, HTTP streaming.
• Give raw access to decoded audio buffers just in case a player wants to do something other than one of the built-in things.
• Try to get other projects to use it to benefit from code reuse.
  • Make the API generic enough to support other music players and other use cases.
  • Get it packaged into Debian and Ubuntu.
  • Make a blog post about it to increase awareness.

After reading up a little bit on the insane open-source soap-opera that was the forking of libav from ffmpeg (here are two sides to the story: libav side, ffmpeg side), I went with libav simply because it is what is in the Debian and Ubuntu package managers, and one of my goals is to get this music player backend into their package managers.

Several iterations later, I now have libgroove, a C library with what I think is a pretty solid API. How it works:

The API user creates a GroovePlaylist which spawns its own thread and is responsible for decoding audio. The user adds and removes items at will from this playlist. They can also call pause, play, and seek on the playlist. As the playlist decodes audio, where does the decoded audio go? This is where those sinks come in.

A sink is a metaphor of a real-life sink that you would find in a bathroom or kitchen. Sinks can fill up with water, and unless the water is drained the sink will continue to fill until it overflows. Likewise, in audio processing, a sink is an object which collects audio buffers in a queue.

In libgroove, decoded audio is stored in reference-counted buffer objects and passed to each connected sink. Each sink does whatever processing it needs to do and then calls "unref" on the buffer. Typically each sink will have its own thread which hungrily waits for buffers and devours them as fast as possible. However the playlist is also decoding audio as fast as possible and pushing it onto each sink's queue. It is quite possible, that a sink's queue fills up faster than it can process the buffers. When the playlist discovers that all its sinks are full, it puts its thread to sleep, waiting to be woken up by a sink which has drained enough.

libgroove provides some higher-level sink types in addition to the basic sink. Each higher level sink runs in its own thread and is built using the basic sink. These include:

• playback sink - opens a sound device and sends the decoded audio to it. This sink fills up with events that signal when the sink has started playing the next track, or when a buffer underflow occurs.
• encoder sink - encodes the audio buffers it receives and fills up with encoded audio buffers. These encoded buffers can then be written to a file or streamed over the network, for example.
• loudness scanner sink - uses the EBU R 128 standard to detect loudness. This sink fills up with information about each track, including loudness, peak, and duration.

The API is designed carefully such that even though the primary use case is for a music player backend, libgroove can be used for other use cases, such as transcoding audio, editing tags, or ReplayGain scanning. Here is an example of using libgroove to for a simple transcode command line application:

/* transcode one or more files into one output file */

#include <groove/groove.h>
#include <groove/encoder.h>
#include <stdio.h>
#include <string.h>
#include <stdlib.h>

static int usage(char *arg0) {
    fprintf(stderr, "Usage: %s file1 [file2 ...] --output outputfile [--bitrate 320] [--format name] [--codec name] [--mime mimetype]\n", arg0);
    return 1;
}

int main(int argc, char * argv[]) {
    // arg parsing
    int bit_rate_k = 320;
    char *format = NULL;
    char *codec = NULL;
    char *mime = NULL;

    char *output_file_name = NULL;

    groove_init();
    atexit(groove_finish);
    groove_set_logging(GROOVE_LOG_INFO);
    struct GroovePlaylist *playlist = groove_playlist_create();

    for (int i = 1; i < argc; i += 1) {
        char *arg = argv[i];
        if (arg[0] == '-' && arg[1] == '-') {
            arg += 2;
            if (i + 1 >= argc) {
                return usage(argv[0]);
            } else if (strcmp(arg, "bitrate") == 0) {
                bit_rate_k = atoi(argv[++i]);
            } else if (strcmp(arg, "format") == 0) {
                format = argv[++i];
            } else if (strcmp(arg, "codec") == 0) {
                codec = argv[++i];
            } else if (strcmp(arg, "mime") == 0) {
                mime = argv[++i];
            } else if (strcmp(arg, "output") == 0) {
                output_file_name = argv[++i];
            } else {
                return usage(argv[0]);
            }
        } else {
            struct GrooveFile * file = groove_file_open(arg);
            if (!file) {
                fprintf(stderr, "Error opening input file %s\n", arg);
                return 1;
            }
            groove_playlist_insert(playlist, file, 1.0, NULL);
        }
    }
    if (!output_file_name)
        return usage(argv[0]);

    struct GrooveEncoder *encoder = groove_encoder_create();
    encoder->bit_rate = bit_rate_k * 1000;
    encoder->format_short_name = format;
    encoder->codec_short_name = codec;
    encoder->filename = output_file_name;
    encoder->mime_type = mime;
    if (groove_playlist_count(playlist) == 1) {
        groove_file_audio_format(playlist->head->file, &encoder->target_audio_format);

        // copy metadata
        struct GrooveTag *tag = NULL;
        while((tag = groove_file_metadata_get(playlist->head->file, "", tag, 0))) {
            groove_encoder_metadata_set(encoder, groove_tag_key(tag), groove_tag_value(tag), 0);
        }
    }

    if (groove_encoder_attach(encoder, playlist) < 0) {
        fprintf(stderr, "error attaching encoder\n");
        return 1;
    }

    FILE *f = fopen(output_file_name, "wb");
    if (!f) {
        fprintf(stderr, "Error opening output file %s\n", output_file_name);
        return 1;
    }

    struct GrooveBuffer *buffer;

    while (groove_encoder_buffer_get(encoder, &buffer, 1) == GROOVE_BUFFER_YES) {
        fwrite(buffer->data[0], 1, buffer->size, f);
        groove_buffer_unref(buffer);
    }

    fclose(f);

    groove_encoder_detach(encoder);
    groove_encoder_destroy(encoder);

    struct GroovePlaylistItem *item = playlist->head;
    while (item) {
        struct GrooveFile *file = item->file;
        struct GroovePlaylistItem *next = item->next;
        groove_playlist_remove(playlist, item);
        groove_file_close(file);
        item = next;
    }
    groove_playlist_destroy(playlist);

    return 0;
}

Note that this code contains no threading. Even so, because of the way libgroove is designed, when this app is run, one thread will work on decoding the audio while the main thread seen in this code will work on writing the encoded buffers to disk.

Once I had this backend built, I needed to use it in Groove Basin, which you may recall is a Node.js app. To do this I built a native add-on node module called groove. It uses libuv and v8 to interface between C and Node.js. I wrote the majority of this code at Hacker School, an experience which I highly recommend.

With the groove node module complete, the new architecture looked like this:

No longer did Groove Basin need to run a third party server to make everything work - just a single Node.js application with the correct libraries installed. And now I was in control of the audio backend code which meant that I had the power to make everything work exactly like I wanted it to.

Packaging

Nothing turns away potential users faster than a cumbersome install process. I knew that I had to make Groove Basin easy to install, so I took several steps to make it so.

One thing I did to make libgroove easy to install is bundle some of the harder to find dependencies along with it. Specifically, libav10, libebur128, and SDL2. This way if the user is on a computer which does not have those packages readily available, they may still install libgroove.

This convenience is less desirable than relying on existing system dependencies, however, so if the configure script detects system libraries, it happily prefers them.

Next, I made a libgroove PPA for Ubuntu users. This makes installing libgroove as easy as:

sudo apt-add-repository ppa:andrewrk/libgroove
sudo apt-get update
sudo apt-get install libgroove-dev libgrooveplayer-dev libgrooveloudness-dev

Then I joined the Debian multimedia packaging team. This team is dedicated to making Debian a good platform for audio and multimedia work. They kindly accepted me and coached me while I worked on packaging up libebur128 and libgroove for Debian. After a few back and forths, a libebur128 Debian package is ready to be installed from testing, and a libgroove Debian package can be installed from experimental. Once the libav10 transition is complete, libgroove can be submitted to unstable, where it will move into testing, and then finally be released to all of Debian!

After a few more months of progress, I'd like to package up Groove Basin itself. This way, the entire installation process could be just an apt-get install away.

Contributions to libav

While working on libgroove several issues came up which led me to contribute code to libav. The first thing I noticed is that if you asked libav for a decoder based on the .ogg extension, by default it would use the FLAC encoder. While it is true that .ogg files can contain FLAC audio, the Xiph.org Foundation recommends that .ogg only be used for Ogg Vorbis audio files.

Unfortunately, the built-in vorbis encoder is considered to not meet quality standards so the compromise that we ended up implementing is to default .ogg to vorbis when libvorbis is available, and default to FLAC when it is not. Fortunately for Debian and Ubuntu users, the libav package that is available in the repository does in fact depend on libvorbis.

Another time I had to dig into libav code was when I discovered that some of my .wma songs would have broken, glitchy playback if you seek to the beginning. Seeking to any other location seemed to work fine. When I investigated this behavior, I noticed that it occurred in avplay, the command line audio playback tool that ships with libav. I proposed a patch which short circuited seeking to 0 and skipped all the complicated seeking code. Janne Grunau not only committed my patch but he took the opportunity to revisit the ASF seeking code, tidy it up, and fix a bunch of issues.

I'm pretty happy about this patch landing in libav as it makes a whole set of songs able to be played for me that previously could not. A pretty important "feature" for a music player.

Finally, while investigating the best way to implement loudness compensation, I realized that simply adjusting the volume on an audio stream is not enough. If the song is so quiet that the amount we would have to turn the volume gain up to exceeds 1.0, we would end up with clipping. The solution to this is to detect whether it is possible that the song could clip and if so, use a compressor instead of a simple volume gain:

A compressor, otherwise known as a limiter, allows us to turn the volume up without clipping. The tradeoff is that it distorts the audio signal. This is why we prefer a simple volume gain, but fall back to a compressor if we need to turn the volume up high enough. This is what VLC does when you turn the volume up past 100% into the red.

In order to have this functionality, I ported the "compand" audio filter from ffmpeg. Now libgroove has the ability to turn the volume up beyond 100% like VLC, although I don't recommend doing it. It's much better for sound quality to turn the gain up on your physical speakers.

Conclusion

3 years, 6 months from git init and Groove Basin is still under active development. Here's what the UI looks like today:

Some of the features that it provides are:

• Fast, responsive UI. It feels like a desktop app, not a web app.
• Dynamic playlist mode which automatically queues random songs, favoring songs that have not been queued recently.
• Drag and drop upload. Drag and drop playlist editing. Rich keyboard shortcuts.
• Lazy multi-core EBU R128 loudness scanning (tags compatible with ReplayGain) and automatic switching between track and album mode. "Loudness Zen"
• Streaming support. You can listen to your music library - or share it with your friends - even when you are not physically near your home speakers.
• MPD protocol support. This means you already have a selection of clients which integrate with Groove Basin. For example MPDroid.
• Last.fm scrobbling.
• File system monitoring. Add songs anywhere inside your music directory and they instantly appear in your library in real time.
• Supports GrooveBasin Protocol on the same port as MPD Protocol - use the `protocolupgrade` command to upgrade.

If you like you can try out the web interface client of Groove Basin on the live demo site. It will probably be chaotic and unresponsive if there is a fair amount of traffic to this blog post, as it's not designed for a large number of anonymous people to use it together; it's more for groups of 10 or less people who actually know each other in person.

The roadmap moving forward looks like this:

1. Tag Editing
2. Music library organization
3. Accoustid Integration
4. Playlists
5. User accounts / permissions rehaul
6. Event history / chat
7. Finalize GrooveBasin protocol spec

Groove Basin still has lots of issues but it's already a solid music player and it's only improving over time.

At some point I plan to write a tutorial article detailing exactly how to get this application running on a Raspberry Pi. It's mostly straightforward but there are enough "gotchas" here and there that I think it could be a useful article.

(update 2014-June-19) I have now written this article: Turn Your Raspberry Pi into a Music Player Server

Feel free to star or watch the Groove Basin GitHub repository if you want to keep track of progress.

Note: this post has been edited to take into account TJ's diligent work in response to this.

I came across this Google+ post mentioning this StackOverflow post in which someone is quite wisely asking whether the express.js framework is secure enough to use for production applications.

This reminds me of one "gotcha" in particular that you could be bitten by if you're not careful.

All servers using express.bodyParser are vulnerable to an attack which creates an unlimited number of temp files on the server, potentially filling up all the disk space, which is likely to cause the server to hang.

Demonstration

This problem is extremely easy to demonstrate. Here's a simple express app:

var express = require('express');
var app = express();

app.use(express.bodyParser());
app.post('/test', function(req, resp) {
  resp.send('ok');
});

app.listen(9001);

Seems pretty innocuous right?

Now check how many temp files you have with something like this:

$ ls /tmp | wc -l
33

Next simulate uploading a multipart form:

$ curl -X POST -F foo=@tmp/somefile.c http://localhost:9001/test
ok

Go back and check our temp file count:

$ ls /tmp | wc -l
34

That's a problem.

Solutions

Always delete the temp files when you use bodyParser or multipart middleware

You can prevent this attack by always checking whether req.files is present for endpoints in which you use bodyParser or multipart, and then deleting the temp files. Note that this is every POST endpoint if you did something like app.use(express.bodyParser()).

This is suboptimal for several reasons:

1. It is too easy to forget to do these checks.
2. It requires a bunch of ugly cleanup code. Why have code when you could not have code?
3. Your server is still, for every POST endpoint that you use bodyParser, processing every multipart upload that comes its way, creating a temp file, writing it to disk, and then deleting the temp file. Why do all that when you don't want to accept uploads?
4. As of express 3.4.0 (connect 2.9.0) bodyParser is deprecated. It goes without saying that deprecated things should be avoided.

Use a utility such as tmpwatch or reap

jfromaniello pointed out that using a utility such as tmpwatch can help with this issue. The idea here is to, for example, schedule tmpwatch as a cron job. It would remove temp files that have not been accessed in a long enough period of time.

It's usually a good idea to do this for all servers, just in case. But relying on this to clean up bodyParser's mess still suffers from issue #3 outlined above. Plus, server hard drives are often small, especially when you didn't realize you were going to have temp files in the first place.

If you ran your cron job every 8 hours for instance, given a hdd with 4 GB of free space, an attacker would need an Internet connection with 145 KB/s upload bandwidth to crash your server.

TJ pointed out that he also has a utility for this purpose called reap.

Avoid bodyParser and explicitly use the middleware that you need

If you want to parse json in your endpoint, use express.json() middleware. If you want json and urlencoded endpoint, use [express.json(), express.urlencoded()] for your middleware.

If you want users to upload files to your endpoint, you could use express.multipart() and be sure to clean up all the temp files that are created. This would still stuffer from problem #3 previously mentioned.

Use the defer option in the multipart middleware

When you create your multipart middleware, you can use the defer option like this:

express.multipart({defer: true})

According to the documentation:

defers processing and exposes the multiparty form object as `req.form`.
`next()` is called without waiting for the form's "end" event.
This option is useful if you need to bind to the "progress" or "part" events, for example.

So if you do this you will use multiparty's API assuming that req.form is an instantiated Form instance.

Use an upload parsing module directly

bodyParser depends on multipart, which behind the scenes uses multiparty to parse uploads.

You can use this module directly to handle the request. In this case you can look at multiparty's API and do the right thing.

There are also alternatives such as busboy, parted, and formidable.

I've seen a fair amount of callback bashing on Hacker News recently.

Among the many proposed solutions, one of them strikes me as particularly clean: "asynchronous wait" or "wait and defer". iced-coffee-script had this a while ago. Kal just debuted with an identical solution, only differing in syntax. Supposedly LiveScript supports this with "backcalls".

I must say, "asynchronous wait" or "monads" - whatever you want to call it - seems like an improvement over callbacks. But I'm here to say that actually... callbacks are pretty okay. Further, given how clean callbacks can be, the downsides of using a compile-to-js language often outweigh the benefits.

I have 2 rules of thumb for code organization which makes clean callback based async code brainless:

1. Avoid nontrivial anonymous functions.
2. Put all function declarations after the code that actually does things.

Example

compile-to-js languages often show examples of deeply nested callback code to show how it can be refactored in the language. Let's take one from Kal:

function getUserFriends(userName, next) {
    db.users.findOne({name:userName}, function (err, user) {
        if (err != null) return next(err);
        db.friends.find({userId:user.id}, function (err, friends) {
            if (err != null) return next(err);
            return next(null, friends);
        });
    });
}

Yikes that does look a bit nested. But let's apply the first rule and un-nest both of those anonymous functions.

function getUserFriends(userName, next) {
    db.users.findOne({name:userName}, foundOne);

    function foundOne(err, user) {
        if (err != null) return next(err);
        db.friends.find({userId:user.id}, foundFriends);
    }

    function foundFriends(err, friends) {
        if (err != null) return next(err);
        return next(null, friends);
    }
}

It's actually longer now, but it's much easier to parse. When you want to learn what any given function does, you only have to understand 1-2 lines. For example, getUserFriends really only has 1 line which is the findOne part. The rest is a list of function declarations. Next you probably want to learn what the foundOne function does, so you jump to it and only have to read 2 lines to know what it does. Finally you probably want to learn what the foundFriends function does, so you jump to it, and again, only have to read 2 lines.

Example

Here's another one taken from Kal (sorry to pick on you rzimmerman; you have good examples):

var async = require('async');

var getUserFriends = function (userName, next) {
    db.users.findOne({name:userName}, function (err, user) {
        if (err != null) return next(err);
        getFriendsById(user.id, function (err, friends) {
            if (err != null) return next(err);
            if (user.type == 'power user') {
                async.map(friends, getFriendsById, function (err, friendsOfFriends) {
                    for (var i = 0; i < friendsOfFriends.length; i++) {
                        for (var j = 0; j < friendsOfFriends[i].length; j++) {
                            if (friends.indexOf(friendsOfFriends[i][j]) != -1) {
                                friends.push(friendsOfFriends[i][j]);
                            }
                        }
                    }
                    return next(null, friends);
                });
            } else {
                return next(null, friends);
            }
        });
    });
}
var getFriendsById = function (userId, next) {
    db.friends.find({userId:userId}, function (err, friends) {
        if (err != null) return next(err);
        return next(null, friends);
    });
}

Yep that is an eyesore. Let's see what the 2 rules do.

var async = require('async');

function getUserFriends(userName, next) {
    db.users.findOne({name:userName}, foundUser);

    function foundUser(err, user) {
        if (err != null) return next(err);
        getFriendsById(user.id, gotFriends);

        function gotFriends(err, friends) {
            if (err != null) return next(err);
            if (user.type == 'power user') {
                async.map(friends, getFriendsById, wtfFriendAction);
            } else {
                return next(null, friends);
            }

            function wtfFriendAction(err, friendsOfFriends) {
                for (var i = 0; i < friendsOfFriends.length; i++) {
                    for (var j = 0; j < friendsOfFriends[i].length; j++) {
                        if (friends.indexOf(friendsOfFriends[i][j]) != -1) {
                            friends.push(friendsOfFriends[i][j]);
                        }
                    }
                }
                return next(null, friends);
            }
        }
    }
}

function getFriendsById(userId, next) {
    db.friends.find({userId:userId}, function (err, friends) {
        if (err != null) return next(err);
        return next(null, friends);
    });
}

Okay actually while refactoring that code I realized it made no sense, hence my naming of wtfFriendAction. But let's run with it.

In this refactored code, we've only reduced the maximum nesting by 1 - from 8 to 7. But consider how much easier it is to follow. When you look at any given function, there are a few lines of synchronous code, followed by function declarations. Exception - I left getFriendsById alone since it is so short.

Quick note on the downsides of compile-to-js languages

First to note - Coffee-Script actually prohibits this kind of code organization, because all functions are necessarily assignments. Other compile-to-js languages solve this problem by providing function declarations. But all compile-to-js languages have some fundamental problems.

For one, you increase the barrier to contributions to your code. People are a bazillion times more likely to create a pull request if they already know the language your module or app is written in. When you pick an obscure language to write your code in, you alienate a large number of potential contributors.

It gets worse. Most people, when evaluating a module or app, will quickly scan the source code to see if the implementation looks reasonable. The depth of analyzation may not be too great; people are looking for obvious problems. When they see that it's written in another language, it makes the codebase seem foreign; possibly even untrusted. At the very least it hampers their ability to judge quality.

Finally, it means adding a build step to your code. Often this is not a big deal; you may already have a build step. But it is an additional moving part in your project that must be understood by you and any potential contributors, or even potential users.

I probably sound like one of those grumps who scoffed at any programming language higher-level than assembly. But I actually really got into Coffee-Script for a while, then switched over to coco due to it solving some problems better. I have a nontrivial music player app written in coco. It was first in JavaScript, then Coffee-Script, then coco. But I've converted back to pure JavaScript in the active working branch. The first version of naught was written in coco but that's now JavaScript as well. I learned the hard way about some of these tradeoffs.

Conclusion

I've outlined 2 simple rules for callback organization that I think make writing async code in pure JavaScript more than adequate. To review:

1. Avoid nontrivial anonymous functions.
2. Function declarations after the code that actually does things.

Both principles aim for the same goal: A reader of your code should be able to look at the code that actually does things synchronously all together. This means placing all the stuff that happens later at the end.

I'm not one to hinder progress in the world of programming languages, but I thought I'd share my perspective on why I still write pure JavaScript for Node.js and browser apps.

It seems that the tech world is trying to thrust upon me an identity which I do not hold. GitHub puts "JavaScript" right under my name:

Here are snippets from actual emails I've received from recruiters:

We are currently expanding our Engineering team here in New York City and are looking for a JavaScript Developer to join our team. I think you'd be a great fit.

Description: HTML 5 Javascript Developer
Looking for an experienced javascript engineer...

This is a lead JavaScript developer that has a great object oriented foundation.

I am also wondering if you would have any interest in learning more about a specific Lead JavaScript Developer position that I am searching on which looks as if it at least might be a good fit for you.

I am not a "JavaScript Developer".

I am a generalist.

I am most engaged when working on hard problems that I have not yet wrapped my brain around.

Sometimes this involves writing a web server.
Sometimes it involves drawing designs or trigonometry problems on paper.
Sometimes it means making a video game in a language I've never used before.
Sometimes it involves creating animations.
Sometimes it means trying to recompile NES games into native executables.
Sometimes it involves using JavaScript.

But never does my thought process go like this:

1. Hmm. I want to write a JavaScript program.
2. What project can I work on so that I can use JavaScript?
3. Hmm, Node.js looks fun, I think I will create a library to help you create Minecraft bots!

It's more like this:

1. Minecraft is fun. I bet it would be even more fun to write bots to do this work for me.
2. (2 days later) get a basic graphical client working with C++ and Qt
3. Hmm, with this framework in place it would be cool to allow people to write their own scripts using a stable API.
4. JavaScript seems like a nice plugin scripting language, and it's built right into Qt!
5. (10 months later) Done! The c++ platform runs user-land JavaScript bots and provides a Minecraft API.
6. (1 year later) Hmm. This abandoned project could probably be re-done pretty cleanly as a Node.js module.
7. (2 months later) Done! Yep this is pretty nice and clean.

Or sometimes it goes like this:

1. I want a program to turn any reasonable audio file into a png of its waveform, and it has to be fast.
2. Okay then, I'll use libsox, libpng, and C.
3. Done! It's nice and fast.

I've created nontrivial applications languages including but not limited to:

• ActionScript 1-3
• C
• C++
• C#
• Go
• Java
• Perl
• Python
• Ruby
• VB.net
• Visual Basic 6

I could write you a genetic algorithm or implement A* search in any of these languages without looking up any reference material.

I've worked on nontrivial projects using frameworks including but not limited to:

• Bison
• Django
• Flex
• LLVM
• MongoDB
• MySQL
• Ogre 3D
• PostgreSQL
• PyGame
• pyglet
• Redis
• SDL
• SFML

Some of the things I've built include but are not limited to:

• Video Games (example 1 2 3 )
• Artificial Intelligence (example 1 2 3 )
• Contributions to lmms, a Digital Audio Workstation.
• Client and server software to control a custom hardware specimen viewer for entomologists. (relevent publication)
• A pascal-to-mips compiler
• Linux utilities
• Windows utilities

I could go on, but I'm not trying to brag; I'm trying to explain why I am annoyed at being labeled a "JavaScript Developer".

If you want to call me something, it should be "Interesting Thing Developer" or "Problem Solver". I'll even accept "Software Developer".

But I am not a "JavaScript Developer".

We all have limited time. This is a guide to help you decide which 7dRTS games to play, and which you might unfortunately have to skip if you don't have time to play them all. It is only one person's opinion.

I haven't played these yet

Definitely check these out

Chess

Play Chess, but without taking turns. Frenetic and very fun. Plays in your browser.

Holo Wars

This game nails it. It's a perfect example of a simple Real Time Strategy game. It gives beginner friendly non-obtrusive help at the beginning to hint what the controls are and what you should do to get started. The graphics and music are fantastic.

Pillagers!

Yes this one is mine and I'm biased. But I do think it is worth your time. It has smooth physics, sophisticated AI, and fantastic music and sound effects created by Michael Weber. It blurs the line between RTS and shmup.

The Great Story of DOTS

Beautifully executed. Gameplay is the right level of difficulty, the tutorial in the beginning is perfect, and it explores a different way of commanding using hand drawn lines of attack. Plays in your browser.

Super Simple Switching Strategist

This Flash game is tastefully minimal. Gameplay is very simple, yet requires strategy to win. It presents 4 stages of increasing difficulty where you must flip switches that decide what units you will create and where they will go.

Age of Rice

Short and sweet, this game pulls off a simple real time strategy game that is fun to play. Plays in your browser and requires only a few minutes of your time.

Sons of a Dark Age

This Flash game is similar to Super Simple Switching Strategist except more frenetic, and more dependent upon your ability to click fast enough. Still, this solid production is worth the 10 minutes of your time it takes to beat.

Worth looking into if you have enough time

Lode Storm (Windows)

This game is a little too hard, but with repeated tries it is possible to win. You have to try to gain control of unit-producing nodes. It has a nice retro feel to it.

Metal Universe

Minimal Flash game. Still, feels like a real-time strategy game. Contains about 5 minutes of fun.

Manifest Destiny (Windows)

Holy shit look at that title screen. This game has some intriguing mechanics but in the end is a bit tedius to play because it requires you to hold down the right mouse button for upwards of 2 minutes at a time, waiting for your armies to increase in number.

Schaak!

This is fascinating - a huge chess board filled with pieces equipped with their own AI. Unfortunately it's local multiplayer - no network, no computer opponent. It's a fun simulation to play with.

Troll Defense

This game is way too hard, but it'll give you a chuckle. Tower defense where meme faces try to get into your website.

Rock, Paper, Geometry!

Although very short, it does feel like a game. You control shapes by giving them "impulses" in the physics engine. You must cause the correct shape on your team to collide with the correct shape on the enemy team.

Mini Wars (Windows)

Short game. Decently balanced. A bit lacking in gameplay depth. The graphics are very green.

Knights & Arrows

The graphics and visual effects are stunning. Unfortunately, the gameplay is extremely frustrating due to your units not automatically defending themselves. Still, it's worth it to ooh and aah at the custom shader.

Newspaperts

Playable. It has some RTS elements to it but they all involve clicking the mouse as fast as you can.

City Fall

It's not quite game status yet but has some impressive 3D graphics. Supposedly when a building is destroyed there is an epic explosion. Controls are tricky to use and AI seems glitchy.

Bunnies on Board

The rules of the game are confusing and arbitrary but if you take the time to learn them it can be fun. Plays in your browser.

Not quite playable yet

• Interstellar - works in your web browser. Looks like it has some promise but there isn't much to do.
• Love Squad (Windows) - this game is hilarious right from the title screen. Unfortunately it ends right before the gameplay is about to start.
• All Your Base!... - training program seems to be working and then quits. Clicking stops working after a while.
• Monkeys vs Crocodiles (Windows) - nothing to do besides watch the bad guys kill you and then you.
• Booty Defense - Plants vs Zombies clone; no challenge yet.
• DEFEND THE CORN - mouse clicking game; not much to do.
• Cherrys To Cherrys: CompEdition (Windows/OSX/Unity) - glitchy and not really clear what to do.
• Generic Space RTS (Prototype) (Windows) - its own description says that it's not playable yet.
• Life & File (Windows/Unity) - unlabeled buttons and impossible gameplay.
• Lightning Fox Kick V: Doom Vortex - not clear what is happening. No strategy.
• BotWars - nothing to do. Tanks don't even move.
• Scarab - there is some gameplay here but it's unfinished and unfun.
• SyncMind - web/text based. You get game over if a 3 sided dice gets a 1. No fun.
• Sting Wars - author says on the description page it's not playable yet.
• The Unfinished - author says it's not ready yet.
• CommaWar - trouble installing, based on description does not look ready.
• Nocturne - no idea what to do. Controls don't seem to work. Something about multiplayer?
• Tower Defense - rudimentary. Some installation issues.

I did not get these to work

• 7dRTS - iforce2d - error while loading shared libraries: libGLEW.so.1.6: cannot open shared object file: No such file or directory - I do have GLEW installed.
• Eggz - spammy download mirror. Source code lacks setup instructions.
• Meanwhile, in Space... - spammy download mirror.
• Nebula Prime Ep1 - Linux version contains Windows exe file. Windows version depends on .NET framework.
• Overseer - Unity web player only.
• FrogForce - Windows only, depends on .NET framework.
• Capture Effect - Unity web player only.
• Polarity - Unity web player only.
• RTS - Unity web player only.
• Strat Souls - Unity web player only.
• Android Arena - Unity web player only.
• At the end - Unity web player only.
• IKON_COLONIZE - Unity web player only.
• Orchammer 1944 - Unity web player only.
• Quebec - Unity web player only.
• Ultipoly - source distribution only; written in an arcane language; no installation instructions.

System

Linux 3.8.0-27-generic #40-Ubuntu SMP Tue Jul 9 00:17:05 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

User agent

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/28.0.1500.71 Chrome/28.0.1500.71 Safari/537.3

wine

Version 1.4.1. No mono or .NET framework.

unity web player

Nope.

No Linux or web version

• Marmoreal: Fleur de Lis (Windows)
• Secrets of the divine -- WIP (Windows)
• Ambush (Windows)
• Defend the Famine (Windows)
• Flaggers (Windows)
• Flash of the Titans: Kallipso Nikon (Windows)
• Mobius (Windows/OSX/Unity)
• Combat Shock (Windows/OSX/Unity)
• Icarus (Windows)
• Lys kai (Windows)
• Monster Defence (Windows)
• SupremeOverlord Work In Progress (Windows)
• Warblocks (Windows/Unity)
• World's Aftermath (Windows)
• Honk Konk (Windows)
• infrontofaradar (Windows)
• Jumpstarter (Windows)
• Ninja Squad Commander (Windows/Unity)
• Pirate AaaaaRTS! (Windows)
• Protectors of the World Tree (Windows)
• Spawner Pawners (Windows)
• Substruct (Windows/Unity)
• TrashScape (Windows)

I have decided to participate in the 7-Day Real Time Strategy Challenge, July 2013 edition.

I am working with Michael Weber who will create sound effects, music, and be in charge of the "atmosphere" of the game.

Day 1

Cross-posted on ludumdare.com.

I am using chem, a canvas-based game engine I made for rapid development occasions such as this. It has been working out quite well and I think it has been playing a large role in my productivity today.

The game is codenamed "pillagers". The idea is to mix space physics with real time strategy and see what comes out. Instead of creating a carefully planned out base, you will be campaigning through levels, pillaging for resources.

That's the idea, anyway. We'll see how it pans out.

Here's a list of the things that work right now:

• Selecting squads and telling them to move around.
• Ships shoot enemy ships, destroying them when their health reaches 0.
• Scrolling around the map.
• Auto generated parallax background with stars and a planet.

The code is open source, hosted on GitHub. It's 941 lines of JavaScript:

   46 src/bullet.js
   33 src/explosion.js
  432 src/main.js
   19 src/militia_ship.js
  241 src/ship_ai.js
  165 src/ship.js
    5 src/uuid.js
  941 total

Screenshot of a squad of ships under attack:

Short video of me playing around with some elements of the game

It was a fun challenge to write the AI to get the ships to stop at the intended destinations. As is it's not optimal, but I think that's okay. Maybe later classes of ships will have better AI.

You can actually play this game right now. Since it's web based, you don't even have to download anything.

Here's what the TODO list looks like currently:

• Edit the main militia ship to have short range.
• Auto targeting enemies should only work when close enough.
• Ability to right click an enemy ship to tell squad to target it.
• Add enemy turrets and ships to level 1.
• Add enemy flag which grants victory when destroyed.
• Add 2nd class of ship which you get some of at beginning of level 1.
• Instead of giving the player ships at the beginning, give them cash and buildings which they can use to create the ships they want. The user will thus be able to choose how many of class 1 and class 2 ships they want.
• Put instruction label text in level 1 to explain the controls.
• Start planning level 2.

Well, I'm off to bed to get some rest. Looking forward to Day 2!

Day 2

Cross-posted on ludumdare.com.

opengameart.org has been very good to me. I think this will be my new go-to game dev art supply. So far I've found everything I've wanted on that site, except for a meteor sprite. Maybe I'll make one and contribute back.

Michael made a lazer sound and a ship thrusting sound, and I added the sound effects to the game. In addition to that, I made a ton of progress today:

• 3pm - draw flags only if selected
• 5pm - add another class of ship
• 8pm - found out I was using t * a ^ 2 + t * v for acceleration instead of a * t ^ 2 + v * t. I fixed all the math and made ships arriving at gather points nice and smooth.
• 9pm - added team colors overlayed on ships
• 10pm - gave the Militia ship an electric melee attack
• 11pm - gave Militia ships smart attacking AI
• 12am - made Ranger ships pursue targets
• 2am - added TurretShip and FlagShip
• 3am - added support for ships with backwards thrusters
• 3am - added move & engage command
• 3am - improved ship selection capabilities
• 5am - added title screen, credits screen, and game over screen. I made it possible to win and lose the game.

Here's the title screen:

Here's a screenshot of a battle underway:

Battles can get hectic and chaotic, but it is fun to watch it play out. Here's a video of me demoing a battle:

Again, you can playtest this game in your browser right now using the same url as Day 1. Feel free to give me feedback or advice.

I came up with some ideas on what bigger picture gameplay will look like which I am pretty pleased with. There will be a bunch of different classes of ships, for example:

Militia

Basic cheap unit. Has short range. Agile. Actually doesn't shoot lazers, shoots a short range but powerful lightning attack out the front. It must resort to charging other ships to attack them.

Ranger

Short range weak lazers. Less agile. Less defense. In a level, a Militia should be able to overtake a Ranger and kill it.

Artillery

Slower, shoots big lazers. Slow moving, weak defense.

FlagShip

Does not attack. Has high defense. Moves very slowly.

Turret

Stationary lazer shooter.

Medi

Heals other ships.

There will be some obvious deficiencies with the ships targeting systems and AI. These can be helped with upgrades, such as:

• Targeting
  • Target the first enemy found (default)
  • Target the closest enemy.
  • Communicate with others to divide up the targets evenly.
• Ranger
  • Upgrade range.
  • Update bullet count to 3.
  • Evasive maneuvers 1.
  • Evasive maneuvers 2.
  • Smarter aiming.
  • Backwards thrusters.
• Militia
  • Evade lazers.
  • Backwards thrusters.
  • Smarter aiming.

You will start with only a flagship, and as you progress throughout the campaign, you will start to build up a convoy that gets ever bigger and more powerful. You'll need it to be bigger and more powerful to get through later levels, in fact.

Here's what's next on the TODO list:

• Build simpler level 1 where you only have to navigate your flag ship around meteors.
• Give the player some cash and let them choose to build Ranger ships or Militia ships using the Flagship.
• Create level 2 where you have to destroy the enemy flagship and then fly your flagship through the created portal.

I'm at a pretty good checkpoint right now. I'm calling it a night.

Day 3

Cross-posted on ludumdare.com.

Today Michael gave me the first music track he composed for the game and I am impressed. I am excited to have professional sounding music for a change.

That being said, the first thing I did after inserting the music into the game was program a mute button so that I could keep listening to techno while I coded.

Michael also delivered an electric explosion sound effect that works perfectly.

I mentioned yesterday that I could not find a meteor graphic to use on opengameart.org. I actually did end up finding art that works really well by searching for "rock".

You can see the three rock types in the current spritesheet for the game which is autogenerated by chem:

The first thing I did today was create a electrical disintegration animation, and I'm pretty pleased with the result:

I wonder if this is considered good enough to submit to opengameart.org.

After I finished that animation, I worked hard and had a productive day:

• 3pm - Finished electrical disintegration animation and added sound effect to game.
• 3pm - Added in Michael's background music and made it so you can toggle it with M key.
• 5pm - Added meteors with collision detection.
• 5pm - Fixed "ships rotating for no reason" bug.
• 6pm - Inserted a new Level 1 - meteor field that you have to navigate through.
• 7pm - Created a spastic portal graphic and added it to the level.
• 7pm - Fixed navigation bug for ships equipped with backwards thrusters.
• 9pm - Fixed scrolling when manual overriding.
• 11pm - Added a UI pane when ships are selected.
• 12am - Added a mini-map.
• 1am - Made it so you can send ships into a portal.
• 3am - Made it so selected portal shows what's inside it.
• 4am - Added ability to send ships out of a portal.
• 5am - Added announcement support.
• 6am - Made your convoy show up on the level complete screen.
• 6am - Added stats to the level complete screen.
• 8am - Finished level complete screen so that you can get to the next level.

The source tree has grown quite a bit since last time I checked, up to 2,704 lines:

   49 src/bullet.js
   77 src/credits_screen.js
   43 src/flag_ship.js
   30 src/fx.js
   73 src/game.js
   34 src/game_over_screen.js
  178 src/level_complete_screen.js
   20 src/main.js
   81 src/meteor.js
   58 src/militia_ship.js
   91 src/physics_object.js
   82 src/portal.js
   60 src/ranger_ship.js
   37 src/sfx.js
  449 src/ship_ai.js
  177 src/ship.js
    7 src/ship_types.js
   57 src/squad.js
  933 src/state.js
   23 src/team.js
  105 src/title_screen.js
   35 src/turret_ship.js
    5 src/uuid.js
 2704 total

I actually feel pretty good about the code organization right now. I have only had to pause progress and refactor 2-3 times and each time it was mostly painless.

Enough of the technical stuff. Let's see some screenshots.

Here's one of the meteor field level you start out in:

Here's what it looks like when you finish the level:

And here's me giving a walkthrough of the progress made today:

My TODO list is getting a bit unruly. I've divided it into "next steps" and "nice-to-have"s:

Next steps:

• Add text instructions in Level 1 to explain the controls and how to beat the level.
• Make it so that you start out the next level with the same fleet that you exited with.
• Show your cash in the UI
• Allow you to create ships that you unlock by spending cash.
• Insert a level between 1 and 2 with some attackers. You'll have to build some Ranger ships to defend your Flagship.

Nice-to-haves:

• Figure out why the game slows to a crawl when many ships are added and then deleted.
• Experiment with speed cap on ships. (I'm reluctant to do this one - I like the idea of high-velocity dogfighting.)
• If Ranger is in range, don't accelerate toward target.
• Fix the thruster sound glitchiness
• Each command should draw something on the screen to indicate that something is commanded. (Some commands do this already.)
• When forming a squad, don't assume all ships have the same radius.

And now I must rest. I am exhausted.

Day 4

Cross-posted on ludumdare.com.

It has been wonderful relying only upon circles in this physics engine. The math is simple and beautiful, and it's easy to write fast code. Who needs polygons anyway?

Today was a good day. Pillagers is now actually a game. What I got done:

• 5pm - Added more of Michael's sound effects into the game.
• 5pm - Tweaked the physics.
• 5pm - Added some cheats to help speed up testing.
• 6pm - Added text in Level 1 to explain controls.
• 7pm - The game shows how much cash you have.
• 10pm - You keep your same fleet when progressing to the next level.
• 12am - Added a new Level 2.
• 12am - Made unlocking ships work.
• 12am - Made it so you get cash from killing enemy ships.
• 12am - Tweak the money system.
• 1am - Made it so you can scroll and give orders while paused.
• 1am - Fixed some game crashes and bugs.
• 2am - Modified attacking AI to stay within certain speed limits. Makes the game less chaotic.
• 3am - Added ability to skip tutorial levels.
• 4am - Better squad formation.
• 4am - Updated the Move & Engage command to be more effective.
• 6am - Added Civilian ships and plan out Level 4.
• 7am - Add Artillery ship and finish implementing Level 4.
• 8am - Fix the crash which happened at the end of the demo below.

Screenshot:

Screencast demo:

Bed.

Day 5

Cross-posted on ludumdare.com.

I spent a good chunk of the day trying to solve a performance problem. After approximately 100,000 bullets were fired, the framerate would drop to an excruciating 16 FPS.

I was able to figure it out by using Google Chrome's heap profiling tool. Here's what the heap snapshot looked like:

I found out that there was a memory leak in the game engine - every time a sprite was created it called setInterval but it did not call clearInterval when the sprite was deleted. Ever since I fixed the issue, FPS has stayed at a nice and smooth 60, no matter how many objects are created and destroyed.

Here are some other things I got done today:

• 9pm - Miscellanous small enhancements.
• 11pm - Fixed the performance issue.
• 12am - Added the new thruster sound that Michael gave me.
• 3am - Added Sandbox Mode.
• 4am - Added more features to Sandbox Mode.
• 6am - Added Ctrl+A to select all your ships. Also made WASD and J work the same as arrow keys and space.
• 7am - Added dogfighting mode.
• 8am - Added graphics so that ships display their targets when selected.
• 8am - Added 2 more dogfighting levels. Enhanced manual piloting of Militia ship so that you can use your melee attack.

Here's a screenshot of Sandbox mode:

Here's me playing through Dogfighting Mode and messing around in Sandbox Mode a bit:

I have only 2 days left to work. The last day will be primarily reserved for gameplay tweaking, bugfixes, touchups, and testing. That leaves only 1 more day to make actual progress.

These are my goals for tomorrow:

• Add a rotating turret to the flagship.
• Add a civilian to Level 1 and use that to explain the Move command vs the Engage command.
• Add a dogfighting level where there is an ongoing battle and you have to rack up a certain number of kills to win.
• Add more campaign levels. I reserve the right to ramp up the difficulty in the later levels!
• Pan sound effects and volume depending on scroll location
• Add some more ship classes and implement upgrades.

Thanks to the folks in the #7dRTS IRC channel for helping me playtest today. Specifically Zapa and Orava.

Day 6

Cross-posted on ludumdare.com.

Today I worked on making the level format and core engine more robust. I added a bunch more dogfighting levels. There are 11 now, and they get pretty crazy at the end.

The cool thing about them is that the core game engine does not even know that you are in "Dogfighting Mode". There's no dogfightingMode variable that tells the engine that's what you're doing. Instead, the level format supports trigger conditions, events, and groups, and the logic of dogfighting is done in the level JSON file.

Michael gave me the battle music today as well. It sounds amazing. I made it so that the game automatically detects when you are entering or exiting a battle and transitions the music appropriately. I'm pleased with the effect.

Here's a list of what I got done today:

• 12am - Added battle music. The game automatically detects when a battle is happening and fades in the battle music, then fades back to normal music once the battle is over.
• 1am - Added more dogfighting levels.
• 3am - Worked on making dogfighting levels more fun. You have to try again if you die rather than respawning.
• 4am - Made the Militia ship show the area of attack when you are manually piloting it.
• 6am - Sandbox Mode: Ability to load and save. I was able to use this to build some dogfighting levels.
• 7am - Added more dogfighting levels.

I did not accomplish most of what I set out to today. Regardless, I am really happy with Dogfighting Mode right now. It's almost certainly more fun than Campaign Mode currently.

Here's a video of me playing through it:

This version of the game is going to be pretty close to the final one that I submit for 7dRTS. I will probably only spend about 6-8 more hours on this debugging on Firefox and bugfixing. I will not be supporting Internet Explorer.

Day 7

Today was short. I added more sound effects that Michael provided, and lowered the difficulty on several levels. The game is in a stable state and I don't want to screw it up before submitting.

I experimented with adding aiming hints when manually piloting a ship, but I found that it actually made the game more difficult because it tricks the player into aiming directly at the enemy ships instead of compensating for relative velocity.

I tried to make my girlfriend play the game, but she said it was too hard. It probably is too hard. I like hard games.

Either way, I'm submitting.

View my entry on ludumdare.com.

Play the game in your web browser.

In JavaScript, we don't have private methods right? We must to resort to using this._somePrivateThing() right?

Wrong.

Before

function Cell(x, y) {
  this.x = x;
  this.y = y;
  this.things = [1, 2, 3];

  // I guess I'll have to use an underscore to indicate that this
  // method is private.
  this._initializeSomethingElse();
}

Cell.prototype._initializeSomethingElse = function() {
  this.dir = Math.atan2(this.y, this.x);
  this.total = 0;
  // hmm, I need a reference to this in that callback. I'll have to
  // save a copy or use bind
  var self = this;
  self.things.forEach(function(thing) {
    self.total += thing;
  });
};

After

function Cell(x, y) {
  this.x = x;
  this.y = y;
  this.things = [1, 2, 3];

  // boom. private method.
  initializeSomethingElse(this);
}

function initializeSomethingElse(self) {
  self.dir = Math.atan2(self.y, self.x);
  self.total = 0;
  self.things.forEach(function(thing) {
    self.total += thing;
  });
}

Notice that as an added benefit, the new private method is given an explicit reference to the instance, so if you need to use a callback you don't have to shuffle around the this pointer.

"But it will fool the optimizer!"

Wrong again.

Let's benchmark the above examples:

var PrivCell = require('./priv');
var NoPrivCell = require('./no-priv');

console.log("Test 1 - no priv method:", Math.round(test(NoPrivCell)) + "ms");
console.log("Test 1 - private method:", Math.round(test(PrivCell)) + "ms");

console.log("Test 2 - no priv method:", Math.round(test(NoPrivCell)) + "ms");
console.log("Test 2 - private method:", Math.round(test(PrivCell)) + "ms");

console.log("Test 3 - no priv method:", Math.round(test(NoPrivCell)) + "ms");
console.log("Test 3 - private method:", Math.round(test(PrivCell)) + "ms");

function test(Cell) {
  var start = new Date();
  var total = 0;
  for (var i = 0; i < 20000000; i += 1) {
    var c = new Cell();
    total += c.total;
  }
  return new Date() - start;
}

Running the benchmark on my machine:

Test 1 - no priv method: 5525ms
Test 1 - private method: 5537ms
Test 2 - no priv method: 5537ms
Test 2 - private method: 5571ms
Test 3 - no priv method: 5572ms
Test 3 - private method: 5595ms

Makes no difference. It's clean, it solves the problem, and it has no performance implications.

Here's a jsperf for further evidence.

Browser Scoping

Note that if you're writing the code in an environment which does not provide scoping, such as a <script> tag in the browser, you'll want to wrap the entire thing in an anonymous function call:

var Cell = (function() {
  function Cell(x, y) {
    this.x = x;
    this.y = y;
    this.things = [1, 2, 3];

    // boom. private method.
    initializeSomethingElse(this);
  }

  function initializeSomethingElse(self) {
    self.dir = Math.atan2(self.y, self.x);
    self.total = 0;
    self.things.forEach(function(thing) {
      self.total += thing;
    });
  }

  return Cell;
})();

Spot the Fail

It's time to have a little fun.

Sometimes when I'm programming, I do something so comically stupid that I feel the need to screenshot the code and share my facepalm moment with someone else.

There are a couple guidelines for my flavor of Spot the Fail:

Guidelines

• The fail should be spottable even if you do not know the context of the code.
• For trickier ones, the mouse or text cursor can optionally be nearby the fail to give a hint.

I have collected a few of these screenshots and explanations for your enjoyment.

1. Labyrinth

South, South. Should be North, South.

2. jax

Classic. Missing break on one of the cases.

3. jax

Bytes are not 4 bits long.

4. "Code from work"

Case 8 should be [0-7] not [0-8].

5. motrs

It's the cliché break statement again. Seriously, gotta watch out for that.

6. motrs

new T[newSizeX + m_sizeY * m_sizeZ] - the addition should be multiplication.

7. stinkomanlevels.com

D'oh! I asked for 1 more character than I should have.

8. motrs

m_tileCountY = value; instead of value should be tileCount.

9. menu item

Tried to press the "OK" button with ALt+O; instead messed up the shortcut.

10. solidcomposer.com

It's MIME TIME!!

11. solidcomposer.com

vim spotted the fail for me. Good job vim.

12. solidcomposer.com

state.user !============== null

13. repatriator

It's highlighted in red. Pretty silly.

14. repatriator

X and Y are swapped

15. Some Rails Code

syntax error: enr

16. motrs

"C:\out.bin" - accidentally escaping the 'o' but more importantly it's not even running on windows.

Fin.

Well, hope that was fun.

On a relevant note, remember when Quixey in an act of hiring PR offered $100 if you could spot the fail in 1 minute? Good times.

Statically Recompiling NES Games into Native Executables with LLVM and Go

I have always wanted to write an emulator. I made a half-hearted attempt in college but it never made it to a demo-able state. I also didn't want to write Yet Another Emulator. It should at least bring something new to the table. When I discovered LLVM, I felt like I finally had worthwhile idea for a project.

This article presents original research regarding the possibility of statically disassembling and recompiling Nintendo Entertainment System games into native executables. I attempt to bring the reader along from beginning to end in a sort of "let's build it together" fashion. I assume as little as possible about what the reader knows about the subjects at hand, without compromising the depth of the article.

See the project on GitHub

Table of Contents

1. Table of Contents
2. Discovering LLVM
3. NES Crash Course
4. Creating a Custom ABI for Testing Purposes
5. Assembly
6. Disassembly
7. Code Generation
8. Optimization
9. Layout of an NES ROM File
10. Challenge: Runtime Address Checks
11. Challenge: Parallel Systems Running Simultaneously
12. Challenge: Handling Interrupts
13. Challenge: Detecting a Jump Table
14. Challenge: Indirect Jumps
15. Challenge: Dirty Assembly Tricks
16. Video: Playing Super Mario Brothers 1 on Native Machine Code
17. Unsolved Challenge: Mappers
18. Community Support
19. Conclusion

Discovering LLVM

I became giddy with excitement when I discovered LLVM. Generate code in LLVM IR format, and LLVM can optimize and generate native code for any of its supported backends. Here's an incomplete list:

• ARM
• STI CBEA Cell SPU [experimental]
• C++ backend
• Hexagon
• MBlaze
• Mips
• Mips64 [experimental]
• Mips64el [experimental]
• Mipsel
• MSP430 [experimental]
• NVIDIA PTX 32-bit
• NVIDIA PTX 64-bit
• PowerPC 32
• PowerPC 64
• Sparc
• Sparc V9
• Thumb
• 32-bit X86: Pentium-Pro and above
• 64-bit X86: EM64T and AMD64
• XCore

Furthermore, you can write additional backends to support additional targets. For example, emscripten provides a JavaScript backend.

To seal the deal, the LLVM project offers clang, a C, C++, Objective C, and Objective C++ front-end for LLVM. This means that because of the way LLVM is designed, emscripten allows you to compile C, C++, Objective C, and Objective C++ code for the browser. Wow!

Just to show you how refreshing this technology is, have a look at the following C code:

#include <stdio.h>
#include <stdint.h>

int main() {
    uint8_t foo = 0xf0;
    if (foo & 0x80 == 0x80) {
        for (int i = 1; i <= 10; ++i) {
            printf("%d\n", i);
        }
    }
}

Simple enough. Assign 11110000 to foo. If the highest bit is 1 (which it is), print the integers from 1 to 10.

Compiling with gcc with default settings, we get:

$ gcc test.c
test.c: In function ‘main’:
test.c:7:5: error: ‘for’ loop initial declarations are only allowed in C99 mode
test.c:7:5: note: use option -std=c99 or -std=gnu99 to compile your code
$

Right, so default gcc doesn't have the nice things that C99 brings. Let's try that again with gcc -std=c99:

$ c99 test.c
$ ./a.out
$

Okay, so now it compiled and ran, but why didn't it print the integers from 1 to 10? Let's try compiling with clang and see what happens:

$ clang test.c
test.c:6:13: warning: & has lower precedence than ==; == will be evaluated
      first [-Wparentheses]
    if (foo & 0x80 == 0x80) {
            ^~~~~~~~~~~~~~
test.c:6:13: note: place parentheses around the '==' expression to silence
      this warning
    if (foo & 0x80 == 0x80) {
            ^
              (           )
test.c:6:13: note: place parentheses around the & expression to evaluate it
      first
    if (foo & 0x80 == 0x80) {
            ^
        (         )
1 warning generated.
$

Aha! In C, & has lower precedence than ==. In addition to outputting with fancy terminal colors, clang found the bug in our program and issued a warning. Once we fix that up and run the code again, we get the expected behavior.

Note: It has been shown to me that gcc -Wall produces a similar warning for the same code example here. I apologize for the bogus example. However, my point still stands. LLVM has a much better explanation of the philosophy and differences in error messages between clang and gcc.

There are many instances where cryptic gcc errors in your code are made clear by clang. In the words of my friend Josh Wolfe,

Clang makes gcc look like a rusty old grandpa.

This is exciting because clang is merely a front-end to LLVM. What this means is that if you generate LLVM IR code, you share the same code generation backend as clang. With this I felt that powerful optimization and wide target platform support were within my grasp.

What if we could translate NES assembly code into LLVM IR code? We could completely, statically recompile NES games into native binaries!

NES Crash Course

Take a break from LLVM for a moment and consider the Nintendo Entertainment System.

The NES consists of 4 systems working together in parallel:

• CPU - 8-bit processor. Basically a MOS 6502.
• Picture Processing Unit
• Audio Processing Unit
• Usually a mapper: custom hardware used to provide extra ROM (and sometimes other stuff)

The NES uses memory-mapped I/O, which means that, to interact with hardware, you read from and write to special memory addresses. Take a look at a simplified version of the memory layout of the NES at runtime:

Let's break this down. First, note that the entire addressable range of the NES is 64KB. Thus all addresses can be represented by 2 bytes, or 4 hex digits. Next, note that in 6502 assembly, $ is standard for hexadecimal notation. So the address range is $0000 to $ffff. Of the 64KB addressable range, 29KB is (usually) completely unused!

Note the $8000 - $ffff range. When you create an NES game, you write 6502 assembly code, which is assembled into a 32KB binary, which is loaded into the $8000 - $ffff range during bootup.

Let's write some example 6502 code to illustrate.

Creating a Custom ABI for Testing Purposes

One of the challenges of NES development is that a basic "Hello, World" program is pretty complicated. This is why I created a custom Application Binary Interface to make it a little bit easier. Refer back to the memory layout above, and recall that many addresses are unused. I decided to add a couple things:

With this add-on ABI, I was able to create a simple 6502 application that prints "Hello, World!\n" to stdout and then exits. In 6502 land, that looks like:

; Remember that this code gets loaded at memory address $8000.
; This instruction tells the assembler that the code and data
; that follow are starting at address $8000.
; For example, the 'W' character in the "Hello, World!\n\0" text
; below is located at memory address $8007.
.org $8000

; This is a label. The assembler will replace references to
; this label with the actual address that it represents, which
; in this case is $8000.
msg:
; 'db' stands for "Data Bytes". This line puts the string
; "Hello, World!\n\0" in bytes $8000 - $800e when this
; code is assembled.
  .db "Hello, World!", 10, 0

Reset_Routine:
  LDX #$00       ; put the starting index, 0, in register X

loop:
; This instruction does 3 things:
; 1. Take the address of msg and add the value of register X to get a pointer.
; 2. Read the byte at the pointer and put it into register A
; 3. Set the zero flag to 1 if A is zero; or 0 if A is nonzero.
  LDA msg, X     ; read 1 char
; If the zero flag is 1, goto loopend. Otherwise, go to the next instruction.
  BEQ loopend    ; end loop if we hit the \0
; Put the value of register A into memory address $2008. But remember,
; with our custom ABI, this means write the value of register A to stdout.
  STA $2008      ; putchar
; Register X = Register X + 1
  INX
  JMP loop       ; goto loop

loopend:

  LDA #$00       ; put the return code, 0, in register A
  STA $2009      ; exit with a return code of register A (0 in this case)

; These are interrupt handlers. Let's not think about interrupts yet.
IRQ_Routine: RTI ; do nothing
NMI_Routine: RTI ; do nothing

; Another org statement. Remember that NES programs are 32KB. This tells the assembler
; to fill the unused space up until $fffa with zeroes and that we are now
; specifying the data at $fffa.
.org   $fffa

; 'dw' stands for "Data Words". These lines put the address of the NMI_Routine
; label at $fffa, the address of Reset_Routine at $fffc, and the address
; of IRQ_Routine at $fffe.
.dw  NMI_Routine
.dw  Reset_Routine
.dw  IRQ_Routine

There are some things to point out here. First, note the entry points into the program. Whereas in most programming environments, there is a main function, in NES games, the data at $fffa - $ffff has hardcoded special meaning. When a game starts, the NES reads the 2 bytes at $fffc and $fffd and uses that as the memory address of the first instruction to run.

Next let's talk about the CPU. In the code snippet above I used register A and register X. There are 6 registers total:

After coming up with this well-defined task - get "Hello World" running natively - it is time to write some code. First things first - this project should be able to assemble our source code into binary machine code format.

Assembly

One tried and true method to parsing source code is to use a lexer to tokenize the source, and then a parser to turn the tokens into an Abstract Syntax Tree. From there we can process the AST and turn it into the 32KB binary payload.

In Go land, this is straightforward thanks to nex, a lexer, and go tool yacc, a parser that is bundled with Go. Here is the parser code that can build an AST for our "Hello, World" example:

%{
package jamulator

import (
	"fmt"
	"strconv"
	"container/list"
)

// Define the structures that make up the abstract syntax tree.

// Example:
//     Label1234:
type LabelStatement struct {
	LabelName string
	Line int
}

// Example:
//     Label1234: BRK
type LabeledStatement struct {
	Label *LabelStatement
	Stmt interface{}
}

// Example:
//     .org $8000
type OrgPseudoOp struct {
	Value int
	Fill byte
	Line int
}

type InstructionType int
const (
	// Example:
	//     ADC #$44
	ImmediateInstruction InstructionType = iota

	// Example:
	//     INX
	ImpliedInstruction

	// Example:
	//     ADC Label1234, X
	DirectWithLabelIndexedInstruction

	// Example:
	//     ADC $44, X
	DirectIndexedInstruction

	// Example:
	//     ADC Label1234
	DirectWithLabelInstruction

	// Example:
	//     ADC $44
	DirectInstruction

	// Example:
	//     ADC ($44, X)
	IndirectXInstruction

	// Example:
	//     ADC ($44), Y
	IndirectYInstruction

	// Example:
	//     JMP ($81cc)
	IndirectInstruction
)

type Instruction struct {
	Type InstructionType
	OpName string
	Line int

	// not all fields are used by all instruction types.
	Value int
	LabelName string
	RegisterName string

	// filled in later
	OpCode byte
	Offset int
	Payload []byte
}

type DataStmtType int
const (
	// Example:
	//     .db $44
	ByteDataStmt DataStmtType = iota

	// Example:
	//     .dw Label1234
	WordDataStmt
)

type DataStatement struct {
	Type DataStmtType
	dataList *list.List
	Line int

	// filled in later
	Offset int
	Payload []byte
}


type IntegerDataItem int
type StringDataItem string
type LabelCall struct {
	LabelName string
}
type ProgramAst struct {
	List *list.List
}

// This is the root node of the abstract syntax tree.
var programAst ProgramAst
%}

// This instructs yacc how to generate a structure that will fit
// any data type that we need to use for a node.
%union {
	integer int
	str string
	list *list.List
	orgPsuedoOp *OrgPseudoOp
	node interface{}
}

// These are the nodes that we build based on other nodes and tokens.
%type <list> statementList
%type <node> statement
%type <node> instructionStatement
%type <node> dataStatement
%type <list> dataList
%type <list> wordList
%type <node> dataItem
%type <orgPsuedoOp> orgPsuedoOp
%type <node> numberExpr
%type <node> numberExprOptionalPound

// These are tokens that we will encounter when reading the output from
// the lexer.
%token <str> tokIdentifier
%token <integer> tokInteger
%token <str> tokQuotedString
%token tokEqual
%token tokPound
%token tokDot
%token tokComma
%token tokNewline
%token tokDataByte
%token tokDataWord
%token tokProcessor
%token tokLParen
%token tokRParen
%token tokDot
%token tokColon
%token tokOrg

%%

// An assembly program is defined as a statementList.

programAst : statementList {
	programAst = ProgramAst{$1}
}

// A statementList is defined as another statementList, plus a single
// statement, or simply as a statement. This creates a linked list
// of statements.
statementList : statementList tokNewline statement {
	if $3 == nil {
		$$ = $1
	} else {
		$$ = $1
		$$.PushBack($3)
	}
} | statement {
	if $1 == nil {
		$$ = list.New()
	} else {
		$$ = list.New()
		$$.PushBack($1)
	}
}

// This defines a statement, which can be many things.

statement : tokDot tokIdentifier instructionStatement {
	$$ = &LabeledStatement{
		&LabelStatement{"." + $2, parseLineNumber},
		$3,
	}
} | tokIdentifier tokColon instructionStatement {
	$$ = &LabeledStatement{
		&LabelStatement{$1, parseLineNumber},
		$3,
	}
} | orgPsuedoOp {
	$$ = $1
} | instructionStatement {
	$$ = $1
} | tokDot tokIdentifier dataStatement {
	$$ = &LabeledStatement{
		&LabelStatement{"." + $2, parseLineNumber},
		 $3,
	 }
} | tokIdentifier tokColon dataStatement {
	$$ = &LabeledStatement{
		&LabelStatement{$1, parseLineNumber},
		$3,
	}
} | dataStatement {
	$$ = $1
} | tokIdentifier tokColon {
	$$ = &LabelStatement{$1, parseLineNumber}
} | {
	// empty statement
	$$ = nil
}

dataStatement : tokDataByte dataList {
	$$ = &DataStatement{
		Type: ByteDataStmt,
		dataList: $2,
		Line: parseLineNumber,
	}
} | tokDataWord wordList {
	$$ = &DataStatement{
		Type: WordDataStmt,
		dataList: $2,
		Line: parseLineNumber,
	}
}

wordList : wordList tokComma numberExprOptionalPound {
	$$ = $1
	$$.PushBack($3)
} | numberExprOptionalPound {
	$$ = list.New()
	$$.PushBack($1)
}

dataList : dataList tokComma dataItem {
	$$ = $1
	$$.PushBack($3)
} | dataItem {
	$$ = list.New()
	$$.PushBack($1)
}

numberExpr : tokPound tokInteger {
	tmp := IntegerDataItem($2)
	$$ = &tmp
} | tokIdentifier {
	$$ = &LabelCall{$1}
}

numberExprOptionalPound : numberExpr {
	$$ = $1
} | tokInteger {
	tmp := IntegerDataItem($1)
	$$ = &tmp
}

dataItem : tokQuotedString {
	tmp := StringDataItem($1)
	$$ = &tmp
} | numberExprOptionalPound {
	$$ = $1
}

orgPsuedoOp : tokOrg tokInteger {
	$$ = &OrgPseudoOp{$2, 0xff, parseLineNumber}
} | tokOrg tokInteger tokComma tokInteger {
	if $4 > 0xff {
		yylex.Error("ORG directive fill parameter must be a single byte.")
	}
	$$ = &OrgPseudoOp{$2, byte($4), parseLineNumber}
}

instructionStatement : tokIdentifier tokPound tokInteger {
	$$ = &Instruction{
		Type: ImmediateInstruction,
		OpName: $1,
		Value: $3,
		Line: parseLineNumber,
	}
} | tokIdentifier {
	$$ = &Instruction{
		Type: ImpliedInstruction,
		OpName: $1,
		Line: parseLineNumber,
	}
} | tokIdentifier tokIdentifier tokComma tokIdentifier {
	$$ = &Instruction{
		Type: DirectWithLabelIndexedInstruction,
		OpName: $1,
		LabelName: $2,
		RegisterName: $4,
		Line: parseLineNumber,
	}
} | tokIdentifier tokInteger tokComma tokIdentifier {
	$$ = &Instruction{
		Type: DirectIndexedInstruction,
		OpName: $1,
		Value: $2,
		RegisterName: $4,
		Line: parseLineNumber,
	}
} | tokIdentifier tokIdentifier {
	$$ = &Instruction{
		Type: DirectWithLabelInstruction,
		OpName: $1,
		LabelName: $2,
		Line: parseLineNumber,
	}
} | tokIdentifier tokInteger {
	$$ = &Instruction{
		Type: DirectInstruction,
		OpName: $1,
		Value: $2,
		Line: parseLineNumber,
	}
} | tokIdentifier tokLParen tokInteger tokComma tokIdentifier tokRParen {
	if $5 != "x" && $5 != "X" {
		yylex.Error("Register argument must be X.")
	}
	$$ = &Instruction{
		Type: IndirectXInstruction,
		OpName: $1,
		Value: $3,
		Line: parseLineNumber,
	}
} | tokIdentifier tokLParen tokInteger tokRParen tokComma tokIdentifier {
	if $6 != "y" && $6 != "Y" {
		yylex.Error("Register argument must be Y.")
	}
	$$ = &Instruction{
		Type: IndirectYInstruction,
		OpName: $1,
		Value: $3,
		Line: parseLineNumber,
	}
} | tokIdentifier tokLParen tokInteger tokRParen {
	$$ = &Instruction{
		Type: IndirectInstruction,
		OpName: $1,
		Value: $3,
		Line: parseLineNumber,
	}
}

%%

And here is the lexer code that generates the tokens listed above:

/\.[dD][bB]/ {
	return tokDataByte
}
/\.[dD][wW]/ {
	return tokDataWord
}
/\.[oO][rR][gG]/ {
	return tokOrg
}
/"[^"\n]*"/ {
	t := yylex.Text()
	lval.str = t[1:len(t)-1]
	return tokQuotedString
}
/[a-zA-Z][a-zA-Z_.0-9]*/ {
	lval.str = yylex.Text()
	return tokIdentifier
}
/%[01]+/ {
	binPart := yylex.Text()[1:]
	n, err := strconv.ParseUint(binPart, 2, 16)
	if err != nil {
		yylex.Error("Invalid binary integer: " + binPart)
	}
	lval.integer = int(n)
	return tokInteger
}
/\$[0-9a-fA-F]+/ {
	hexPart := yylex.Text()[1:]
	n, err := strconv.ParseUint(hexPart, 16, 16)
	if err != nil {
		yylex.Error("Invalid hexademical integer: " + hexPart)
	}
	lval.integer = int(n)
	return tokInteger
}
/[0-9]+/ {
	n, err := strconv.ParseUint(yylex.Text(), 10, 16)
	if err != nil {
		yylex.Error("Invalid decimal integer: " + yylex.Text())
	}
	lval.integer = int(n)
	return tokInteger
}
/=/ {
	return tokEqual
}
/:/ {
	return tokColon
}
/#/ {
	return tokPound
}
/\./ {
	return tokDot
}
/,/ {
	return tokComma
}
/\(/ {
	return tokLParen
}
/\)/ {
	return tokRParen
}
/[ \t\r]/ {
	// ignore whitespace
}
/;[^\n]*\n/ {
	// ignore comments
	parseLineNumber += 1
	return tokNewline
}
/\n+/ {
	parseLineNumber += len(yylex.Text())
	return tokNewline
}
/./ {
	yylex.Error(fmt.Sprintf("Unexpected character: %q", yylex.Text()))
}

//

package jamulator

import (
	"strconv"
	"os"
	"fmt"
)

var parseLineNumber int
var parseFilename string
var parseErrors ParseErrors

type ParseErrors []string

func (errs ParseErrors) Error() string {
	return strings.Join(errs, "\n")
}

func Parse(reader io.Reader) (ProgramAst, error) {
	parseLineNumber = 1

	lexer := NewLexer(reader)
	yyParse(lexer)
	if len(parseErrors) > 0 {
		return ProgramAst{}, parseErrors
	}
	return programAst, nil
}

func ParseFile(filename string) (ProgramAst, error) {
	parseFilename = filename

	fd, err := os.Open(filename)
	if err != nil { return ProgramAst{}, err }
	programAst, err := Parse(fd)
	err2 := fd.Close()
	if err != nil { return ProgramAst{}, err }
	if err2 != nil { return ProgramAst{}, err2 }
	return programAst, nil
}

func (yylex Lexer) Error(e string) {
	s := fmt.Sprintf("%s line %d %s", parseFilename, parseLineNumber, e)
	parseErrors = append(parseErrors, s)
}

Let's demonstrate our ability to parse the source code by printing and inspecting the abstract syntax tree.

import (
	"fmt"
	"reflect"
	"os"
)

func astPrint(indent int, n interface{}) {
	for i := 0; i < indent; i++ {
		fmt.Print(" ")
	}
	fmt.Println(reflect.TypeOf(n))
}

func (ast ProgramAst) Print() {
	for e := ast.List.Front(); e != nil; e = e.Next() {
		astPrint(0, e.Value)
		switch t := e.Value.(type) {
		case *LabeledStatement:
			astPrint(2, t.Label)
			astPrint(2, t.Stmt)
		case *DataStatement:
			for de := t.dataList.Front(); de != nil; de = de.Next() {
				astPrint(2, de.Value)
			}
		}
	}
}
func main() {
	programAst, err := jamulator.ParseFile("hello.asm")
	if err != nil {
		fmt.Fprintf(os.Stderr, "%s\n", err.Error())
		os.Exit(1)
	}
	programAst.Print()
}

Now we need to compile this code. At this point, we can no longer rely on Go to build everything because we now have 2 generated Go files:

The most straightforward way to build the project at this point is to use a Makefile. Simple enough:

build: jamulator/y.go jamulator/asm6502.nn.go
	go build -o jamulate main.go

jamulator/y.go: jamulator/asm6502.y
	go tool yacc -o jamulator/y.go -v /dev/null jamulator/asm6502.y

jamulator/asm6502.nn.go: jamulator/asm6502.nex
	${GOPATH}/bin/nex -e jamulator/asm6502.nex

clean:
	rm -f jamulator/asm6502.nn.go
	rm -f jamulator/y.go
	rm -f jamulate

.PHONY: build clean

Now we can build the project with one command:

$ make
go tool yacc -o jamulator/y.go -v /dev/null jamulator/asm6502.y
/home/andy/golang/bin/nex -e jamulator/asm6502.nex
go build -o jamulate main.go

And, sure enough, when we run this code, we get a nice breakdown of the source:

$ ./jamulate
*jamulator.OrgPseudoOp
*jamulator.LabelStatement
*jamulator.DataStatement
  *jamulator.StringDataItem
  *jamulator.IntegerDataItem
  *jamulator.IntegerDataItem
*jamulator.LabelStatement
*jamulator.Instruction
*jamulator.LabelStatement
*jamulator.Instruction
*jamulator.Instruction
*jamulator.Instruction
*jamulator.Instruction
*jamulator.Instruction
*jamulator.LabelStatement
*jamulator.Instruction
*jamulator.Instruction
*jamulator.LabeledStatement
  *jamulator.LabelStatement
  *jamulator.Instruction
*jamulator.LabeledStatement
  *jamulator.LabelStatement
  *jamulator.Instruction
*jamulator.OrgPseudoOp
*jamulator.DataStatement
  *jamulator.LabelCall
*jamulator.DataStatement
  *jamulator.LabelCall
*jamulator.DataStatement
  *jamulator.LabelCall

Once we have this AST, building a binary is a cinch:

1. Loop over the AST and compute the byte offset for each instruction.
2. Use the computed byte offsets to resolve the instructions with labels.
3. Final pass to write the payload to the disk.

Disassembly

Now we need to work backwards - we have the binary payload and we want to get the source. There is a challenge here. Remember that in assembly programs, we can insert arbitrary data with .db or .dw statements alongside instructions. In order to disassemble effectively, we have to be able to figure out what is "data" and what is "code".

One possible technique is to emulate the assembly program, and then record the ways in which memory addresses are accessed. After playing a game for a while, you would have a pretty good record of exactly which sections are data and which are code. I decided not to use this technique, however, since the goal of this project is static recompilation. I want to explore just how much is possible to do at compile-time.

So what can we do?

First, recall that the last 6 bytes in NES programs are 3 memory addresses which are the 3 entry points into the program:

.org $fffa
    .dw NMI_Routine
    .dw Reset_Routine
    .dw IRQ_Routine

Given this, a workable strategy becomes clear:

1. Create an AST where every single byte is a single .db statement.
2. Replace the .db statements at $fffa and $fffb with a .dw statement which references an NMI_Routine label.
3. Calculate the address that the .db statements at $fffa and $fffb were referring to, and insert a LabelStatement with the NMI_Routine label before the .db statement at that address.
4. Mark the .db statement at that address as an instruction.

When I say "mark as an instruction", what I mean is that we should interpret the .db statement at that location as an op code, and then use that to replace the following .db statements as part of the instruction as necessary. Then, based on the instruction, we want to recursively mark other locations as instructions:

The instructions that start with "B" are all conditional branch instructions. This means that they test some condition, and then either transfer control flow to the next instruction, or to a different label. This means that we can mark the possible branch address and the next address as instructions.

JSR stands for "Jump to SubRoutine". This will transfer control to a target address and then later when the RTS ("ReTurn from Subroutine") instruction is reached, continue execution where the JSR instruction left off.

It is possible for assembly programmers to use JSR and then inside the subroutine, do tricks with the stack to return to a different location. This is a problem that will be tackled later. It's not an issue with our "Hello World" example.

RTI, RTS, and BRK modify control flow, but the destination address is not constant, so these instructions do not help us know what else to mark as instructions.

As seen in this table, there are 2 types of JMP instructions: absolute and indirect:

The instructions in the earlier table are the only ones that modify control flow. All other instructions execute serially. Thus if we encounter one of these, we can reliably decode the next byte as an instruction.

Here's what it looks like when we apply this algorithm to our Hello World binary code:

.org $8000
Label_8000:
    .db $48
    .db $65
    .db $6c
    .db $6c
    .db $6f
    .db $2c
    .db $20
    .db $57
    .db $6f
    .db $72
    .db $6c
    .db $64
    .db $21
    .db $0a
    .db $00
Reset_Routine:
    LDX #$00
Label_8011:
    LDA Label_8000, X
    BEQ Label_801d
    STA $2008
    INX
    JMP Label_8011
Label_801d:
    LDA #$00
    STA $2009
IRQ_Routine:
    RTI
NMI_Routine:
    RTI
    .db $ff

    ...
    (this is repeated about 30,000 times)
    ...

    .db $ff
    .dw NMI_Routine
    .dw Reset_Routine
    .dw IRQ_Routine

This technique was able to decode all of the instructions, but we can't read the text, and it's pretty annoying having $ff - the filler byte value - repeated so many times. Let's add a pass to detect ASCII strings. We can do this by counting how many characters in a .db statement in a row are considered "ASCII" and when a threshold of 4 is reached, replace the .db statements with a quoted string:

.org $8000
Label_8000:
    .db "Hello, World!"
    .db $0a
    .db $00
Reset_Routine:
    LDX #$00
Label_8011:
    LDA Label_8000, X
    BEQ Label_801d
    STA $2008
    INX
    JMP Label_8011
Label_801d:
    LDA #$00
    STA $2009
IRQ_Routine:
    RTI
NMI_Routine:
    RTI
    .db $ff
    .db $ff

    ...
    (repeated about 30,000 times)
    ...

    .db $ff
    .db $ff
    .dw NMI_Routine
    .dw Reset_Routine
    .dw IRQ_Routine

Better! Let's solve the problem of the repeating $ff by adding a pass to detect where it would make sense to place .org statements. We can do this in much the same way as ASCII detection, but in this case we will look out for repeating bytes rather than bytes which fit in the ASCII range. 64 seems like a good threshold. If a byte repeats 64 times, replace all the repeated occurences with a .org statement:

.org $8000
Label_8000:
    .db "Hello, World!"
    .db $0a
    .db $00
Reset_Routine:
    LDX #$00
Label_8011:
    LDA Label_8000, X
    BEQ Label_801d
    STA $2008
    INX
    JMP Label_8011
Label_801d:
    LDA #$00
    STA $2009
IRQ_Routine:
    RTI
NMI_Routine:
    RTI
.org $fffa
    .dw NMI_Routine
    .dw Reset_Routine
    .dw IRQ_Routine

And finally, one minor detail. Let's do a final pass to collapse .db statements together:

.org $8000
Label_8000:
    .db "Hello, World!", $0a, $00
Reset_Routine:
    LDX #$00
Label_8011:
    LDA Label_8000, X
    BEQ Label_801d
    STA $2008
    INX
    JMP Label_8011
Label_801d:
    LDA #$00
    STA $2009
IRQ_Routine:
    RTI
NMI_Routine:
    RTI
.org $fffa
    .dw NMI_Routine
    .dw Reset_Routine
    .dw IRQ_Routine

Not bad. This is as close as we can get to the original source. It's impossible to know what label names were used, but we can give good names to the interrupt vectors. We just turned a binary machine code program into human-readable assembly.

Now that we can figure out the assembly source code from 6502 machine code, we can start the fun part - converting the assembly program into native machine code.

Code Generation

Our code generation code will generate an LLVM bitcode module. We can then use llc to compile the bitcode into an object file, the same as if we used gcc -c module.c and looked at the resulting module.o.

LLVM is written in C++, but it also exposes a C interface. This means we can integrate cleanly using cgo. In fact, Andrew Wilkins maintains a convenient Go module called gollvm which gives us seamless integration.

At any time we can debug the LLVM module we are generating by calling module.Dump() which prints the LLVM IR code for the module to stderr. Let's start by manually creating the IR code that we want to generate for Hello World, so we know what to work toward. We can get a head start by writing it in C and using clang to generate the IR code for us:

#include <stdio.h>

char * msg = "Hello, World!\n";

int main() {
    char * ptr = msg;
    while (*ptr) {
        putchar(*ptr);
        ++ptr;
    }
}

$ clang -emit-llvm -S test.c
$

Now looking at test.s:

; ModuleID = 'test.c'
target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
target triple = "x86_64-pc-linux-gnu"

@.str = private unnamed_addr constant [15 x i8] c"Hello, World!\0A\00", align 1
@msg = global i8* getelementptr inbounds ([15 x i8]* @.str, i32 0, i32 0), align 8

define i32 @main() nounwind uwtable {
  %1 = alloca i32, align 4
  %ptr = alloca i8*, align 8
  store i32 0, i32* %1
  %2 = load i8** @msg, align 8
  store i8* %2, i8** %ptr, align 8
  br label %3

; <label>:3                                       ; preds = %7, %0
  %4 = load i8** %ptr, align 8
  %5 = load i8* %4, align 1
  %6 = icmp ne i8 %5, 0
  br i1 %6, label %7, label %14

; <label>:7                                       ; preds = %3
  %8 = load i8** %ptr, align 8
  %9 = load i8* %8, align 1
  %10 = sext i8 %9 to i32
  %11 = call i32 @putchar(i32 %10)
  %12 = load i8** %ptr, align 8
  %13 = getelementptr inbounds i8* %12, i32 1
  store i8* %13, i8** %ptr, align 8
  br label %3

; <label>:14                                      ; preds = %3
  %15 = load i32* %1
  ret i32 %15
}

declare i32 @putchar(i32)

Alright - that looks a bit different than the code we're going to generate, but it's a good start. If that looks complicated, don't worry - we're going to do 3 things to make it less so:

1. Read up on the language reference.
2. Delete the stuff that seems unnecessary and see if it still works.
3. Modify the code to look like what we want to generate.

Here's an updated version with inline comments breaking it down:

; Here we declare the text that we will print.

; "private" means that only this module can see it - we do not export this
; symbol. Always declare private when possible. There are optimizations to be
; had when a symbol is not exported.

; "constant" means that this data is read-only. Again use constant when
; possible so that optimization passes can take advantage of this fact.

; [15 x i8] is the type of this data. i8 means an 8-bit integer.
@msg = private constant [15 x i8] c"Hello, World!\0a\00"


; Here we declare a dependency on the `putchar` symbol.

; When this module is linked, `putchar` must be defined somewhere, and with
; this signature.
declare i32 @putchar(i32)


; Same thing for `exit`.

; `noreturn` indicates that we do not expect this function to return. It will
; end the process, after all.

; `nounwind` has to do with LLVM's error handling model. We use `nounwind`
; because we know that `exit` will not throw an exception.
declare void @exit(i32) noreturn nounwind


; Note that we will be performing the final link step with gcc, which will
; automatically statically link against libc. This will provide the `putchar`
; and `exit` symbols, as well as set up the executable entry point to call `main`.
define i32 @main() {

; This label statement indicates the start of a basic block.
Entry:

; Here we allocate some variables on the stack. These are X, Y, and A,
; 3 of the 6502's 8-bit registers.
      %X = alloca i8
      %Y = alloca i8
      %A = alloca i8

; Note that here we are allocating variables which are single bits.
; These represent 2 of the bits from the status register.
; After this source listing there is a table explaining each bit
; of the status register.
      %S_neg = alloca i1
      %S_zero = alloca i1

; Send control flow to the Reset_Routine basic block.
      br label %Reset_Routine

Reset_Routine:

; This is the code to generate for
; LDX #$00

  ; Store 0 in the X register.
      store i8 0, i8* %X

  ; Clear the negative status bit, because we just stored 0 in X,
  ; and 0 is not negative.
      store i1 0, i1* %S_neg

  ; Set the zero status bit, because we just stored 0 in X.
      store i1 1, i1* %S_zero


      br label %Label_loop

Label_loop:

; This is the code to generate for
; LDA msg, X

  ; Load the value of X into %0.
      %0 = load i8* %X

  ; Get a pointer to the character in msg indexed by %0, which contains the
  ; value of X.
      %1 = getelementptr [15 x i8]* @msg, i64 0, i8 %0

  ; Read a byte of memory located at the pointer we just computed into %2.
      %2 = load i8* %1

  ; Store the byte we just loaded into %A, which is the variable we have
  ; allocated for A.
      store i8 %2, i8* %A

  ; Now we need to set the negative status bit correctly.

  ; The byte of memory we just loaded into %A is negative if
  ; and only if the highest bit is set.

  ; Perform a bitwise AND with 1000 0000.
      %3 = and i8 128, %2

  ; Test if the result is equal to 1000 0000.
      %4 = icmp eq i8 128, %3

  ; Save the answer to the negative status bit.
      store i1 %4, i1* %S_neg

  ; Now we need to set the zero status bit correctly.

  ; Test if the byte is equal to zero.
      %5 = icmp eq i8 0, %2

  ; Store the answer to the zero status bit.
      store i1 %5, i1* %S_zero


; This is the code to generate for
; BEQ loopend
      %6 = load i1* %S_zero

  ; If zero bit is set, go to Label_loopend. Otherwise, go to AutoLabel_0
      br i1 %6, label %Label_loopend, label %AutoLabel_0

AutoLabel_0:

; This is the code to generate for
; STA $2008
      %7 = load i8* %A

  ; Convert the 8-bit integer that we just loaded from A into
  ; a 32-bit integer to match the signature of `putchar`.
      %8 = zext i8 %7 to i32

      %9 = call i32 @putchar(i32 %8)
      br label %AutoLabel_1
AutoLabel_1:
; This is the code to generate for
; INX
      %10 = load i8* %X
      %11 = add i8 %10, 1
      store i8 %11, i8* %X

  ; Set the negative status bit correctly.
      %12 = and i8 128, %11
      %13 = icmp eq i8 128, %12
      store i1 %13, i1* %S_neg

  ;Set the zero status bit correctly.
      %14 = icmp eq i8 0, %11
      store i1 %14, i1* %S_zero

; This is the code to generate for
; JMP loop
      br label %Label_loop

Label_loopend:
; This is the code to generate for
; LDA #$00

      store i8 0, i8* %A

      ; Clear the negative status bit.
      store i1 0, i1* %S_neg

      ; Set the zero status bit.
      store i1 1, i1* %S_zero

; This is the code to generate for
; STA $2009
      %15 = load i8* %A
      %16 = zext i8 %15 to i32
      call void @exit(i32 %16) noreturn nounwind

  ; Terminate this basic block with `unreachable` because
  ; exit never returns.
      unreachable

; Generate dummy basic blocks for the
; interrupt vectors, because we don't support them yet.
IRQ_Routine:
      unreachable

NMI_Routine:
      unreachable
}

In this code we use S_neg and S_zero, 2 of the status register bits. These bits, along with the other status bits that we did not mention yet, are updated after certain instructions and used for things such as branching. Here is a full description of all the status bits:

Let's make sure our goal LLVM code does what we expect:

$ llc -filetype=obj hello.llvm
$ gcc hello.llvm.o
$ ./a.out
Hello, World!
$

llc converts the LLVM IR code into a native object file, and then gcc does the final link step, statically linking against libc to hook up main, putchar, and exit. By default, gcc creates an executable named a.out, which we run, and viola!

The next step is to generate this code from our disassembly. With the help of gollvm we will:

1. Create a LLVM module.
2. Declare our dependency on putchar and exit.
3. Create the main function and allocate stack variables for the registers.
4. Perform one pass over the AST to identify labeled data and create global variables. We save the index of label name to global variable in a map.
5. Perform a second pass over the AST to generate the basic blocks and save an index of label name to basic block in a map.
6. Final pass over the AST to generate code inside of the basic blocks.

Here is a structure that contains the state we need while compiling:

type Compilation struct {
	// Keep track of the warnings and errors
	// that occur while compiling.
	Warnings []string
	Errors   []string

	// program is our abstract syntax tree -
	// we will make several passes over it during
	// the compilation process.
	program   *Program

	// LLVM variable which represents the module we
	// are creating.
	mod       llvm.Module

	// This is the object that we will use to do all the
	// code generation. It's used to create every kind
	// of statement.
	builder   llvm.Builder

	// Reference to our main function.
	mainFn    llvm.Value

	// References to the functions we declared, so we
	// can call them.
	putCharFn llvm.Value
	exitFn    llvm.Value

	// References to the variables we allocate on the stack
	// so we can use them in instructions.
	rX        llvm.Value
	rY        llvm.Value
	rA        llvm.Value
	rSNeg     llvm.Value
	rSZero    llvm.Value

	// Maps label name to the global variables that we add,
	// for when the code loads data from a label.
	labeledData   map[string]llvm.Value

	// Maps label name to basic block, so that when
	// code branches to another label, we can branch to
	// the relevant basic block.
	labeledBlocks map[string]llvm.BasicBlock

	// Keeps track of the basic block we are currently
	// generating code for, if any.
	currentBlock *llvm.BasicBlock

	// Keeps a reference to the reset routine basic block
	// so we know where to first jump to from the entry point.
	resetBlock     *llvm.BasicBlock
}

With this structure, we can execute our plan:

func (p *Program) Compile(filename string) (c *Compilation) {
	llvm.InitializeNativeTarget()

	c = new(Compilation)
	c.program = p
	c.Warnings = []string{}
	c.Errors = []string{}
	c.mod = llvm.NewModule("asm_module")
	c.builder = llvm.NewBuilder()
	defer c.builder.Dispose()
	c.labeledData = map[string]llvm.Value{}
	c.labeledBlocks = map[string]llvm.BasicBlock{}

	// First pass to identify labeled data and create
	// global variables, saving the indexes in `c.labeledData`.
	c.identifyLabeledDataPass()

	// declare i32 @putchar(i32)
	i32Type := llvm.Int32Type()
	putCharType := llvm.FunctionType(i32Type, []llvm.Type{i32Type}, false)
	c.putCharFn = llvm.AddFunction(c.mod, "putchar", putCharType)
	c.putCharFn.SetLinkage(llvm.ExternalLinkage)

	// declare void @exit(i32) noreturn nounwind
	exitType := llvm.FunctionType(llvm.VoidType(), []llvm.Type{i32Type}, false)
	c.exitFn = llvm.AddFunction(c.mod, "exit", exitType)
	c.exitFn.AddFunctionAttr(llvm.NoReturnAttribute | llvm.NoUnwindAttribute)
	c.exitFn.SetLinkage(llvm.ExternalLinkage)

	// main function / entry point
	mainType := llvm.FunctionType(i32Type, []llvm.Type{}, false)
	c.mainFn = llvm.AddFunction(c.mod, "main", mainType)
	c.mainFn.SetFunctionCallConv(llvm.CCallConv)
	entry := llvm.AddBasicBlock(c.mainFn, "Entry")
	c.builder.SetInsertPointAtEnd(entry)
	c.rX = c.builder.CreateAlloca(llvm.Int8Type(), "X")
	c.rY = c.builder.CreateAlloca(llvm.Int8Type(), "Y")
	c.rA = c.builder.CreateAlloca(llvm.Int8Type(), "A")
	c.rSNeg = c.builder.CreateAlloca(llvm.Int1Type(), "S_neg")
	c.rSZero = c.builder.CreateAlloca(llvm.Int1Type(), "S_zero")

	// Second pass to build basic blocks.
	c.buildBasicBlocksPass()

	// Finally, one last pass for codegen.
	c.codeGenPass()

	// hook up the first entry block to the reset block
	c.builder.SetInsertPointAtEnd(entry)
	c.builder.CreateBr(*c.resetBlock)

	err := llvm.VerifyModule(c.mod, llvm.ReturnStatusAction)
	if err != nil {
		c.Errors = append(c.Errors, err.Error())
		return
	}

	// Uncomment this to print the LLVM IR code we just generated
	// to stderr.
	//c.mod.Dump()

	fd, err := os.Create(filename)
	if err != nil {
		c.Errors = append(c.Errors, err.Error())
		return
	}

	err = llvm.WriteBitcodeToFile(c.mod, fd)
	if err != nil {
		c.Errors = append(c.Errors, err.Error())
		return
	}

	err = fd.Close()
	if err != nil {
		c.Errors = append(c.Errors, err.Error())
		return
	}
}

Here's what it the module looks like after only the first pass which identifies labeled data:

; ModuleID = 'asm_module'

@Label_c000 = private global [15 x i8] c"Hello, World!\0A\00"

That's it. All we've done is created a global variable for msg (which is known as Label_c000 in our disassembly) and saved it in the c.labeledData map.

Now add in the declares and main function definition, and we get:

; ModuleID = 'asm_module'

@Label_c000 = private global [15 x i8] c"Hello, World!\0A\00"

declare i32 @putchar(i32)

declare void @exit(i32) noreturn nounwind

define i32 @main() {
Entry:
  %X = alloca i8
  %Y = alloca i8
  %A = alloca i8
  %S_neg = alloca i1
  %S_zero = alloca i1
}

Nothing fancy here. Next let's add in the second pass to identify basic blocks:

; ModuleID = 'asm_module'

@Label_c000 = private global [15 x i8] c"Hello, World!\0A\00"

declare i32 @putchar(i32)

declare void @exit(i32) noreturn nounwind

define i32 @main() {
Entry:
  %X = alloca i8
  %Y = alloca i8
  %A = alloca i8
  %S_neg = alloca i1
  %S_zero = alloca i1

Reset_Routine:                                    ; No predecessors!

Label_c011:                                       ; No predecessors!

Label_c01d:                                       ; No predecessors!

IRQ_Routine:                                      ; No predecessors!

NMI_Routine:                                      ; No predecessors!
}

At this point we've created basic blocks for each of our labels and saved them in our c.labeledBlocks map.

And finally, we do the actual code generation pass and add the jump from the entry block to the reset block.

The actual code generation code is surprisingly simple. It consists of a switch case, and a few calls into the llvm builder object. For example, here's the code generation code for LDX #$00:

func (i *Instruction) Compile(c *Compilation) {
	v := llvm.ConstInt(llvm.Int8Type(), uint64(i.Value), false)
	switch i.OpCode {
	case 0xa2: // ldx
		c.builder.CreateStore(v, c.rX)
		c.testAndSetZero(i.Value)
		c.testAndSetNeg(i.Value)
	// more cases for other immediate instructions
	}
}

func (c *Compilation) testAndSetZero(v int) {
	if v == 0 {
		c.setZero()
		return
	}
	c.clearZero()
}

func (c *Compilation) setZero() {
	c.builder.CreateStore(llvm.ConstInt(llvm.Int1Type(), 1, false), c.rSZero)
}

func (c *Compilation) clearZero() {
	c.builder.CreateStore(llvm.ConstInt(llvm.Int1Type(), 0, false), c.rSZero)
}

func (c *Compilation) testAndSetNeg(v int) {
	if v&0x80 == 0x80 {
		c.setNeg()
		return
	}
	c.clearNeg()
}

func (c *Compilation) setNeg() {
	c.builder.CreateStore(llvm.ConstInt(llvm.Int1Type(), 1, false), c.rSNeg)
}

func (c *Compilation) clearNeg() {
	c.builder.CreateStore(llvm.ConstInt(llvm.Int1Type(), 0, false), c.rSNeg)
}

For completeness's sake, let's look at one more code gen example. Here's the code generation code for LDA msg, X:

func (i *Instruction) Compile(c *Compilation) {
	switch i.OpCode {
	case 0xbd: // LDA label, X
		// Look up the global module variable based on the label name.
		dataPtr := c.labeledData[i.LabelName]
		index := c.builder.CreateLoad(c.rX, "")
		// This is what we index into the global variable with.
		indexes := []llvm.Value{
			llvm.ConstInt(llvm.Int8Type(), 0, false),
			index,
		}
		// Obtain a pointer to the element that we want to load.
		ptr := c.builder.CreateGEP(dataPtr, indexes, "")
		v := c.builder.CreateLoad(ptr, "")
		c.builder.CreateStore(v, c.rA)
		c.dynTestAndSetNeg(v)
		c.dynTestAndSetZero(v)
	// more cases for other direct-with-label-indexed instructions
	}
}

func (c *Compilation) dynTestAndSetNeg(v llvm.Value) {
	x80 := llvm.ConstInt(llvm.Int8Type(), uint64(0x80), false)
	masked := c.builder.CreateAnd(v, x80, "")
	isNeg := c.builder.CreateICmp(llvm.IntEQ, masked, x80, "")
	c.builder.CreateStore(isNeg, c.rSNeg)
}

func (c *Compilation) dynTestAndSetZero(v llvm.Value) {
	zeroConst := llvm.ConstInt(llvm.Int8Type(), uint64(0), false)
	isZero := c.builder.CreateICmp(llvm.IntEQ, v, zeroConst, "")
	c.builder.CreateStore(isZero, c.rSZero)
}

After implementing codegen for the rest of the instructions, here is the module dump after the code generation pass:

; ModuleID = 'asm_module'

@Label_c000 = private global [15 x i8] c"Hello, World!\0A\00"

declare i32 @putchar(i32)

declare void @exit(i32) noreturn nounwind

define i32 @main() {
Entry:
  %X = alloca i8
  %Y = alloca i8
  %A = alloca i8
  %S_neg = alloca i1
  %S_zero = alloca i1
  br label %Reset_Routine

Reset_Routine:                                    ; preds = %Entry
  store i8 0, i8* %X
  store i1 true, i1* %S_zero
  store i1 false, i1* %S_neg
  br label %Label_c011

Label_c011:                                       ; preds = %else, %Reset_Routine
  %0 = load i8* %X
  %1 = getelementptr [15 x i8]* @Label_c000, i8 0, i8 %0
  %2 = load i8* %1
  store i8 %2, i8* %A
  %3 = and i8 %2, -128
  %4 = icmp eq i8 %3, -128
  store i1 %4, i1* %S_neg
  %5 = icmp eq i8 %2, 0
  store i1 %5, i1* %S_zero
  %6 = load i1* %S_zero
  br i1 %6, label %Label_c01d, label %else

else:                                             ; preds = %Label_c011
  %7 = load i8* %A
  %8 = zext i8 %7 to i32
  %9 = call i32 @putchar(i32 %8)
  %10 = load i8* %X
  %11 = add i8 %10, 1
  store i8 %11, i8* %X
  %12 = and i8 %11, -128
  %13 = icmp eq i8 %12, -128
  store i1 %13, i1* %S_neg
  %14 = icmp eq i8 %11, 0
  store i1 %14, i1* %S_zero
  br label %Label_c011

Label_c01d:                                       ; preds = %Label_c011
  store i8 0, i8* %A
  store i1 true, i1* %S_zero
  store i1 false, i1* %S_neg
  %15 = load i8* %A
  %16 = zext i8 %15 to i32
  call void @exit(i32 %16)
  unreachable

IRQ_Routine:                                      ; preds = %Label_c01d
  unreachable

NMI_Routine:                                      ; No predecessors!
  unreachable
}

Great! This looks just like the code we wanted to get, minus the comments and with a few things renamed. Let's make sure it runs as expected:

$ llc -filetype=obj hello.bc
$ gcc hello.bc.o
$ ./a.out
Hello, World!
$

At this point in the project we are able to recompile a simple "Hello, World" 6502 program with a small custom ABI into native machine code and then execute it.

Optimization

Did you notice that we never used the value of S_neg? We only ever stored it. This is a waste of CPU cycles. We can do better. However, we don't want to completely remove the ability to compute S_neg - although this "Hello World" example does not use the value, other code might.

Optimization is an enormously complicated topic, with its own well-deserved field. Luckly, we won't have to wrap our heads around it in order to benefit. LLVM IR code is designed to be optimized. LLVM comes with state of the art optimization techniques, in the form of passes that you run on a module.

Let's run several optimization passes on the module we generate before rendering bitcode:

Here's what it looks like to add these optimization passes to our Compile code:

	// ...
	err := llvm.VerifyModule(c.mod, llvm.ReturnStatusAction)
	if err != nil {
		c.Errors = append(c.Errors, err.Error())
		return
	}

	// This creates an engine object which has useful information
	// about our machine as a target.
	engine, err := llvm.NewJITCompiler(c.mod, 3)
	if err != nil {
		c.Errors = append(c.Errors, err.Error())
		return
	}
	defer engine.Dispose()

	pass := llvm.NewPassManager()
	defer pass.Dispose()

	pass.Add(engine.TargetData())
	pass.AddConstantPropagationPass()
	pass.AddInstructionCombiningPass()
	pass.AddPromoteMemoryToRegisterPass()
	pass.AddGVNPass()
	pass.AddCFGSimplificationPass()
	pass.Run(c.mod)

	// Uncomment this to print the LLVM IR code we just generated
	// to stderr.
	//c.mod.Dump()

	// ...

LLVM offers many more available passes, and it's up to the user to choose which ones, and in which order, to run. Let's see how the ones chosen in the table above perform on our "Hello World" code:

; ModuleID = 'asm_module'

@Label_c000 = private global [15 x i8] c"Hello, World!\0A\00"

declare i32 @putchar(i32)

declare void @exit(i32) noreturn nounwind

define i32 @main() {
Entry:
  br label %Label_c011

Label_c011:                                       ; preds = %else, %Entry
  %storemerge = phi i8 [ 0, %Entry ], [ %6, %else ]
  %0 = sext i8 %storemerge to i64
  %1 = getelementptr [15 x i8]* @Label_c000, i64 0, i64 %0
  %2 = load i8* %1, align 1
  %3 = icmp eq i8 %2, 0
  br i1 %3, label %Label_c01d, label %else

else:                                             ; preds = %Label_c011
  %4 = zext i8 %2 to i32
  %5 = call i32 @putchar(i32 %4)
  %6 = add i8 %storemerge, 1
  br label %Label_c011

Label_c01d:                                       ; preds = %Label_c011
  call void @exit(i32 0)
  unreachable
}

Wow! The code looks completely different, and much simpler. But does it still run?

$ llc -filetype=obj hello.bc
$ gcc hello.bc.o
$ ./a.out
Hello, World!
$

Sure does!

Now, not only are we recompiling a simple 6502 program for native machine code, but we're actually generating highly optimized code. But it's not time to congratulate ourselves yet. This is a contrived case. Let's see if these techniques can work on an actual NES game.

Layout of an NES ROM File

The generally accepted standard for distributing NES games is the .nes format, originally started by the iNES emulator. A .nes file looks like:

1. 16 byte header with metadata, such as:
   • What mapper, if any, the game uses.
   • Whether the PPU uses vertical or horizontal mirroring. For now, don't worry so much about exactly what this means, as much as that it is a setting that the PPU needs to know about at bootup.
2. The 32 KB of assembled code which gets loaded into $8000 - $ffff on bootup. If there is a mapper, there might be more of this program data, but we'll talk about that later.
3. 8KB of graphics data, known as CHR-ROM. This gets loaded into the PPU on bootup. Again if there is a mapper, there might be more of this graphics data.

If you're looking for a reference for the gory details of this format, see the Nesdev Wiki or the official specification.

The first "real" ROM I tried to recompile is this demo ROM made by Chris Covell in 1998 called Zelda Title Screen Simulator. If you run this ROM in an emulator, you get a title screen that looks nearly identical to Zelda 1:

Now that we're recompiling a real ROM, we'll have to do several more things to make it work:

• Include the graphics data in the binary.
• Include the mirroring setting (vertical or horizontal) in the binary.
• Support code gen for more instructions.
• Include a runtime to create a window and render the video to the screen.

Including the graphics data and mirroring setting in the binary is trivial. We get all the information we need when we read the ROM file; all we need to do is convert it to global variables in our LLVM module:

// In this snippet, `program` is an object which contains
// the information we loaded from the ROM file.
// chrRom is [][]byte - n banks of 8KB. At this point,
// we only support 1 bank.
if len(program.chrRom) != 1 {
	panic("Only 1 bank of CHR ROM is supported.")
}
dataLen := 0x2000 * len(chrRom)
chrDataValues := make([]llvm.Value, 0, dataLen)
int8type := llvm.Int8Type()
for _, bank := range chrRom {
	for _, b := range bank {
		chrDataValues = append(chrDataValues, llvm.ConstInt(int8type, uint64(b), false))
	}
}
chrDataConst := llvm.ConstArray(llvm.ArrayType(llvm.Int8Type(), dataLen), chrDataValues)
// `c.mod` is our `llvm.Module` object. `c` is an instance of our
// `Compilation` struct from a previous code listing.
chrDataGlobal := llvm.AddGlobal(c.mod, chrDataConst.Type(), "rom_chr_data")
// Setting the linkage to private here. We'll figure out what to do
// with this information soon.
chrDataGlobal.SetLinkage(llvm.PrivateLinkage)
chrDataGlobal.SetInitializer(chrDataConst)
chrDataGlobal.SetGlobalConstant(true)

// Now we add the global variable specifying the mirroring setting.
mirroringValue := llvm.ConstInt(llvm.Int8Type(), uint64(program.Mirroring), false)
mirroringGlobal := llvm.AddGlobal(c.mod, mirroringValue.Type(), "rom_mirroring")
mirroringGlobal.SetLinkage(llvm.PrivateLinkage)
mirroringGlobal.SetInitializer(mirroringValue)
mirroringGlobal.SetGlobalConstant(true)

The next step to getting our selected ROM to work is to support the additional instructions that this program uses. Let's unpack that rom and see what we're up against:

$ ./jamulate -unrom roms/Zelda.NES
loading roms/Zelda.NES
disassembling to roms/Zelda
$ ls -l roms/Zelda
-rw-rw-r-- 1 andy andy 8192 May 24 23:53 chr0.chr
-rw-rw-r-- 1 andy andy 7977 May 24 23:53 prg.asm
-rw-rw-r-- 1 andy andy  485 May 24 23:53 Zelda.jam

The code to unpack a .nes file into CHR data, PRG data, and metadata is left as an exercise to the reader.

chr0.chr is binary data; it is the 8KB of graphics data mentioned before which is loaded into the PPU on bootup.

Zelda.jam is where I decided to put the program metadata in a simple key-value text format:

# output file name when this rom is assembled
filename=Zelda.NES
# see http://wiki.nesdev.com/w/index.php/Mapper
mapper=0
# 'Horizontal', 'Vertical', or 'FourScreenVRAM'
# see http://wiki.nesdev.com/w/index.php/Mirroring
mirroring=Horizontal
# whether SRAM in CPU $6000-$7FFF is present
sram=true
# whether the SRAM in CPU $6000-$7FFF, if present, is battery backed
battery=false
# 'NTSC', 'PAL', or 'DualCompatible'
tvsystem=NTSC
# assembly code
prg=prg.asm
# video data
chr=chr0.chr

In retrospect, I think it would have been simpler to have the assembler support special declarations for this metadata rather than having a separate file. But hey, it works.

You may notice there is some metadata there which is never addressed in this article. For the purposes of this project, we don't need to think about SRAM, battery, or tvsystem. None of the examples we look at use them.

Without further ado, let's see how the disassembler fared:

.org $c000
    .db "Zelda Simulator, ", $a9, "1998 Chris Covell (ccovell@direct.ca)"
Reset_Routine:
    CLD
    SEI
Label_c039:
    LDA $2002
    BPL Label_c039
    LDX #$00
    STX $2000
    STX $2001
    DEX
    TXS
    LDY #$06
    STY $01
    LDY #$00
    STY $00
    LDA #$00
Label_c052:
    STA ($00), Y
    DEY
    BNE Label_c052
    DEC $01
    BPL Label_c052
    LDX #$3f
    STX $2006
    LDX #$00
    STX $2006
    LDX #$27
    LDY #$20
Label_c069:
    STX $2007
    DEY
    BNE Label_c069
    LDX #$3f
    STX $2006
    LDX #$00
    STX $2006
    LDX #$00
    LDY #$20
Label_c07d:
    LDA Label_c101, X
    STA $2007
    INX
    DEY
    BNE Label_c07d
    LDX #$23
    STX $2006
    LDX #$c0
    STX $2006
    LDX #$00
    LDY #$40
Label_c095:
    LDA Label_c121, X
    STA $2007
    INX
    DEY
    BNE Label_c095
    LDX #$20
    STX $2006
    LDX #$00
    STX $2006
    LDX #$00
    LDY #$00
Label_c0ad:
    LDA Label_c261, X
    STA $2007
    INX
    DEY
    BNE Label_c0ad
    LDX #$00
    LDY #$00
Label_c0bb:
    LDA Label_c361, X
    STA $2007
    INX
    DEY
    BNE Label_c0bb
    LDX #$00
    LDY #$00
Label_c0c9:
    LDA Label_c461, X
    STA $2007
    INX
    DEY
    BNE Label_c0c9
    LDX #$00
    LDY #$c0
Label_c0d7:
    LDA Label_c561, X
    STA $2007
    INX
    DEY
    BNE Label_c0d7
    LDX #$00
    STX $2003
    LDX #$00
    LDY #$00
Label_c0ea:
    LDA Label_c161, X
    STA $2004
    INX
    DEY
    BNE Label_c0ea
    LDA #$90
    STA $2000
    LDA #$1e
    STA $2001
Label_c0fe:
    JMP Label_c0fe
Label_c101:
    .db $36, $0d, $00, $10, $36, $17, $27, $0d
    .db $36, $08, $1a, $28, $36, $30, $31, $22
    .db $36, $30, $31, $11, $36, $15, $15, $15
    .db $36, $08, $1a, $28, $36, $30, $31, $22
Label_c121:
    .db $05, $05, $05, $05, $05, $05, $05, $05
    .db $08, "jZZZZ", $9a, $22, $00, "fUUUU", $99, $00
    .db $00, $6e, $5f, $55, $5d, $df, $bb, $00
    .db $00, $0a, $0a, $0a, $0a, $0a, $0a, $00
    .db $00, $00, $c0, $30, $00, $00, $00, $00
    .db $00, $00, $cc, $33, $00, $00, $00, $00
    .db $00, $20, $fc, $f3, $00, $00, $f0, $f0
Label_c161:
    .db $27, $ca, $02, $28, $2f, $cb, $02, $28
    .db $27, $cc, $02, $30, $2f, $cd, $02, $30
    .db $27, $d6, $02, $50, $2c, $d6, $02, $a0
    .db $27, $cc, $42, $c8, $2f, $cd, $42, $c8
    .db $27, $ca, $42, $d0, $2f, $cb, $42, $d0
    .db $31, $d2, $02, $57, $31, $d4, $02, $5f
    .db $3f, $d4, $02, $24, $41, $d4, $02, $63
    .db $4f, $d6, $02, $2c, $4f, $d2, $02, $cb
    .db $57, $ce, $02, $73, $5f, $cf, $02, $73
    .db $57, $d0, $02, $7b, $5f, $d1, $02, $7b
    .db $67, $d2, $02, $7a, $7b, $d4, $02, $90
    .db $7b, $d6, $02, $bc, $82, $d2, $02, $50
    .db $77, $cb, $82, $28, $7f, $ca, $82, $28
    .db $77, $cd, $82, $30, $7f, $cc, $82, $30
    .db $77, $cd, $c2, $c8, $7f, $cc, $c2, $c8
    .db $77, $cb, $c2, $d0, $7f, $ca, $c2, $d0
    .db $af, $a3, $00, $50, $af, $a5, $00, $58
    .db $af, $a7, $00, $60, $af, $a9, $00, $68
    .db $b7, $b2, $00, $50, $b7, $b4, $00, $58
    .db $b7, $b6, $00, $60, $b7, $b8, $00, $68
    .db $c9, $c2, $00, $50, $d1, $c3, $00, $50
    .db $c9, $c4, $00, $58, $d1, $c5, $00, $58
    .db $c9, $c6, $00, $60, $d1, $c7, $00, $60
    .db $c9, $c8, $00, $68, $d1, $c9, $00, $68
    .db $d9, $c2, $00, $50, $e1, $c3, $00, $50
    .db $d9, $c4, $00, $58, $e1, $c5, $00, $58
    .db $d9, $c6, $00, $60, $e1, $c7, $00, $60
    .db $d9, $c8, $00, $68, $e1, $c9, $00, $68
    .db $67, $a0, $03, $58, $67, $a0, $03, $60
    .db $67, $a0, $03, $68, $67, $a0, $03, $70
    .db $67, $a0, $03, $78, $67, $a0, $03, $80
    .db $67, $a0, $03, $88, $00, $b0, $00, $00
Label_c261:
    .db "$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$", $e0, $d5, "$$$$$$$$$$$$$$$$$$$$$$$$$$$$", $d4, $e0, $dc, $d7
    .db "$$$$$$$$$$$$$$$$$$$$$$$$$$$$", $d6, $dd, $dc, $ee, "$$$$$$$$$$$$$$$$$$$$$$$$$$$$", $d6, $db
    .db $de, $d7, $24, $24, $24, $e6, $e4, $e5
    .db $e4, $e5, $e4, $e5, $e4, $e5, $e4, $e5
    .db $e4, $e5, $e4, $e5, $e4, $e5, $e4, $e5
    .db $e4, $e5, $e6, $24, $24, $24, $d6, $db
    .db $dc, $d7, $24, $24, $24, $e2, "$$$$$$$$$$$$$$$$$$$$", $e3
    .db $24, $24, $24, $d6, $df, $de, $ee, $24
    .db $24, $24, $e3, "$$qrstuvwxyyyz{$$$$$", $e2, $24, $24, $24
    .db $d6, $db
Label_c361:
    .db $de, $d8, $ef, $24, $24, $e2, $24, $7c
    .db $7d, $7e, $7f, $80, $81, $82, $83, $84
    .db $85, $86, $87, $88, $89, $8a, $8b, $24
    .db $24, $24, $e3, $24, $24, $24, $d6, $df
    .db $dc, $da, $d7, $24, $24, $e3, $24, $8c
    .db $8d, $8e, $8f, $90, $91, $92, $93, $94
    .db $95, $96, $97, $98, $99, $9a, $9b, $9c
    .db $24, $24, $e2, $24, $24, $d4, $d9, $db
    .db $dc, $d9, $ee, $24, $24, $e2, $24, $9d
    .db $9e, $9f, $a0, $a1, $a2, $a3, $a4, $a5
    .db $a6, $a7, $a8, $a9, $aa, $ab, $ac, $ad
    .db $ae, $24, $e3, $24, $24, $d6, $db, $df
    .db $de, $db, $d7, $24, $24, $e3, $70, $af
    .db $b0, $b1, $b2, $b3, $b4, $b5, $b6, $b7
    .db $b8, $b9, $ba, $bb, $bc, $bd, $be, $bf
    .db $c0, $24, $e2, $24, $24, $d6, $db, $db
    .db $dc, $dd, $d7, $24, $24, $e2, "$$$$$$$", $c1
    .db $c2, $c3, $c4, $c5, "$$$$$$$$", $e3, $24, $24
    .db $d6, $db, $df, $de, $db, $ee, $24, $24
    .db $e3, $24, $c6, $c7, $c8, $c8, $c8, $24
    .db $c9, $ca, $cb, $cc, $cd, $c8, $c8, $c8
    .db $c8, $e8, $e9, $d3, $24, $e2, $24, $24
    .db $d6, $db, $db, $dc, $db, $d7, $24, $24
    .db $e2, "$$$$$$$$$", $ce, $cf, "$$$$$", $ea, $eb, $ec
    .db $24, $e3, $24, $24, $d6, $db, $df, $dc
    .db $db, $d7, $24, $24, $e3, "$$$$$$$$$", $d1, $d2
    .db "$$$$$$$$$", $e2, $24, $24, $d6, $db, $db
Label_c461:
    .db $dc, $d8, $e1, $d5, $24, $e6, $e4, $e5
    .db $e4, $e5, $e4, $e5, $e4, $e5, $e4, $e5
    .db $e4, $e5, $e4, $e5, $e4, $e5, $e4, $e5
    .db $e4, $e5, $e6, $24, $d4, $e1, $d9, $dd
    .db $dc, $da, $dc, $d7, "$$$$$$$$$", $f0, $01, $09
    .db $08, $06, $24, $17, $12, $17, $1d, $0e
    .db $17, $0d, $18, $24, $d6, $dc, $db, $df
    .db $dc, $da, $dc, $ee, "$$$$$$$$$$$$$$$$$$$$$$$$", $d6, $de, $ed
    .db $dd, $dc, $da, $de, $d7, "$$$$$$$$$$$$$$$$$$$$$$$$", $d6, $de
    .db $db, $dd, $e1, $d9, $dc, $ed, $e0, $ef
    .db $24, $24, $19, $1e, $1c, $11, $24, $1c
    .db $1d, $0a, $1b, $1d, $24, $0b, $1e, $1d
    .db $1d, $18, $17, $24, $24, $24, $d6, $d8
    .db $e1, $d9, $dd, $ed, $de, $d8, $e1, $e1
    .db $d5, "$$$$$$$$$$$$$$$$$$$$$", $d6, $da, $dd, $ed, $dd, $db
    .db $de, $da, $dc, $dd, $d8, $e0, $e0, $ef
    .db $24, $24, $24, $24, $d4, $e0, $e0, $d5
    .db "$$$$$$$$", $d4, $ef, $da, $da, $df, $db, $df
    .db $db, $dc, $da, $de, $df, $da, $dc, $dd
    .db $db, $26, $26, $26, $26, $da, $dc, $dd
    .db $ed, $e0, $e0, $ef, $24, $d4, $e0, $e0
    .db $e0, $d9, $db, $da, $da, $df, $db
Label_c561:
    .db $ed, $d8, $e1, $d9, $de, $d8, $e1, $d9
    .db $df, $db, $26, $26, $26, $26, $da, $dc
    .db $d8, $e1, $d9, $dc, $d8, $e0, $d9, $dc
    .db $dd, $dd, $d8, $e1, $e1, $d9, $dd, $ed
    .db $ed, $da, $dd, $ed, $de, $da, $dc, $db
    .db $dd, $db, $26, $26, $26, $26, $da, $de
    .db $da, $dc, $db, $de, $da, $dd, $ed, $dc
    .db $dc, $dd, $da, $dc, $dc, $db, $dd, $ed
    .db $ed, $da, $dd, $d8, $e1, $da, $dc, $db
    .db $df, $db, $26, $26, $26, $26, $da, $d8
    .db $d9, $dc, $db, $dc, $da, $d8, $e1, $d9
    .db $de, $df, $da, $d8, $e1, $e1, $d9, $e1
    .db $ed, $d9, $df, $da, $dc, $da, $de, $db
    .db $dd, $db, $26, $26, $26, $26, $da, $da
    .db $ed, $de, $d8, $e1, $e1, $d9, $dc, $db
    .db $de, $d8, $e1, $d9, $dd, $dd, $db, $dc
    .db $df, $db, $df, $da, $dc, $db, $de, $db
    .db $dd, $db, $26, $26, $26, $26, $da, $da
    .db $db, $dd, $da, $de, $d8, $e1, $d9, $db
    .db $de, $da, $dd, $db, $de, $df, $d8, $e1
    .db $df, $db, $df, $da, $dc, $db, $de, $db
    .db $dd, $db, $26, $26, $26, $26, $da, $da
    .db $db, $dd, $da, $de, $da, $dd, $db, $db
    .db $de, $da, $dd, $db, $de, $df, $dc, $e1
NMI_Routine:
    LDX #$00
    STX $2005
    STX $2005
    RTI
IRQ_Routine:
    RTI
.org $fffa, $00
    .dw NMI_Routine
    .dw Reset_Routine
    .dw IRQ_Routine

Looking at this disassembly, several things come to mind:

• Our ASCII detection did a good job with that string at the beginning. Nice.
• This code introduces only a few new instructions. It won't be too cumbersome to add code gen to support this.
• It looks like the disassembler was able to disassemble the entire program. You can see at the end there is an infinite loop and then data follows.
• ASCII detection found some false positives. Oh well. It makes no difference to the execution of the program.
• The IRQ interrupt is not used, but it looks like the NMI interrupt might be used.

Let's see if we can once again delay dealing with interrupts at all. Modify the NMI_Routine block so that it looks like this:

NMI_Routine:
    RTI

Good thing we bothered to make an assembler. Now we can put the ROM back together, run it in an emulator, and see if it still works:

$ ./jamulate -rom roms/Zelda/Zelda.jam
building rom from roms/Zelda/Zelda.jam
saving rom Zelda.NES

Again, writing the code to pack the rom back together into an .nes file is left as an exercise to the reader.

Now we observe the ROM running in an emulator and see if it still works:

Ha! It works but the sword is bent. That's actually pretty convenient - now when we want to make interrupts work, we know how to test them. But like true software developers, we're going to ignore any and all problems as long as possible.

The only thing left to do then, is to add code gen support for the new instructions. This leads us to our first challenge.

Challenge: Runtime Address Checks

Looking at the disassembled code for Zelda Title Screen Simulator, there is one instruction that stands out as a bit more tricky to recompile than the others:

    STA ($00), Y

This is an indirect instruction. It tells the CPU to:

1. Interpret the values in $0000 and $0001 as a memory address.
2. Add the value of register Y to the address to compute a new address.
3. Store the value of register A to the new address.

Quite a useful instruction, but it poses a challenge for recompilation. Because values in memory addresses $0000 and $0001 could be anything, only at runtime will we learn which memory address is about to be updated.

Recall that the NES uses memory mapped I/O. This means that this instruction could be saving a value in memory, but it could also be talking to the PPU, the APU, the game pad controller, or even a mapper. Since this one instruction could be doing any of these things depending on runtime state, we will have to add a runtime address check.

Here's what the code generation looks like for STA-indirect-Y:

func (i *Instruction) Compile(c *Compilation) {
	switch i.OpCode {
	// ... more cases for other indirect instructions

	case 0x91: // sta indirect y
		// load the base address at the address indicated by the instruction
		baseAddr := c.loadWord(i.Value)
		// load the 8-bit value of register Y
		rY := c.builder.CreateLoad(c.rY, "")
		// zero extend the 8-bit value of Y to 16 bits
		// so that we can add it to baseAddr
		rYw := c.builder.CreateZExt(rY, llvm.Int16Type(), "")
		// compute the pointer to the address to store a byte
		addr := c.builder.CreateAdd(baseAddr, rYw, "")
		// load the value of register A
		rA := c.builder.CreateLoad(c.rA, "")
		// call dynStore with our computed address and value of A.
		// dynStore will figure out what to do with the value and address.
		c.dynStore(addr, rA)
	}
}

func (c *Compilation) loadWord(addr int) llvm.Value {
	// Load a little endian word.

	// Load the low byte.
	ptrByte1 := c.load(addr)
	// Load the high byte.
	ptrByte2 := c.load(addr + 1)

	// Zero extend the 8-bit values to 16-bit.
	ptrByte1w := c.builder.CreateZExt(ptrByte1, llvm.Int16Type(), "")
	ptrByte2w := c.builder.CreateZExt(ptrByte2, llvm.Int16Type(), "")

	// Shift the high byte left by 8.
	shiftAmt := llvm.ConstInt(llvm.Int16Type(), 8, false)
	word := c.builder.CreateShl(ptrByte2w, shiftAmt, "")

	// Bitwise OR the high and low bytes together and return that.
	return c.builder.CreateOr(word, ptrByte1w, "")
}

func (c *Compilation) load(addr int) llvm.Value {
	// This function is used to load a byte from an address that we know
	// at compile-time.
	switch {
	default:
		c.Errors = append(c.Errors, fmt.Sprintf("reading from $%04x not implemented", addr))
		return llvm.ConstNull(llvm.Int8Type())
	case 0x0000 <= addr && addr < 0x2000:
		// 2KB working RAM. mask because mirrored
		maskedAddr := addr & (0x800 - 1)
		indexes := []llvm.Value{
			llvm.ConstInt(llvm.Int16Type(), 0, false),
			llvm.ConstInt(llvm.Int16Type(), uint64(maskedAddr), false),
		}
		ptr := c.builder.CreateGEP(c.wram, indexes, "")
		v := c.builder.CreateLoad(ptr, "")
		return v
	case 0x2000 <= addr && addr < 0x4000:
		// PPU registers. mask because mirrored
		switch addr & (0x8 - 1) {
		case 2:
			return c.builder.CreateCall(c.ppuReadStatusFn, []llvm.Value{}, "")
		case 4:
			return c.builder.CreateCall(c.ppuReadOamDataFn, []llvm.Value{}, "")
		case 7:
			return c.builder.CreateCall(c.ppuReadDataFn, []llvm.Value{}, "")
		default:
			c.Errors = append(c.Errors, fmt.Sprintf("reading from $%04x not implemented", addr))
			return llvm.ConstNull(llvm.Int8Type())
		}

	// ... There are more cases to handle; here we only include WRAM and the PPU.

	}
	panic("unreachable")
}

func (c *Compilation) dynStore(addr llvm.Value, val llvm.Value) {
	// runtime memory check
	storeDoneBlock := c.createBlock("StoreDone")
	x2000 := llvm.ConstInt(llvm.Int16Type(), 0x2000, false)
	inWRam := c.builder.CreateICmp(llvm.IntULT, addr, x2000, "")
	notInWRamBlock := c.createIf(inWRam)
	// this generated code runs if the write is happening in the WRAM range
	maskedAddr := c.builder.CreateAnd(addr, llvm.ConstInt(llvm.Int16Type(), 0x800-1, false), "")
	indexes := []llvm.Value{
		llvm.ConstInt(llvm.Int16Type(), 0, false),
		maskedAddr,
	}
	ptr := c.builder.CreateGEP(c.wram, indexes, "")
	c.builder.CreateStore(val, ptr)
	c.builder.CreateBr(storeDoneBlock)
	// this generated code runs if the write is > WRAM range
	c.selectBlock(notInWRamBlock)
	x4000 := llvm.ConstInt(llvm.Int16Type(), 0x4000, false)
	inPpuRam := c.builder.CreateICmp(llvm.IntULT, addr, x4000, "")
	notInPpuRamBlock := c.createIf(inPpuRam)
	// this generated code runs if the write is in the PPU RAM range
	maskedAddr = c.builder.CreateAnd(addr, llvm.ConstInt(llvm.Int16Type(), 0x8-1, false), "")
	badPpuAddrBlock := c.createBlock("BadPPUAddr")
	sw := c.builder.CreateSwitch(maskedAddr, badPpuAddrBlock, 7)
	// this generated code runs if the write is in a bad PPU RAM addr
	c.selectBlock(badPpuAddrBlock)
	c.createPanic("invalid store address: $%04x\n", []llvm.Value{addr})

	ppuCtrlBlock := c.createBlock("ppuctrl")
	sw.AddCase(llvm.ConstInt(llvm.Int16Type(), 0, false), ppuCtrlBlock)
	c.selectBlock(ppuCtrlBlock)
	c.builder.CreateCall(c.ppuCtrlFn, []llvm.Value{val}, "")
	c.builder.CreateBr(storeDoneBlock)

	ppuMaskBlock := c.createBlock("ppumask")
	sw.AddCase(llvm.ConstInt(llvm.Int16Type(), 1, false), ppuMaskBlock)
	c.selectBlock(ppuMaskBlock)
	c.builder.CreateCall(c.ppuMaskFn, []llvm.Value{val}, "")
	c.builder.CreateBr(storeDoneBlock)

	oamAddrBlock := c.createBlock("oamaddr")
	sw.AddCase(llvm.ConstInt(llvm.Int16Type(), 3, false), oamAddrBlock)
	c.selectBlock(oamAddrBlock)
	c.builder.CreateCall(c.oamAddrFn, []llvm.Value{val}, "")
	c.builder.CreateBr(storeDoneBlock)

	oamDataBlock := c.createBlock("oamdata")
	sw.AddCase(llvm.ConstInt(llvm.Int16Type(), 4, false), oamDataBlock)
	c.selectBlock(oamDataBlock)
	c.builder.CreateCall(c.setOamDataFn, []llvm.Value{val}, "")
	c.builder.CreateBr(storeDoneBlock)

	ppuScrollBlock := c.createBlock("ppuscroll")
	sw.AddCase(llvm.ConstInt(llvm.Int16Type(), 5, false), ppuScrollBlock)
	c.selectBlock(ppuScrollBlock)
	c.builder.CreateCall(c.setPpuScrollFn, []llvm.Value{val}, "")
	c.builder.CreateBr(storeDoneBlock)

	ppuAddrBlock := c.createBlock("ppuaddr")
	sw.AddCase(llvm.ConstInt(llvm.Int16Type(), 6, false), ppuAddrBlock)
	c.selectBlock(ppuAddrBlock)
	c.builder.CreateCall(c.ppuAddrFn, []llvm.Value{val}, "")
	c.builder.CreateBr(storeDoneBlock)

	ppuDataBlock := c.createBlock("ppudata")
	sw.AddCase(llvm.ConstInt(llvm.Int16Type(), 7, false), ppuDataBlock)
	c.selectBlock(ppuDataBlock)
	c.builder.CreateCall(c.setPpuDataFn, []llvm.Value{val}, "")
	c.builder.CreateBr(storeDoneBlock)

	// this generated code runs if the write is >= 0x4000
	// There are more cases to handle; here we only show
	// handling WRAM and the PPU.
	c.selectBlock(notInPpuRamBlock)
	c.createPanic()

	// done. X_X
	c.selectBlock(storeDoneBlock)
}

func (c *Compilation) createIf(cond llvm.Value) llvm.BasicBlock {
	// Create a conditional branch along with the 2 target blocks.
	// Returns the else block and set the current block to the then block.

	elseBlock := c.createBlock("else")
	thenBlock := c.createBlock("then")
	c.builder.CreateCondBr(cond, thenBlock, elseBlock)
	c.selectBlock(thenBlock)
	return elseBlock
}

func (c *Compilation) createBlock(name string) llvm.BasicBlock {
	bb := llvm.InsertBasicBlock(*c.currentBlock, name)
	bb.MoveAfter(*c.currentBlock)
	return bb
}

func (c *Compilation) selectBlock(bb llvm.BasicBlock) {
	c.builder.SetInsertPointAtEnd(bb)
	c.currentBlock = &bb
}

func (c *Compilation) createPanic() {
	bytePointerType := llvm.PointerType(llvm.Int8Type(), 0)
	ptr := c.builder.CreatePointerCast(c.runtimePanicMsg, bytePointerType, "")
	c.builder.CreateCall(c.putsFn, []llvm.Value{ptr}, "")
	exitCode := llvm.ConstInt(llvm.Int32Type(), 1, false)
	c.builder.CreateCall(c.exitFn, []llvm.Value{exitCode}, "")
	c.builder.CreateUnreachable()
}

Let's take a look at what the IR code might look like for this instruction. First, a simplified assembly source file:

.org $c000

Reset_Routine:
    STA ($00), Y
    STA $2009 ; exit

NMI_Routine:
    RTI

IRQ_Routine:
    RTI

.org $fffa
    .dw NMI_Routine
    .dw Reset_Routine
    .dw IRQ_Routine

And then the LLVM module that would be generated, before optimizations:

; ModuleID = 'asm_module'

@wram = private global [2048 x i8] zeroinitializer
@panicMsg = private global [51 x i8] c"panic: attempted to write to invalid memory address"

declare i32 @putchar(i32)

declare i32 @puts(i8*)

declare void @exit(i32) noreturn nounwind

declare i8 @rom_ppu_read_status()

declare void @rom_ppu_write_control(i8)

declare void @rom_ppu_write_mask(i8)

declare void @rom_ppu_write_address(i8)

declare void @rom_ppu_write_data(i8)

declare void @rom_ppu_write_oamaddress(i8)

declare void @rom_ppu_write_oamdata(i8)

declare void @rom_ppu_write_scroll(i8)

define i32 @main() {
Entry:
  %X = alloca i8
  %Y = alloca i8
  %A = alloca i8
  %SP = alloca i8
  %S_neg = alloca i1
  %S_zero = alloca i1
  %S_dec = alloca i1
  %S_int = alloca i1
  br label %Reset_Routine

Reset_Routine:                                    ; preds = %Entry
  %0 = load i8* getelementptr inbounds ([2048 x i8]* @wram, i8 0, i8 0)
  %1 = load i8* getelementptr inbounds ([2048 x i8]* @wram, i8 0, i8 1)
  %2 = zext i8 %0 to i16
  %3 = zext i8 %1 to i16
  %4 = shl i16 %3, 8
  %5 = or i16 %4, %2
  %6 = load i8* %Y
  %7 = zext i8 %6 to i16
  %8 = add i16 %5, %7
  %9 = load i8* %A
  %10 = icmp ult i16 %8, 8192
  br i1 %10, label %then, label %else

then:                                             ; preds = %Reset_Routine
  %11 = and i16 %8, 2047
  %12 = getelementptr [2048 x i8]* @wram, i16 0, i16 %11
  store i8 %9, i8* %12
  br label %STA_done

else:                                             ; preds = %Reset_Routine
  %13 = icmp ult i16 %8, 16384
  br i1 %13, label %then2, label %else1

then2:                                            ; preds = %else
  %14 = and i16 %8, 7
  switch i16 %14, label %BadPPUAddr [
    i16 0, label %ppuctrl
    i16 1, label %ppumask
    i16 3, label %oamaddr
    i16 4, label %oamdata
    i16 5, label %ppuscroll
    i16 6, label %ppuaddr
    i16 7, label %ppudata
  ]

BadPPUAddr:                                       ; preds = %then2
  %15 = call i32 @puts(i8* getelementptr inbounds ([51 x i8]* @panicMsg, i32 0, i32 0))
  call void @exit(i32 1)
  unreachable

ppuctrl:                                          ; preds = %then2
  call void @rom_ppu_write_control(i8 %9)
  br label %STA_done

ppumask:                                          ; preds = %then2
  call void @rom_ppu_write_mask(i8 %9)
  br label %STA_done

oamaddr:                                          ; preds = %then2
  call void @rom_ppu_write_oamaddress(i8 %9)
  br label %STA_done

oamdata:                                          ; preds = %then2
  call void @rom_ppu_write_oamdata(i8 %9)
  br label %STA_done

ppuscroll:                                        ; preds = %then2
  call void @rom_ppu_write_scroll(i8 %9)
  br label %STA_done

ppuaddr:                                          ; preds = %then2
  call void @rom_ppu_write_address(i8 %9)
  br label %STA_done

ppudata:                                          ; preds = %then2
  call void @rom_ppu_write_data(i8 %9)
  br label %STA_done

else1:                                            ; preds = %else
  %16 = call i32 @puts(i8* getelementptr inbounds ([51 x i8]* @panicMsg, i32 0, i32 0))
  call void @exit(i32 1)
  unreachable

STA_done:                                         ; preds = %ppudata, %ppuaddr, %ppuscroll, %oamdata, %oamaddr, %ppumask, %ppuctrl, %then
  %17 = load i8* %A
  %18 = zext i8 %17 to i32
  call void @exit(i32 %18)
  br label %NMI_Routine

NMI_Routine:                                      ; preds = %STA_done
  unreachable

IRQ_Routine:                                      ; No predecessors!
  unreachable
}

With this framework in place, we can recompile instructions that store to memory addresses only known at runtime. Code generation for the other instructions in this assembly program is straightforward at this point.

So now we can generate a new, native program to run. But how do we get the video to actually display on the screen?

This question brings us to the next challenge.

Challenge: Parallel Systems Running Simultaneously

To find out how to get something rendering on the screen, I looked at Fergulator - an already-working NES emulator written in Go.

Fergulator correctly emulates Super Mario Brothers 1 as well as Zelda Title Screen Simulator, so it will certainly work for our purpose - understanding how the video gets onto the screen.

The answer is easy to find in this well factored codebase. Looking at the main loop in machine.go, the core logic is revealed:

cycles = cpu.Step()

for i := 0; i < 3*cycles; i++ {
	ppu.Step()
}

for i := 0; i < cycles; i++ {
	apu.Step()
}

The CPU runs one instruction and returns how many cycles the instruction took to run, and then the PPU is stepped for 3 times as many cycles, and the APU is stepped the same number of cycles as the CPU.

A problem presents itself here. Our recompiled code replaces the CPU stepping, but we still have the PPU and the APU code to reckon with. We can start by eliminating audio from the equation and dealing with sound later. But no matter how you spin it, the fact remains that the PPU and the CPU run independently of one another, and at differing speeds.

You can choose to recompile program code and run that as the main loop, but then after every instruction you must emulate the PPU for 3 times as many cycles as the instruction took. Or you can choose to run the PPU as the main loop, but then after the appropriate amount of cycles you must emulate the CPU for one instruction.

One of the systems must be emulated.

This is how I solved the problem:

• Have code generation call rom_cycle, an external function, after every instruction completes, passing the number of cycles the instruction took as a parameter.
• Bundle in a runtime with the generated executable which implements rom_cycle and emulates the PPU for the appropriate amount of steps.

It is a shame that we have to do some amount of emulation in this project, where the goal is to statically recompile as much as possible. The solution to this challenge represents a slight compromise to the project's integrity. Yet we press on.

Given this solution, I ported the PPU code as well as the SDL and OpenGL front-end code from Fergulator to a small C runtime, which is compiled with clang. Here is a snippet explaining the rom_cycle and main functions:

#include "rom.h"
#include "assert.h"
#include "ppu.h"
#include "SDL/SDL.h"
#include "GL/glew.h"

Ppu* p;

// This function is called by the generated module after every instruction.
void rom_cycle(uint8_t cycles) {
    // Check the SDL event loop and quit if the Close event occurs.
    flush_events();

    // Step the PPU for 3 times number of cycles that just finished.
    for (int i = 0; i < 3 * cycles; ++i) {
        Ppu_step(p);
    }
}

// This function is our new main entry point. We rename the
// main rom entry point to `rom_start` so that we can call it
// from this function.
int main(int argc, char* argv[]) {
    // Create a new instance of the PPU emulator core.
    p = Ppu_new();

    // The PPU code will call `render` when there is a frame ready to display
    // on the screen. The `render` function performs the SDL and OpenGL
    // calls to render the frame to the window.
    p->render = &render;

    // Remember that in the generated rom module, we export the ROM mirroring
    // setting after reading it from the .nes file. Here we use it to configure
    // the nametable code.
    Nametable_setMirroring(&p->nametables, rom_mirroring);

    // We currently only support ROMs with 1 CHR bank.
    assert(rom_chr_bank_count == 1);

    // In the generated rom module, we have the CHR ROM data in memory, as well
    // as `rom_read_chr`, an exported function which will copy the data to a
    // pointer. We use that to initialize the video RAM.
    rom_read_chr(p->vram);

    // This does the SDL and OpenGL setup such as creating the display window.
    init_video();

    // Here we call into the main entry point to the recompiled ROM code.
    rom_start();

    // Free up memory associated with the PPU emulator instance.
    Ppu_dispose(p);
}

uint8_t rom_ppu_read_status() {
    return Ppu_readStatus(p);
}

void rom_ppu_write_control(uint8_t b) {
    Ppu_writeControl(p, b);
}

void rom_ppu_write_mask(uint8_t b) {
    Ppu_writeMask(p, b);
}

void rom_ppu_write_oamaddress(uint8_t b) {
    Ppu_writeOamAddress(p, b);
}

void rom_ppu_write_address(uint8_t b) {
    Ppu_writeAddress(p, b);
}

void rom_ppu_write_data(uint8_t b) {
    Ppu_writeData(p, b);
}

void rom_ppu_write_oamdata(uint8_t b) {
    Ppu_writeOamData(p, b);
}

void rom_ppu_write_scroll(uint8_t b) {
    Ppu_writeScroll(p, b);
}

Notice that we include rom.h. This is a header file that defines the contract that the generated rom module will fulfill:

#include "stdint.h"

enum {
    ROM_MIRRORING_VERTICAL,
    ROM_MIRRORING_HORIZONTAL,
    ROM_MIRRORING_SINGLE_UPPER,
    ROM_MIRRORING_SINGLE_LOWER,
};

uint8_t rom_mirroring;
uint8_t rom_chr_bank_count;

// write the chr rom into dest
void rom_read_chr(uint8_t* dest);

// starts executing the PRG ROM.
// this function returns when the the program exits.
void rom_start();

// called after every instruction with the number of
// cpu cycles that have passed.
void rom_cycle(uint8_t);

// PPU hooks
uint8_t rom_ppu_read_status();
void rom_ppu_write_control(uint8_t);
void rom_ppu_write_mask(uint8_t);
void rom_ppu_write_oamaddress(uint8_t);
void rom_ppu_write_oamdata(uint8_t);
void rom_ppu_write_scroll(uint8_t);
void rom_ppu_write_address(uint8_t);
void rom_ppu_write_data(uint8_t);

When we build the final executable file, we can link against this runtime to get a fully operational executable with a video display.

After adding runtime compilation instructions, make does this:

$ make
go tool yacc -o jamulator/y.go -v /dev/null jamulator/asm6502.y
/home/andy/gocode/bin/nex -e jamulator/asm6502.nex
clang -o runtime/main.o -c runtime/main.c
clang -o runtime/ppu.o -c runtime/ppu.c
clang -o runtime/nametable.o -c runtime/nametable.c
ar rcs runtime/runtime.a runtime/main.o runtime/ppu.o runtime/nametable.o
go build -o jamulate main.go
$

Next we can implement a command which will read a .nes ROM file and perform the recompilation process:

$ ./jamulate -recompile roms/Zelda.NES
loading roms/Zelda.NES
Disassembling...
Decompiling to /tmp/302858262/prg.bc...
llc -o /tmp/302858262/prg.o -filetype=obj /tmp/302858262/prg.bc
gcc /tmp/302858262/prg.o runtime/runtime.a -lGLEW -lGL -lSDL -lSDL_gfx -o roms/Zelda
Done: roms/Zelda
$ file ./roms/Zelda
./roms/Zelda: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=0xa79ee93a78a4745b60ba9d10a58b209d756e8ffa, not stripped
$

And so here we have it: an executable binary that depends on SDL and OpenGL, which supposedly will execute the Zelda Title Screen Simulator when run. Does it work?

Note: There is an obvious discrepancy between the screenshot and the output listed above. All output is real; the difference is that the article is organized, ordered, and simplified for clarity and understanding. I did not necessarily write the code in the same order that it is listed in this article.

So we made a small concession to solve this challenge, and we have a simple demo NES ROM running natively. Next let's see if we can fix the dent in that sword.

Challenge: Handling Interrupts

In NES programming, after any instruction, it is possible that the program counter is yanked away from the next expected instruction and instead sent to a predefined location. This is called an interrupt.

There are 3 kinds of interrupts in NES programming:

Reset

Occurs when the user presses the reset button. This is also where the program counter starts when the NES powers on.

IRQ

Stands for Interrupt ReQuest. This interrupt can only be fired if a game uses a mapper or uses the BRK instruction. This request can be enabled and disabled with the CLI and SEI instructions.

NMI

Stands for Non-Maskable Interrupt because there are no instructions to enable and disable this interrupt. It occurs when the vertical blank begins.

When an interrupt occurs, the program counter and status register are pushed onto the stack, and the program counter is set to the location defined by the interrupt vector table, as seen in code listings above:

.org $fffa
    .dw NMI_Routine
    .dw Reset_Routine
    .dw IRQ_Routine

When a program has finished processing an interrupt, it typically executes the RTI instruction to return control flow back to where it was before the interrupt occurred.

We don't yet have to deal with handling the reset button being pressed, mappers, or games which execute the BRK instruction, so let's work on solving this problem for the NMI interrupt only.

The PPU performs the vertical blanking which signals the NMI interrupt, and the PPU emulation code is in the runtime. In order to handle NMI interrupts correctly, we have to be ready to jump to the interrupt routine after every single instruction.

We already call into the runtime by calling rom_cycle after every instruction. This is where the PPU emulation is performed, which is where the NMI interrupt is generated. Given this, a natural solution might be something like this:

• Have rom_cycle return a value indicating which interrupt, if any, has occurred.
• After every instruction, check if an interrupt has occurred, and if so, jump to the interrupt routine. Otherwise, continue.

This solution is rather ugly, however. Branching after every instruction decreases execution speed and executable leanness. Further, it leaves a critical problem unsolved: how to jump back to where the program counter was before the interrupt occurred. The solution I came up with instead feels better, but it comes with a caveat. Here's the idea:

• Switch the register variables from being allocated on the stack in rom_start, to global variables in the generated module.
• Update rom_start, the main entry point, to take a parameter indicating which interrupt vector to execute.
• In the runtime, when rom_start is called for the first time, pass Reset as the interrupt vector to execute.
• In the generated rom_start code, insert code at the beginning of the NMI interrupt routine block to push the program counter and the processor status to the stack.
• In rom_cycle, if there is no interrupt, return as normal. However, if there is an interrupt, call rom_start, passing the correct interrupt vector to execute.
• Generate a return statement for the RTI instruction.

The beauty of this solution is that it uses the real native stack for interrupts, in what is probably the most efficient and elegant way to get the desired behavior.

The caveat is that if the game uses RTI for its side-effects instead of the usual "return from interrupt" behavior, the executable will unexpectedly exit. Further, if the game simulates returning from an interrupt without using RTI, the game will crash due to a stack overflow.

Acknowledging these weaknesses, let's plow ahead until we are forced to solve this problem a different way.

Let's see what it looks like to implement the plan.

Update rom_start to take a parameter indicating which interrupt vector to execute:

func (p *Program) Compile(filename string) (c *Compilation) {
	// ...

	mainType := llvm.FunctionType(llvm.VoidType(), []llvm.Type{llvm.Int8Type()}, false)
	c.mainFn = llvm.AddFunction(c.mod, "rom_start", mainType)
	c.mainFn.SetFunctionCallConv(llvm.CCallConv)

	// ...
}

Add a reset vector enum to rom.h:

enum {
    ROM_INTERRUPT_NONE,
    ROM_INTERRUPT_NMI,
    ROM_INTERRUPT_RESET,
    ROM_INTERRUPT_IRQ,
};

In the runtime, when rom_start is called for the first time, pass Reset as the interrupt vector to execute. Also, in rom_cycle, if there is no interrupt, return as normal. However, if there is an interrupt, call rom_start, passing the correct interrupt vector to execute:

int interruptRequested = ROM_INTERRUPT_NONE;

void rom_cycle(uint8_t cycles) {
    flush_events();

    for (int i = 0; i < 3 * cycles; ++i) {
        Ppu_step(p);
    }

    int req = interruptRequested;
    if (req != ROM_INTERRUPT_NONE) {
        interruptRequested = ROM_INTERRUPT_NONE;
        rom_start(req);
    }
}

void vblankInterrupt() {
    interruptRequested = ROM_INTERRUPT_NMI;
}

int main(int argc, char* argv[]) {
    p = Ppu_new();
    p->render = &render;

    // The PPU code will call this function when an NMI interrupt occurs.
    p->vblankInterrupt = &vblankInterrupt;

    Nametable_setMirroring(&p->nametables, rom_mirroring);
    assert(rom_chr_bank_count == 1);
    rom_read_chr(p->vram);
    init_video();

    // Start the rom executing at the Reset interrupt routine.
    rom_start(ROM_INTERRUPT_RESET);

    Ppu_dispose(p);
}

In the generated rom_start code, insert code at the beginning of the NMI interrupt routine block to push the program counter and the processor status to the stack:

func (c *Compilation) addNmiInterruptCode() {
	c.builder.SetInsertPointBefore(c.nmiBlock.FirstInstruction())
	// * push PC high onto stack
	// * push PC low onto stack
	c.pushWordToStack(c.builder.CreateLoad(c.rPC, ""))
	// * push processor status onto stack
	c.pushToStack(c.getStatusByte())
}

func (c *Compilation) pushWordToStack(word llvm.Value) {
	high16 := c.builder.CreateLShr(word, llvm.ConstInt(llvm.Int16Type(), 8, false), "")
	high := c.builder.CreateTrunc(high16, llvm.Int8Type(), "")
	c.pushToStack(high)
	low16 := c.builder.CreateAnd(word, llvm.ConstInt(llvm.Int16Type(), 0xff, false), "")
	low := c.builder.CreateTrunc(low16, llvm.Int8Type(), "")
	c.pushToStack(low)
}

func (c *Compilation) pushToStack(v llvm.Value) {
	// write the value to the address at current stack pointer
	sp := c.builder.CreateLoad(c.rSP, "")
	spZExt := c.builder.CreateZExt(sp, llvm.Int16Type(), "")
	addr := c.builder.CreateAdd(spZExt, llvm.ConstInt(llvm.Int16Type(), 0x100, false), "")
	c.dynStore(addr, v)
	// stack pointer = stack pointer - 1
	spMinusOne := c.builder.CreateSub(sp, llvm.ConstInt(llvm.Int8Type(), 1, false), "")
	c.builder.CreateStore(spMinusOne, c.rSP)
}

func (c *Compilation) getStatusByte() llvm.Value {
	// zextend
	s7z := c.builder.CreateZExt(c.builder.CreateLoad(c.rSNeg, ""), llvm.Int8Type(), "")
	s6z := c.builder.CreateZExt(c.builder.CreateLoad(c.rSOver, ""), llvm.Int8Type(), "")
	s4z := c.builder.CreateZExt(c.builder.CreateLoad(c.rSBrk, ""), llvm.Int8Type(), "")
	s3z := c.builder.CreateZExt(c.builder.CreateLoad(c.rSDec, ""), llvm.Int8Type(), "")
	s2z := c.builder.CreateZExt(c.builder.CreateLoad(c.rSInt, ""), llvm.Int8Type(), "")
	s1z := c.builder.CreateZExt(c.builder.CreateLoad(c.rSZero, ""), llvm.Int8Type(), "")
	s0z := c.builder.CreateZExt(c.builder.CreateLoad(c.rSCarry, ""), llvm.Int8Type(), "")
	// shift
	s7z = c.builder.CreateShl(s7z, llvm.ConstInt(llvm.Int8Type(), 7, false), "")
	s6z = c.builder.CreateShl(s6z, llvm.ConstInt(llvm.Int8Type(), 6, false), "")
	s4z = c.builder.CreateShl(s4z, llvm.ConstInt(llvm.Int8Type(), 4, false), "")
	s3z = c.builder.CreateShl(s3z, llvm.ConstInt(llvm.Int8Type(), 3, false), "")
	s2z = c.builder.CreateShl(s2z, llvm.ConstInt(llvm.Int8Type(), 2, false), "")
	s1z = c.builder.CreateShl(s1z, llvm.ConstInt(llvm.Int8Type(), 1, false), "")
	// or
	s0z = c.builder.CreateOr(s0z, s1z, "")
	s0z = c.builder.CreateOr(s0z, s2z, "")
	s0z = c.builder.CreateOr(s0z, s3z, "")
	s0z = c.builder.CreateOr(s0z, s4z, "")
	s0z = c.builder.CreateOr(s0z, s6z, "")
	s0z = c.builder.CreateOr(s0z, s7z, "")
	return s0z
}

Generate a return statement for the RTI instruction:

func (i *Instruction) Compile(c *Compilation) {
	switch i.OpCode {
	// ... other cases
	case 0x40: // RTI implied
		// Restore the old status register values from the stack.
		c.pullStatusReg()

		// We get the PC from the stack, but since we rely on the native
		// stack, we trash the value.
		_ = c.pullWordFromStack()

		// This will call the `rom_cycle` function. RTI always takes
		// 6 cycles.
		c.cycle(6)

		// Generate a return statement.
		c.builder.CreateRetVoid()

		// So that when another block is encountered, we do not
		// create an unconditional branch to it.
		c.currentBlock = nil

	// ... other cases
	}
}

func (c *Compilation) cycle(count int) {
	v := llvm.ConstInt(llvm.Int8Type(), uint64(count), false)
	c.builder.CreateCall(c.cycleFn, []llvm.Value{v}, "")
}

And so here we have it: an elegant but imperfect solution to the interrupt problem. Let's see if it fixes the bent sword:

Looks like progress to me! At this point the project can recompile a small, simple title screen demo NES program. The real challenge awaits: can it be made to work for a real NES game?

There are many games to choose from. Ideally we would pick one that poses fewer additional challenges. One filter we can apply is to eliminate games that use mappers, since we have hitherto ignored mapper support entirely.

This limits the choices significantly. The only games worth noting that do not use a mapper are:

• Donkey Kong
• Ice Climber
• Excitebike
• Mario Bros.
• Super Mario Brothers 1
• Pac-Man

Of these, there is an obvious answer to which game we should support first, which, of course, is Super Mario Brothers 1.

Our next challenge is revealed when we crack open the disassembly for SMB.

Challenge: Detecting a Jump Table

Here's a small section of Super Mario Brothers 1 disassembly:

Label_8212:
    LDA $0770
    JSR Label_8e04
    AND ($82), Y
    .db $dc, $ae, $8b, $83, $18, $92

Notice anything peculiar?

After the value at $0770 is loaded into register A, and we return from the Label_8e04 subroutine, there is an uncommon indirect AND instruction, followed by data. What could possibly be happening here?

Super Mario Brothers 1 is using a common assembly programming technique called a dynamic jump table. Take a look at the Label_8e04 subroutine:

Label_8e04:
    ASL
    TAY
    PLA
    STA $04
    PLA
    STA $05
    INY
    LDA ($04), Y
    STA $06
    INY
    LDA ($04), Y
    STA $07
    JMP ($0006)

Notice that although this label is jumped to with JSR, it never uses the RTS instruction. Let's break it down further into readable pseudocode:

; Dynamic Jump Table. Call this label with JSR
; so that the old PC is on the stack.
; Immediately following the JSR statement should be
; .dw statements indicating the labels to jump to
; depending on the value of register A.
Label_8e04:
    ; Register A holds the index of the label that we wish to jump to.
    ; Multiply A by 2 because each table entry is 2 bytes.
    ASL           ; A = A * 2

    ; The useful indirect instructions use Y as the index, and we need
    ; to repurpose A.
    TAY           ; Y = A

    ; Since this label was called with JSR, the old PC is on the top
    ; of the stack. Here we get the lower byte since this is a little
    ; endian system.
    PLA           ; A = Stack.Pop()

    ; Save the lower byte of the old PC into memory.
    STA $04       ; Memory[$04] = A

    ; Get the higher byte of the old PC off the stack.
    PLA           ; A = Stack.Pop()

    ; Save the higher byte of the old PC into memory.
    STA $05       ; Memory[$05] = A

    ; JSR pushes the address - 1 of the next instruction to the stack.
    ; So we add 1 to Y to get the index of the first byte of the jump
    ; destination.
    INY           ; Y = Y + 1

    ; Get the first byte of the jump destination.
    LDA ($04), Y  ; A = Memory[Memory[$04] + Y]

    ; Save the first byte of the jump destination.
    STA $06       ; Memory[$06] = A

    ; Increment Y to get the index of the 2nd byte of the jump destination.
    INY           ; Y = Y + 1

    ; Get the 2nd byte of the jump instruction.
    LDA ($04), Y  ; A = Memory[Memory[$04] + Y]

    ; Save the 2nd byte of the jump instruction.
    STA $07       ; Memory[$07] = A

    ; Jump to the location that was just constructed.
    JMP ($0006)     ; Jump to address at $0006 - $0007

If we know that Label_8e04 is a jump table, we can mark the bytes following the JSR as .dw labels which enables us to further disassemble the program. The disassembled snippet from earlier would look like this:

Label_8212:
    LDA $0770
    JSR Label_8e04
    .dw Label_8231
    .dw Label_aedc
    .dw Label_838b
    .dw Label_9218

Without this jump table detection, the bytes at each of those labels remain .db statements, unable to be decoded. This is problematic because our strategy currently depends on all instructions being completely disassembled so that they can be decompiled and recompiled.

This is not a mere corner-case either. This technique is repeated in many games, including Super Mario Brothers 3 and Pac-Man.

Fortunately, it is straightforward to identify and process a jump table like this without changing too much code. I solved it with a state machine - essentially pattern-matching or a regular expression for instructions:

func (d *Disassembly) detectJumpTable(addr int) bool {
	const (
		expectAsl = iota
		expectTay
		expectPlaA
		expectStaA
		expectPlaB
		expectStaB
		expectInyC
		expectLdaC
		expectStaC
		expectInYD
		expectLdaD
		expectStaD
		expectJmp
	)
	state := expectAsl
	var memA, memC int
	for elem := d.prog.elemAtAddr(addr); elem != nil; elem = elem.Next() {
		switch state {
		case expectAsl:
			i, ok := elem.Value.(*Instruction)
			if !ok {
				return false
			}
			if i.OpCode != 0x0a {
				return false
			}
			state = expectTay
		case expectTay:
			i, ok := elem.Value.(*Instruction)
			if !ok {
				return false
			}
			if i.OpCode != 0xa8 {
				return false
			}
			state = expectPlaA
		case expectPlaA:
			i, ok := elem.Value.(*Instruction)
			if !ok {
				return false
			}
			if i.OpCode != 0x68 {
				return false
			}
			state = expectStaA
		case expectStaA:
			i, ok := elem.Value.(*Instruction)
			if !ok {
				return false
			}
			if i.OpCode != 0x85 && i.OpCode != 0x8d {
				return false
			}
			memA = i.Value
			state = expectPlaB
		case expectPlaB:
			i, ok := elem.Value.(*Instruction)
			if !ok {
				return false
			}
			if i.OpCode != 0x68 {
				return false
			}
			state = expectStaB
		case expectStaB:
			i, ok := elem.Value.(*Instruction)
			if !ok {
				return false
			}
			if i.OpCode != 0x85 && i.OpCode != 0x8d {
				return false
			}
			if i.Value != memA+1 {
				return false
			}
			state = expectInyC
		case expectInyC:
			i, ok := elem.Value.(*Instruction)
			if !ok {
				return false
			}
			if i.OpCode != 0xc8 {
				return false
			}
			state = expectLdaC
		case expectLdaC:
			i, ok := elem.Value.(*Instruction)
			if !ok {
				return false
			}
			if i.OpCode != 0xb1 {
				return false
			}
			if i.Value != memA {
				return false
			}
			state = expectStaC
		case expectStaC:
			i, ok := elem.Value.(*Instruction)
			if !ok {
				return false
			}
			if i.OpCode != 0x85 && i.OpCode != 0x8d {
				return false
			}
			memC = i.Value
			state = expectInYD
		case expectInYD:
			i, ok := elem.Value.(*Instruction)
			if !ok {
				return false
			}
			if i.OpCode != 0xc8 {
				return false
			}
			state = expectLdaD
		case expectLdaD:
			i, ok := elem.Value.(*Instruction)
			if !ok {
				return false
			}
			if i.OpCode != 0xb1 {
				return false
			}
			if i.Value != memA {
				return false
			}
			state = expectStaD
		case expectStaD:
			i, ok := elem.Value.(*Instruction)
			if !ok {
				return false
			}
			if i.OpCode != 0x85 && i.OpCode != 0x8d {
				return false
			}
			if i.Value != memC+1 {
				return false
			}
			state = expectJmp
		case expectJmp:
			i, ok := elem.Value.(*Instruction)
			if !ok {
				return false
			}
			if i.OpCode != 0x6c {
				return false
			}
			if i.Value != memC {
				return false
			}
			return true
		}
	}
	return false
}

Given this detection function, it is a matter of adding logic to the JSR disassembly. If a jump table is detected, mark the following bytes as .dw label statements. Otherwise, continue marking the next address as an instruction as usual.

After adding jump table detection, it looks like all the program instructions are disassembled successfully. But there are still some tricks up the assembly programmers' sleeves.

Challenge: Indirect Jumps

We just looked at a dynamic jump table implementation which included this instruction:

JMP ($0006)

This is problematic because we do not know what will be in the memory addresses $0006 and $0007 until the instruction is actually executed.

One solution is to use our own jump table. We can use our knowledge of the address of each label to create a basic block to jump to when an indirect jump is encountered:

func (c *Compilation) addDynJumpTable() {
	c.dynJumpBlock = llvm.AddBasicBlock(c.mainFn, "DynJumpTable")
	c.builder.SetInsertPointAtEnd(c.dynJumpBlock)
	pc := c.builder.CreateLoad(c.rPC, "")
	// Here, panic block causes a runtime error and crashes.
	sw := c.builder.CreateSwitch(pc, panicBlock, len(c.dynJumpAddrs))
	for addr, block := range c.dynJumpAddrs {
		addrVal := llvm.ConstInt(llvm.Int16Type(), uint64(addr), false)
		sw.AddCase(addrVal, block)
	}
}

And when a JMP indirect is encountered:

func (i *Instruction) Compile(c *Compilation) {
	switch i.OpCode {
	// ... other cases

	case 0x6c: // jmp indirect
		newPc := c.loadWord(i.Value)
		c.builder.CreateStore(newPc, c.rPC)
		c.cycle(5)
		c.builder.CreateBr(c.dynJumpBlock)
	 	c.currentBlock = nil

	// ... other cases
  }
}

This ends up looking something like this in the generated module:

DynJumpTable:
  %63674 = load i16* @PC
  switch i16 %63674, label %PanicBlock [
    i16 -11559, label %Label_d2d9
    i16 -3527, label %Label_f239
    i16 -27243, label %Label_9595
    i16 -28888, label %Label_8f28

    ; (about 2,000 more cases)

    i16 -3045, label %Label_f41b
    i16 -18147, label %Label_b91d
  ]

This solution will work as long as the indirect jump chooses to jump to one of the labels. However it causes a runtime crash if the indirect jump sets the PC to anything else. As we will soon find out, assembly programmers not only force the processor to do this, but even more heinous acts.

Challenge: Dirty Assembly Tricks

Super Mario Brothers 1 is an amazing technical feat. Every last byte of the 32KB available program space is utilized. In fact, some bytes are even dual purposed to save space. Have a look at this code from our Super Mario 1 disassembly:

Label_8220:
    LDY #$00
    .db $2c
Label_8223:
    LDY #$04
    LDA #$f8
Label_8227:
    STA $0200, Y
    INY
    INY
    INY
    INY
    BNE Label_8227
    RTS

Note the .db $2c.

$2c is the op code for BIT absolute, which is 3 bytes - 1 byte op code and then 2 bytes for the absolute address.

LDY #$04 is 2 bytes - $a0 for the op code and then $04 for the immediate value.

So how this works is if you jump to Label_8220, it does Y = $00, and then sabotages the next instruction, causing the Y = $04 to not happen. Instead, the BIT instruction sets some status bits in a way that does not matter. Then it picks up the next instruction, LDA #$f8, as if you had jumped to Label_8223 with a different Y.

This occurs over a dozen times in Super Mario 1. Similarly, there are instances where the program jumps into the middle of an instruction.

Yet another trick is adding an address on the stack and then using the RTS instruction to jump there, kind of like a homebrew JMP indirect instruction. And even with indirect JMP instructions, the programmer may choose to jump to RAM or somewhere other than a label.

These issues must be resolved if we want a playable game. Sadly, the solution marks the final nail in the coffin of the integrity of this project.

The solution is to embed an interpreter runtime in the generated binary:

• Instead of identifying data that is read and only including that in the generated module, include all the PRG ROM, since we don't know which addresses may be accessed at runtime.
• After every instruction, update the program counter variable.
• Include a basic block in rom_start called Interpret which reads the program counter variable, reads the op code from the PRG ROM, performs the necessary operation, and then jumps to the DynJumpTable block.
• Update the DynJumpTable block so that the default case jumps to the Interpret block instead of panicking.
• When control flow runs into data, branch to the Interpret block.

Here's a diagram to help clarify:

This strategy ensures that NES games will run correctly, at the cost of efficiency. Normally, doing something like updating the program counter variable after every instruction would be optimized away, but this is thwarted by interrupts in our case. Because after every instruction we call rom_cycle, which in turn might call rom_start with an interrupt, all the global variable state must be correct before we call rom_cycle. This defeats the entire point of this project. At this point we might as well emulate. In fact, with the new Interpret block, we are doing just that.

Although, to be fair, I only had to add emulation support for 6 op codes to get Super Mario Brothers 1 working.

Video: Playing Super Mario Brothers 1 on Native Machine Code

In this video I demonstrate my (poor) Super Mario 1 skills in a recompiled executable. I also demonstrate the movie playback feature that was instrumental in debugging.

Unsolved Challenge: Mappers

At this point in the project we have Super Mario Brothers 1 running mostly on native code, although not very highly optimized. We've learned that static compilation, while possible, is rendered pointless by some of the inherent challenges that emulating a system presents. Thus there is no reason to solve the challenge that mappers provide.

The only thing I will say about mappers here is that they present an additional layer of complexity for static disassembly, and they make it blatantly obvious that Just-In-Time compilation is a better technique than static recompilation. More on that in the conclusion.

Community Support

NES seems to be a common system for first-time emulator programmers. As such, I was happy to find a large swath of documentation online explaining in great detail how the NES works, emulator tutorials, fascinating optimization articles, the invaluable Nesdev wiki, and more. Even so, there is nothing like asking a question and having a knowledgeable person answer in real-time. The folks in #nesdev on EFNet are fun, engaging, working on all kinds of interesting projects, and helpful. Thanks especially to Ulfalizer, sherlock_, Bisqwit, Movax12, and scottferg for answering my questions, even when they seemed stupid. The Nesdev Forums are nice as well.

There were also instances where I asked for some help with Go - how to write good code, what is the best way to implement certain things, etc. #go-nuts on Freenode is great for that. The channel is active - you'll nearly always get an answer immediately.

As for my contributions back to the community, I did help scottferg fix a bug in his emulator, which fixed support for Maniac Mansion.

I also filed an llvm feature request asking for the ability to generate comments in IR code. This would make it much, much easier to debug generated IR code.

Conclusion

After completing this project, I believe that static recompilation does not have a practical application for video game emulation. It is thwarted by the inability to completely disassemble a game without executing it as well as the fact that multiple systems are executing in parallel, possibly causing interrupts in the game code. There is a constant struggle between correctness and optimized code. Nearly all optimizations must be tossed out the window in the interest of correctness. Even more compromises would have to be made to start supporting advanced emulator features such as saving state or rewinding.

A comparison could be made between a console game and an interpreted language such as JavaScript. There are amazingly fast JavaScript interpreters such as V8, but they do not work by statically compiling the script. Instead they use Just-In-Time compilation, along with some advanced techniques, to achieve great speeds. These techniques could be applied to JIT console game compilation.

For example, one such technique is to identify a section of code, make some assumptions based on heuristics which allow for highly optimized native code generation, and then detect if those assumptions are broken. If the assumptions are broken, the generated native code is tossed, and emulation takes over. However, if the assumptions are upheld, the recompiled block of code will execute with blazing fast native speed.

This technique is much more suited to an emulator with a JIT core, rather than trying to do everything statically, especially since the emulator can "notice" at runtime which memory addresses are used as instructions and which memory addresses are used as data.

Furthermore, distributing static executables that function as games would be problematic as far as copyright infringement is concerned. By keeping ROMs separate from the emulator executable, the emulator can be distributed freely and easily without risking trouble.

That being said, I do feel like this project was worthwhile in that it was intellectually stimulating and highly effective at teaching me, and hopefully now, you, how the Nintendo Entertainment System works, how to use LLVM effectively, and an introduction to a wide range of problems that compilers and interpreters face.

Contents

1. Sending Automated Emails in Node.js
2. node-email-templates Gotchas
3. Fundamental Flaws
   1. Includes VS Template Inheritance
   2. Sharing CSS
   3. Dummy Context
4. Conclusion

Sending Automated Emails in Node.js

When I was tasked with solving the age-old problem of sending automatic email messages to our users at Indaba Music, I surveyed the Node.js landscape to find out the state of affairs.

I was pleased to immediately discover node-email-templates and Nodemailer, and within two days had a proof of concept email notification server deployed to production.

node-email-templates helps organize your project and make it easy to render templates for sending via email by using juice to inline css.

Nodemailer does the actual email dispatching - given an email dispatch service, a subject, to, and body, Nodemailer will get your mail to its destination.

Nodemailer is a wonderful piece of software. It worked, and continues to work, exactly as advertised and without any hiccups. It even has a convenient API that makes integration with common email dispatchers, such as SendGrid (which I also recommend), quite painless.

Unfortunately this journey was not without a few obstacles that node-email-templates provided for me to solve.

What follows is an explanation of the bumps along the road that caused me to write these modules to improve the state of email templates in node.js:

• boost
• swig-dummy-context
• swig-email-templates

More on these in a bit.

node-email-templates Gotchas

Assume that I have 2 templates named reminder and notice.

node-email-templates requires your project to have a folder structure that looks like this:

./templates/reminder/html.ejs
./templates/reminder/text.ejs
./templates/reminder/style.css
./templates/notice/html.ejs
./templates/notice/text.ejs
./templates/notice/style.css

There are several problems here. There are some module smells:

• Regardless of whether or not you need text version of a template, you must have the text.ejs file there or node-email-templates will throw an error.
• This project is poorly maintained. As of this writing, daeq submitted a pull request to fix the above problem 3 months ago, and despite a promise to merge it, niftylettuce still has not done so.

But there are also some fundamental problems with the approach that the module takes.

Fundamental Flaws

• This flavor of ejs is limited. You can do includes, but not layouts or template inheritance, which is where the true value of using templates comes in.
• The html templates have no way of sharing css between them.
• Because ejs depends on using eval, it is impossible to, given a template, create a dummy context with which to generate a preview of the template. More on this later.

Includes VS Template Inheritance

To demonstrate the template inheritance problem, let me give you 2 versions of a template, one using ejs with includes, and one using swig with template inheritance:

ejs includes

notice.ejs

<% include header %>
<div>
  <p>Hey <%= username %>,</p>
  <p>This is a notice that your offer is about to expire.</p>
</div>
<% include footer %>

reminder.ejs

<% include header %>
<div>
  <p>Hey <%= username %>,</p>
  <p>Don't forget! You probably wanted to do that thing.</p>
</div>
<% include footer %>

header.ejs

<div>
  <img src="logo.png">
</div>

footer.ejs

<div>
  Super Cool &amp; Co., LLC.
  <a>Privacy Policy</a>
</div>

swig template inheritance

reminder.html

{% extends "base.html" %}

{% block content %}
  <p>Don't forget! You probably wanted to do that thing.</p>
{% endblock %}

notice.html

{% extends "base.html" %}

{% block content %}
  <p>This is a notice that your offer is about to expire.</p>
{% endblock %}

base.html

<div>
  <img src="logo.png">
</div>
<div>
  <p>Hey {{ username }},</p>
  {% block content %}
  {% endblock %}
</div>
<div>
  Super Cool &amp; Co., LLC.
  <a>Privacy Policy</a>
</div>

Template Inheritance Wins

This is an oversimplified example, but even so it starts to become obvious why template inheritance is superior to includes.

You can also have includes in swig, by the way.

Sharing CSS

node-email-templates uses juice to inline css. Give juice html and css, and it returns html with the css inlined on each element for maximum email client compatibility.

This setup seems good at first, but it is crippled by the fact that templates are completely unable to share css. Each template has its own independent style.css file.

It is not node-email-templates's fault. Given the way that juice works, it isn't really possible to share css.

This is where boost comes in. boost depends on juice and adds 2 key features that make sharing CSS possible.

• Ability for the html to have <link rel="stylesheet"> tags which are resolved correctly and have the resulting CSS applied.
• Ability to have <style>...</style> elements and have that CSS applied as well.

When you add this capability with template inheritance, sharing CSS becomes a solved problem. For example:

base.html

<html>
<head>
  {% block css %}
    <link rel="stylesheet" href="base.css">
  {% endblock %}
</head>
<body>
  {% block content %}
  {% endblock %}
</body>
</html>

reminder.html

{% extends "base.html" %}

{% block css %}
  {% parent %}
  <link rel="stylesheet" href="reminder.css">
{% endblock %}

{% block content %}
  <h1>Reminder</h1>
  <p>Don't forget!</p>
{% endblock %}

notice.html

{% extends "base.html" %}

{% block css %}
  {% parent %}
  <link rel="stylesheet" href="notice.css">
{% endblock %}

{% block content %}
  <h1>Notice</h1>
  <p>This is a notice.</p>
{% endblock %}

Dummy Context

In order to rapidly build email templates, we need to see what we are building as we build it.

Because you need to supply a template with a context in order to render it and see it, this makes seeing what you are doing while you build templates a two step process.

We can remove this extra step by taking advantage of swig-dummy-context, a module I wrote which, given a swig template, gives you a "dummy" context - a fill-in-the-blank structure you can use to immediately preview your template.

Given:

<div>
  {{ description }}
</div>
{% if articles %}
  <ul>
  {% for article in articles %}
    <li>{{ article.name }}</li>
  {% endfor %}
  </ul>
{% else %}
  <p>{{ defaultText }}</p>
{% endif %}

swig-dummy-context produces:

{
  "description": "description",
  "articles": {
    "name": "name"
  },
  "defaultText": "defaultText"
}

And if you render the template with the generated dummy context, you get:

<div>
  description
</div>
<ul>
  <li>name</li>
</ul>

Conclusion

swig-email-templates gives you all the ingredients you need to build well-organized templates and gives you the tooling that you need to build a live preview tool.

At Indaba Music we have such a tool. It lets you select a template from the templates folder and fill in the substitutions to preview how an email will look. To be extra sure of how an email will render, you can use the tool to send a test email to your email address.

This tool is currently private as it is not decoupled from SendGrid or even polished up for 3rd party use at all; however if there is sufficient interest I may open source it.

A short series to help you win. Covers these topics:

1. What PyWeek is, and Why You Should Participate
2. Video Games in Python
3. Speed Developing
4. Game Theory - How to Create Fun
5. Getting the Most Out of the Community
6. The Psychology of Judging
7. Conclusion

What PyWeek is, and Why You Should Participate

PyWeek is a challenge. It asks you to spend one week of your life developing a new video game, using Python.

PyWeek is difficult. It is not easy to create a new video game in just one week, especially one that is fun, innovative, and polished. But it is fun to try, especially with peers from all around the world working alongside you toward the same goal.

PyWeek will strengthen you as a game developer. If you go solo it will strengthen you as a generalist, and if you join a team it will strengthen you as a team member.

As of this writing, the next competition is the week of September 11, 2011 to September 18, 2011. Registration opens August 12, 2011.

You can check out the competition's Rules and FAQ.

Video Games in Python

Python is a great language for rapid video game development:

• Large selection of libraries to use, for example PyGame and Pyglet
• Python is known for being a prototyping language due to its ease of use.
• It is simple to create arbitrary one-off data structures that you often need in game programming.
• You can use objects and inheritance if you need it.
• Cross platform

There are some drawbacks though:

• Deployment is cumbersome. For PyWeek this isn't too much of an issue since everyone will run from source, but for an official game, it would be an issue.
• Code is about 10x less efficient than in C or C++. This will not matter for graphics on screen, since you will likely use a game library which uses the GPU for graphics, but it might make your main loop or physics slow.

Just keep these things in mind when developing.

Speed Developing

The biggest thing preventing you from success during PyWeek is lack of time. Therefore, the more time you conserve, and the more time-consuming things you can do before the development week, the better chances you have for success.

Allocate as much time as possible.

PyWeek is inspired by the Ludum Dare competition[1], which only lasts 48 hours. In that competition, you pretty much have to use all 48 hours if you want to be successful. PyWeek is one week long to make it easier on people who have commitments to things other than the Internet. Regardless, your success will be directly proportional to the amount of time you spend on PyWeek.

• Get ahead at work and/or school so that you can spend as little time on them as possible.
• Make sure your significant other knows you don't want to be bothered this particular week. You'll make up for it later.
• Remember not to agree to go to any social activities that will take place during PyWeek. If someone asks you to hang out, try to change the time to before or after.

Use version control.

You may be tempted to forgo version control, in the interest of time. One of two things are true. Either,

1. You are new to version control, and you honestly would spend more time struggling with the version control software than developing your game, or
2. You are comfortable with version control, but think that you can save time by not using it since this is a speed-developing contest.

In the first case, you should use version control anyways, because that skill should take priority over game development, and this is a great opportunity to learn. GitHub makes it dead simple if you are okay with making your project open source.

In the second case, let me assure you that version control will actually save you time:

• When you're tired and you forgot what you were working on, you can use the diff tool to quickly remember what you were working on.
• You can spend a few minutes working on a feature or experimenting with a different style of gameplay to see if it's fun. If not, it's one command to scrap it and try again.
• If you're used to using version control and you forgo it for the competition, it will throw off your groove to not use it, messing with your psychology in a time-wasting manner.

Practice beforehand.

Development is all about solving problems.

There are some problems you are bound to run into that are simply a function of Python, the game library you are relying on, and the fact that you are writing a game. You would run into these problems no matter what game you decided to write.

Therefore, it makes sense to solve these problems before the actual competition. Practice writing a simple game engine with the library of your choice before the competition. You will run into and solve some issues that you would have otherwise had to waste time solving during the competition.

Examples of things you may want to practice:

• Basic game engine / main loop
• Displaying graphics
• Animating things on the screen
• Responding to user input
• Sound playback
• Transitioning state between a title screen or menu system and gameplay.
• 3D graphics
• Networking
• Deploying on Windows, Mac, and Linux

Practice using the tools you are going to use as well. Before the competition starts, you should feel comfortable creating assets, such as:

• Art
• Animations
• Sound effects
• Music

Often you can make at least half of your sound effects using only a microphone, Audacity, and some creativity.

PyWeek allows you to use artwork, music, and sounds whose copyrights explicitly allow you to use them. If you are going to use other people's artwork, find the databases and websites where you are going to get it from and make sure the citation process is efficient and will work.

Have your game idea ready before the theme is decided.

You have one week between finding out what the 5 possible themes are, and the start of development. Use this week to brainstorm several game ideas for each theme, before you even know which one will win.

Brainstorming is a lengthy process. Sometimes it can help to give yourself fake limitations in order to get your brain to think of ideas. For each possible theme, try to think of a game that fits each category:

• Action
• Adventure
• Puzzle
• Role-Playing
• Simulation
• Other - a crazy wacky idea that probably will be stupid

Sometimes trying to think of an idea for one category will make you think of an idea for another category. That's great, write it down! And sometimes trying to think of an idea for one theme will make you think of an idea for a different theme. That's great, write it down! At this point you want as many ideas as possible.

A friend or room mate can be very useful when brainstorming ideas. Bouncing ideas off each other is a great way to come up with material.

Once the theme is decided, you can often pick your best idea, regardless of what theme you thought of it for, and adapt it to fit the real theme. My winning entry, Lemming, was originally an idea inspired by the "Fry Cook on Venus" theme, but I adapted it to the "Nine Times" theme when it was chosen. [2]

Rely on other projects as much as possible.

As a general rule, do not try to write a level editor during PyWeek. You need as much time as possible to spend on the game itself. Consider using Tiled and DR0ID's tiledtmxloader.

Get enough sleep.

Your body needs sleep. You cannot cheat around that fact. The longer you sleep-deprive yourself, the less clearly you will think, the more mistakes you will make, and the more slowly you will work.

When you rest, you give your unconscious mind a chance to sort things out and surface solutions and ideas that you wouldn't have thought of consciously.

Sleep is directly linked to creativity.[3]

Game Theory - How to Create Fun

It is easy to start programming and create a simple platformer, without thinking about what makes platformers fun. It is easy, but not nearly as rewarding as it is to actually think about Game Theory - a fascinating subject that philosophers have been thinking about for a very long time.

Think about game theory.

If you have never read that Wikipedia page, do. Be sure to check out the simple example, Prisoner's Dilemna, and especially the See Also section of that page for other great examples.

In short, you should be constantly asking yourself these questions while designing your game:

• What do I want the player to try to do?
• What is the player actually motivated to try to do?
• Is what I want the player to try to do the same as what the player is actually motivated to try to do? Why or why not, exactly? What elements naturally drive the player to act in a certain way?
• Is what I want the player to try to do fun? What exactly makes it so, or not so?
• Are the player's short term goals conflicting with the player's long term goals? (This is one way to create good gameplay.)
• What compels the player to keep playing?
• Is the player compelled to keep playing compulsively, or because they are having fun?
• Do I want the experience of playing to be the same every time, or have random elements that change up the gameplay?
• Are there any parts of the gameplay that are definitely not fun?
• Is it possible to make it fun to lose?
• What forces is the player working against?
• Does the player feel like the forces working against him are acting fairly?

There is a plethora of reading (and watching) material on creating interesting gameplay. Here are some materials to get you started thinking about game design:

• Extra Credits: The Skinner Box
• Extra Credits: Narrative Mechanics

Get your game into a playable state as soon as possible.

Developing a game is an evolutionary process. The sooner you have a playable game, the sooner you can begin to morph it and mold it into a better and better game.

Ask yourself if it is fun. Try to figure out what makes it fun, and expand on those elements of gameplay.

Try out your gameplay on friends and see what they do. Try to not talk to them while they are playing. Figure out what parts your friends get frustrated at and change those parts.

Sketch your levels on paper before coding anything.

This will give you an idea of the world you're trying to create, giving you focus and direction when you code.

Getting the Most Out of the Community

PyWeek is only as fun as the people who participate. After all, there are no prizes, no reason to compete, other than for personal development and community interaction.

You get what you give.

Keep a journal.

One great way to participate is to write a diary entry after each day of work. Write about:

• Any progress made
• Things that are impeding progress
• Lessons learned that you want to remember for next time
• Teaser screenshots, art, videos, sounds, or music
• Decisions that you are pondering about in your game design

It is fun for others to follow along with your progress and for you to follow along with others. It is customary to be positive, uplifting, and encouraging in response to diary entries, but don't be afraid to tactfully include constructive criticism. On the flip side, be ready to accept critique, using it to improve your design.

Join the IRC channel.

You can feel the camaraderie by hanging out in #pyweek on Freenode.

Live chat is personal and you can get to know the other participants fairly well. This place is especially fun during judging time. At one point we broke out into a spontaneous one-hour game development competition. There is almost always someone ready to give instant feedback about a game design decision or to give tips on development in general.

One thing you want to be careful of, however, is spending too much time distracted by IRC during development time. You should be spending almost all your development time developing.

Follow along with the forum discussion.

Diary entries are posted to the forum so that everyone sees them and can comment on them.

Play and judge as many games as you can.

As developers, we thrive on feedback. When it is time to judge, give as much valuable and honest feedback as you can to as many games as you can stand.

Giving a silly award can be a fun way to give praise to a weird aspect of a game.

The Psychology of Judging

Realize that in the end you are creating a demo. Your project is not going to be a polished, ready-to-ship game in one week. Your goal is to make sure judges play your entire game experience and do not get distracted by nuances.

Test your game on several people.

Find a few friends who have not been following along with your development and shove the game in front of their face. Set them up with the title screen, but say nothing else.

Watch their face, watch their hands, and watch the screen. You are now getting an accurate preview of how judges will react to your game. Pay attention to what parts of your gameplay they don't understand. Notice when they ignore instructions you didn't want them to. Take note of what parts they get stuck at. All of these things are what you need to work on. If you don't, this is everything that judges will talk about in your game's feedback.

One mistake I made was testing my game on someone who is exceptionally good at video games. He didn't have too much of a problem with the levels that the real judges simply got stuck at and gave up.

Remember that judges have hundreds of games to play. If at any time during your game they get stuck, they are likely to shrug and move on to a different game.

The first few minutes of your game should be ridiculously easy.

It is critical that you gradually ramp up the difficulty, starting off stupidly easy.

Nobody likes a game that treats the player like an idiot. However, a player is far more willing to tolerate a Level 1 that they can pass without thinking, than they are to try to disassemble a masterpiece puzzle at the very beginning. Judges are much more likely to give up and move on.

As long as each new section, or level, is at least a little tiny bit more challenging than the last, the player will be intrigued and want to keep playing.

There is a time and a place for ridiculously hard challenging bonus levels, and that time and place is after the player has beaten the normal game, as a reward for their success. The test level that you use for 90% of your debugging will soon become boring and easy to you, but ridiculously hard for a new player. It has no place in the normal game; it should be in bonus-land.

Make stupidly easy levels come first, and then slowly introduce your more complex gameplay elements as the player progresses and gains confidence.

Again, realize that your PyWeek game is essentially a demo. You might want to include a level select on the title screen, so that a frustrated judge can skip levels that were hard in ways that you did not expect. The easier you make it for fellow PyWeekers to experience the fullness of your creation, the better feedback you will get.

Post a walkthrough with your final submission.

Once again, the goal here is to get each judge to experience all of your content, not to stump them with your enigmatic gameplay.

Post a clear and detailed walkthrough that explains how to get through and beat each level.

Reading instructions is boring.

No matter how complex your gameplay is, there is no excuse for making your players read a wall of text.

There should be no instructions necessary. The player should be able to learn the gameplay by immediately beginning to play, and then being gradually given small bits of learning to digest. Tutorial levels are highly encouraged.

Remember that one of the categories you are being rated on is Fun[4], and reading instructions is not fun.

Remember that not everyone has your setup.

Some people use alternate keyboard layouts.

In PyWeek 12, 11% of participants used a keyboard other than QWERTY.[5]

Be polite to alternate keyboard users. Defaulting to WASD is acceptable, but the player should be able to remap the keys.

A config file is an acceptable method for remapping keys, as long as it is obvious how to do it without a lot of instruction-reading.

However, alternate keyboard users are often proactive, and willing to put forth a bit of extra effort to compensate for their questionable life choices*. If you are running low on development time, it would be better to focus on gameplay rather than being nice to alternate keyboard users.

*Note: I am a Dvorak user.

Most operating systems are case sensitive.

Time to pick on Windows users.

It is unacceptable to rely on your file system's case insensitivity in your code. It is a huge pain to rename all media files in someone else's project and then grep through their code to find the filenames in the effort to get the program to not crash.

Make sure you use the same case in your filenames as you do in your code.

Everyone's screen situation is different.

The simplest thing to do is use a fixed sized windowed application - that way you don't have to take into account different screen resolutions and ratios.

Some people consider it rude for games to go fullscreen without explicit instruction.

Do not make a tarbomb.

When you create a tar file, everything should be inside of a single folder with your game name and version number.

Test your final submission.

Once you upload your final submission to the website, immediately download it again into a new folder and try to play it. This ensures that you didn't make any stupid mistakes.

Test on Windows, Mac, and Linux. Get a friend to test it for you if you don't have direct access to any of these OS's. Linux is free, so there is no excuse for not testing it on Ubuntu.

You would be amazed at how many people forget to include their media folder, or make some trivial mistake that renders their entire game unplayable.

Conclusion

I wrote this article selfishly. I feel compelled to play every contestant's game, and I want the games that I play to be fun. Hopefully this article is helpful enough to raise the standard of PyWeek.

198 miles - July 24, 2011 through August 4, 2011

Journal Entries

1. Day 1 - First Day of Hiking
2. Day 2 - Donahue Pass to Some Lake
3. Day 3 - Some Lake to Deer Creek Crossing
4. Day 4 - Halfway to Duck Lake to Mono Creek
5. Day 5 - Mono Creek to Senger Creek
6. Day 6 - Senger Creek to Something Meadow
7. Day 7 - Muir Pass
8. Day 8 - Mather Pass, Almost
9. Day 9 - The "Matherhorn" Pass and Pinchot Pass
10. Day 10 - Glen Pass - 15 miles
11. Day 11 - Forrester Pass - 13 miles
12. Day 12 - Mt. Whitney - 22 miles

Day 1 - First Day of Hiking

July 24, 2011

I woke up to a cool and crisp morning. The tent Dad and I brought is extremely fast to construct and tear down, lightweight, and just the right size for the two of us. Dad crowded me a bit but it didn't bother me at all. He's on old man; he needs his space.

Mike, Paul, Dad, and I took our time packing up camp. We had 5 hours to kill waiting for Bruce and Chris to arrive. I tried organizing my pack a different way, which made it easier to carry, but made my food less accessible.

When we left camp, Dad grumbled something about not paying the $5 / person fee for staying in the campgrounds. I told him I didn't understand how he thought that was morally different than cracking your neighbor's DirecTV service, a reference to a discussion we had had earlier. He said that he thought I was right, so we walked back and paid the $20 for the 4 of us.

We drove the rental car just down the road to a convenience store, where I purchased a sewing kit for my sleeping bag and sunglasses for the snowy passes that lied ahead. Just then, Bruce and Chris walked in. It was fun to see them, not expecting them for another four and a half hours.

Dad and I called the rental place and found out that the nearest drop off was not 6 miles away as we thought, but rather 50 minutes away. Dad and Mike drove the rental car and Bruce's PT Cruiser down and came back in the PT Cruiser 2 hours later.

Soon we were at the trail head taking the obligatory photo, and then on our way. The trail ran alongside a beautiful creek/river with dozens of shallow crossings. My new boots worked like champions, water and mud alike sliding off the top of the boots, leaving them looking spotless. Pretty cool.

After a short food and water break, intense uphill, and a short but painful case of the runs, I arrived at the beautiful snowy valley where we are camped. After hot dinner thanks to Bruce's stove, Paul, Mike, Dad, and I played 3 rounds of hearts. Now I'm about to undress and sleep until first light.

These guys are such great company. Courteous and caring, yet comfortable with crude humor. I'm looking forward to the next 13 days.

Day 2 - Donahue Pass to Some Lake

July 25, 2011

I tossed and turned and dreamt all night. I dreamt that I wasn't being faithful to my girlfriend, and that I turned gay for a bit, accidentally.

I woke to Dad gently shaking my leg. Looking around I saw that everybody was just beginning to get up. I ended up being one of the first ones packed, so I scouted the river we would soon be crossing. Soon, we were on our way. With my belly full of oatmeal, and my legs fresh from sleep, I felt good.

Donahue Pass was somewhat tricky as far as staying on the trail. Although it was a beautiful sunny morning, there was still a significant amount of snow remaining.

The hike went on for a long, long time. I am happiest when debating or talking with someone. Bruce is very intelligent but his wordiness makes it difficult to really build an argument against him. On the other hand, wordiness is great in the context of killing time. Mike is straight out obstinate. He told me that I couldn't prove that science could handle morality in a logical manner. When I asked him to define "morality," he would not, instead telling me to define it, and then refusing to accept my definition. Every time I finally was beginning to make some progress in the discussion, he would change the subject. He would answer direct questions not with a definite answer, but by a lengthy allegory that was barely relevant.

The views that we witness are breathtaking. At least I'm sure they would be if I actually appreciated beauty. Still, I get pleasure from pointing out views and wildlife and watching my companions' reactions.

Toward the end of the hike we met 2 guys who knew the Ridenours through Katelin. How weird is that?

I enjoy the implicit camaraderie that nearly every hiker feels. Warm welcomes, trading of information, and sometimes even sharing food is common with the hikers we meet.

Dad asked about Sara today. She's been on my mind off and on this whole trip. It's weird to have a successful first date with someone I seem to "click" with, and then immediately leave town for 2 weeks. Oh well. Hopefully she'll still be interested in getting together when I get back.

The last 4 miles of the trail hurt. I forced achy tired muscles uphill, downhill, uphill, downhill, and then more up. Everybody agreed the uphill is killer. Nothing is worse than switchbacks.

At this point we were near a lake, with moderate mosquitoes and ants. It was our target destination, and a decent camp site. I voted to stop, but the guys wanted to go another mile uphill before stopping. I bit the bullet and joined the march. After finally arriving at the new destination, we quickly realized that this marshy, mosquito-infested, snowy area would make for a miserable night. We back tracked 1/3 of the way back to our original destination, to a camp site I remembered seeing. It is a good spot, but with another nearby lake, it is thick with mosquitoes. Not as many as the marsh though.

Dad tried out the instant mashed potatoes, mixing boiling water and bacon bits. It was heavenly. We ate that along with soup - teriyaki flavoring + Ramen + raw chicken (mixed with boiling water). That was good too but it didn't compare to the mashed potatoes + bacon bits.

The sun is about to go down. I'm going to force myself up off my sleeping pad and join back with the guys sitting on some rock. I am really looking forward to sleeping tonight. Hopefully I don't dream that I'm gay again, that was lame. We'll see what kind of wacky story my brain puts together.

Day 3 - Some Lake to Deer Creek Crossing

July 26, 2011

I woke up often during the night to air out my hot sleeping bag. The air was cool, but my bag is just so warm. Regardless, I got decent sleep.

After a quick breakfast, deconstructing the tent, and brushing my teeth, I was packed and about to head off down the trail. I get a kick out of being more responsible than Dad at least as far as taking care of my teeth. Paul politely requested that I wait a bit since it was going to be a while before everyone else was done packing, and it would be better if our group hiked together. So I went ahead to fill up water from snowmelt and waited. Soon enough I was on the trail with Dad, Mike, and Paul. The word was that Bruce and Chris were still packing but would catch up soon.

Despite a good night's sleep, I didn't feel great. I was sore and my feet already hurt. I mentally braced myself for an entire day of enduring pain. At one point Paul suggested that I hike my pack up on my back and tighten the waist strap. This helped tremendously.

Bruce and Chris were nowhere to be seen when it came time to go a bit off the trail to Red's Meadow Resort to pick up our resupply package. We hiked a half mile uphill to get there. The package was huge. Along with four 20-something hikers next to us, we had to toss some food out. I traded bacon bits for dried fruit with those hikers, telling them how great it tasted with instant mashed potatoes. We also saw the two girls that passed us the day before. Their names were Alley and Jule. Alley is a microbiologist. They plan to be out on the trail for an entire month!

After 50 minutes of lunch and resupply, we hiked back down the hill hoping to meet up with Bruce and Chris. Once there, we realized that the trail was actually back uphill, so we had to backtrack half a mile uphill.

Along the Pacific Crest Trail, we saw a bunch of construction workers working on the trail. When we approached, they would shout, "HIKERS!" and get out of the way. At one point I thanked a guy for working on the trail and he said, "Don't thank me, I have to do this to pay off my fines."

After a trek uphill, another meeting with Alley and Jule, and a lot of walking, I was thinking to myself, "My feet hurt, my back hurts, and I'm tired, but this is still a new day, so gear up." But much to my pleasant surprise, it was 3:40pm, late in the day, and only 20 minutes to our destination. At this point I started to mentally relax. Bruce and Chris were waiting for us at Deer Creek Crossing. That was good news. The bad news was they had been waiting for 45 minutes and wanted to move on. I switched from boots to sandals and again mentally braced myself for 6 more miles uphill.

I explained the Mineflayer AI I am working on to Bruce, who works on algorithms for a living, and he said that there were a few algorithms that fit the bill. I'm going to email him and ask what they are. I also told Paul I would show him Demetri Martin.

At 5:23pm, we found a decent spot and settled. A creek flows nearby and the mosquitoes are just as bad as last night. I think I killed at least 100 today.

While we were eating dinner and having a jolly old time debating various topics, a 20-something year old hiker named Kevin came by and asked if he could camp nearby. We were happy to have him come up and join the discussion. He is an American History major who "finally" got a minimum wage Starbucks job. I'm so lucky to have a profitable and enjoyable profession.

Everyone else is going to sleep. I don't want to feel like crap when I wake up so I'm going to put down the pen and zonk out.

Day 4 - Halfway to Duck Lake to Mono Creek

July 27, 2011

It was slightly cooler last night, so I woke up less times to air out my bag.

We did our morning routine and hit the trail. Kevin hadn't woken up yet. After a small crossing where Dad and I hopped across rocks and the rest of the crew trudged through, we had a small pass ahead of us. I was feeling good. So good, in fact, that I hummed the Muse cover of the song Feeling Good. At least it was something different than Micah by Five Iron Frenzy which has been stuck in my head this entire trip.

Anyway, I soon left the rest of the pack behind with my speed. After 400 feet of climbing and walking through a small pass, I entered a beautiful hidden meadow with a pristine lake in the middle. I took a break, taking off my boots to let my feet taste the grass. It was hot and noonish, so I stripped to my underwear and jumped in. By this time, the group caught up. Everybody loved the spot, so we all had lunch, enjoyed the meadow, and got eaten by mosquitoes.

At this point our 6-person troupe was on the trail together. I started discussing philosophy with Chris and Bruce. We talked about quality of humans, what gives human life value, and various contrived examples of abortion. Bruce did not agree with my criticisms of grad school.

We had been walking and talking for a long time. Having a discussion is a great way to walk miles upon miles without realizing it. Eventually Bruce wanted a break, but Chris and I pressed on. We began to go up Silver Lake Pass. I told him the story about sleeping under my desk. I could tell he enjoyed the story, especially the part about my fortress in the Capstone room.

Storytelling is something that happens often and naturally on this trip. As I write, the guys are sitting around a fire and Dad is telling Chris about a biking injury.

Once Chris and I got almost to the top of Silver Lake Pass, we lost the trail. 4 other hikers joined us, in the same boat. I decided to wait for everyone to catch up, just to make sure we all went the same direction. Bruce came along first, scouted for the trail, and yelled back when he found it. Chris took off while I waited for Dad and Paul. Once they arrived we trudged up the snow. Looking down we saw a frozen green/blue lake. As we trudged on, we couldn't tell where the trail was due to snow. Good riddance! Instead of tedious switchbacks, we climbed up a near vertical wall of snow, getting to the top of the peak fairly quickly. Here was a stunning view, a quick snack, and relief from mosquitoes. Paul read from his JMT guidebook about what the next 6 miles held for us.

We decided to try to get ahead on the itinerary by going another 6 miles downhill. We took off, excited to get ahead of schedule. After a few miles my feet started to hurt so I swapped my boots with sandals. This felt much better. I fell into step with Dad as we hiked down. At one point we saw a really cool waterfall. As we walked further, we learned that we would cross just under that waterfall, which was right before another. What fun! I almost fell in.

After just a few more switchbacks, we arrived at another river crossing. This one was violent. Paul fell several times, skinning both knees, but somehow saving his camera. Dad fell a little bit and his pack briefly ducked under water. So far everyone had borrowed Bruce's poles to cross the stream. I, "Feeling Good," tried to cross without them. A few steps in it was clear to me that if I took another step I would topple over and be carried downstream. I motioned to Dad, who threw one of Bruce's hiking poles to me. I deftly caught it in the air and used it to maneuver across. Good thing I didn't miss!

At this point, it was obvious that Dad was not feeling good. He had some bad blisters that were aggravated by the crossings. It was well past 5pm, our "usual" "look for a campsite" time, but the rest of the guys wanted to press on. We walked for another few miles after that, and finally, after yet another crossing, Chris, Bruce, and I made the executive decision to call it quits for the day and set up camp.

This is a good spot; as I mentioned, we have a fire going, and there are some dry, flat spots for tents. Dad and I had our favorite mashed potatoes + bacon bits. Dad feels much better after warming up and relaxing a bit. His bag got wet but only a little. He put it out to dry and it looks pretty dry now. The fire seems to ward off the mosquitoes.

Everyone is tired and going to sleep now. Dad is pumping up his mat. I think we walked 20 miles today. A lot of my food is gone - we'll see if I make it to the next resupply.

Day 5 - Mono Creek to Senger Creek

July 28, 2011

I hate mosquitoes.

I woke several times throughout the night, thankful, each time, that it was still dark and I didn't have to get up. I noticed that every time, the pain in my feet went down. Just when one more snooze would have completely obliterated my feet pain, it was time to get up.

Bruce started the fire again and I put a few sticks on to make it hot enough to burn some trash. As I burned the blue nylon t-shirt I had been wearing the entire trip, empty plastic bags, and moldy summer sausage, I smiled at the weight I would no longer be carrying.

After lazily finishing my morning routine, Dad, Mike, and Paul had already hit the trail. I wanted company so I waited until Bruce was done packing. I took the lead on the trail, however, and Bruce didn't keep up with my brisk pace, so I quickly got ahead.

This part of the trail was about a 1000 foot climb. Switchbacks became monotonous very quickly, but much to my delight, I had The Receiving End of It All by Streetlight Manifesto in my head instead of Micah. It wasn't to last though; by noon I caught myself mentally singing those silly lyrics with each step.

By 11am I became ravenously hungry. Dad gave me several large handfuls of sunflower seeds he had handy, but my stomach demanded more. We agreed to stop for a lunch break after crossing Bear Creek several miles ahead. I hiked with a fury so that I might get there sooner and abate my hunger. Along the way, I noticed that my feet were starting to feel quite bad even though it was still early in the day.

I ate pepper jack cheese, pine nuts, sunflower seeds, pecans, tomato & basil crackers, and pepperoni. Finally my stomach was content. The mosquitoes, on the other hand, were not. They continued feasting on us all day. There was naught but one or two moments of relief.

After lunch, I switched to sandals and we began another ascent up to Seldon Pass. I fell into step with Chris; we talked about music conventions like Magfest and percussion events, our similarly aged younger brothers and their various life issues, and the pros and cons of raising kids in a small town versus a big city.

This was quite enjoyable. Before we knew it we were looking at Marie lake - icy blue water surrounded by a meadow at the top of Seldon Pass. It seems amazing to me that there can be a lake at the peak of a mountain.

I sat down for a bit to wait for Dad and Paul, swatting at my legs, arms, hands, neck, and face. It was clear that Dad was having a tough time today. He has huge blisters from the balls of each foot to their respective toes.

Nevertheless, he trudged on with the rest of us. This part was several miles downhill - much easier on the body. My experiment with sandals was a huge success. My feet continued to feel much better ever since switching footwear at lunchtime.

I saw a marmot scitter across the trail. It was fun to see, but I still think the woodpecker that Paul and I heard the morning before was more interesting.

After a few miles I began to feel the familiar end-of-the-day fatigue. It is kind of an inexplicable pain that makes me want to stop and set up camp. If I try to pinpoint it, I find that yes, my feet hurt and my back is a bit sore, but that itself is not the fatigue. In the morning I can have the same pain but continue to hike 16 more miles.

The mosquitoes were relentless. Even while walking they attacked with a fury. At one point I stopped to wait for Dad, donated about a pint of blood, and didn't even get $20.

Finally we crossed Senger Creek and began to look for a campsite. We settled on one with a modest walk to water, mostly flat tent spots, and a fire pit. Dad and I shared our favorite potatoes + bacon bits and some rice. I have one more dinner and half a lunch left of food, but it should be fine since we're resupplying at John Muir Ranch first thing tomorrow.

I built a fire to drive away mosquitoes but it's hardly effective. They bite through cloth, nylon, and netting. When you kill one, the others are unfazed. I pulled my pants down for 5 seconds to drop a deuce, and now I have 3 itchy bites on my rump. I'm cowering in Dad's and my tent, with about 30 of them patiently perched on the netting, waiting for me to come out or accidentally press my back against one of the tent sides.

It might rain tonight. I felt a few drops and heard thunder in the distance.

The guys are sitting around in the "kitchen" talking. I'm deciding between joining them or going to bed immediately. The sun is still out but I'm sore, itchy, and lying on this pad feels oh so good.

Day 6 - Senger Creek to Something Meadow

July 29, 2011

According to Dad, a hiker named "Stick Man" camped neighboring ours, and he had been hiking the Pacific Crest Trail all the way from Mexico, starting May 1st. He was headed for Canada, but had to stop somewhere along the way to make more money for his expedition.

I fell asleep shortly after yesterday's entry. My aching body was happy with this turn of events, fixing itself up completely while I slept. Once I finally stepped onto the trail, I felt fresh, strong, and ready for the day.

It was only a few short miles downhill to John Muir Ranch, our final resupply. The food we packed there had to last us 7 days. Along the way, Paul and I saw a peculiar native bird. It looked and acted like a combination of a quail and a chicken. I heard its baby chicks softly chirping in a nearby bush.

At the ranch, our resupply bucket was a welcome sight. In the words of Paul, "This is better than Christmas!" Bruce used the hiker buckets with leftover food to resupply for himself and Chris, who woke up late and was missing. I scored a rice bar and a peanut-butter-and-honey tortilla for an immediate snack, and a bag of instant potatoes for dinner later.

Dad bought some Deet, and we all tried our cell phones for reception, but of course nobody had signal. There was a handy spring water facet for filling up water bottles, and a scale for weighing packs. Everyone's packs were between 34-37 lbs, my pack setting the high bound. We spent about 90 minutes at the ranch, organizing our food, eating, and discussing what to do about Chris. Just when we were about to take off, leaving Bruce behind to wait, Chris waltzed in. He had taken a wrong turn and had to double back.

"Chris!" Bruce shouted from around the corner.

"Yeah Dad?" Chris replied.

"Welcome!" Bruce exclaimed, with just a hint of patronization. Mike and I chuckled.

We left the ranch, virtually staggering under the weight of our new food. We took a breather at a junction hardly even a mile away, next to a roaring creek with a wooden bridge. Dad gave me his camera to borrow for a while, a suggestion I made a few days ago.

After we got up and crossed the bridge, the trail gradually took us for a tour along the lower left side of a thriving valley, with waterfalls to our right above, and a violent river to our right below. I fell into step right behind Dad for a while, until he suddenly stopped and I bumped into him.

Looking ahead, not 20 feet away, I saw a doe and her 2 fawns. I snapped a couple photos with Dad's camera and waited till Paul was done doing the same before moving on.

Eventually there was a convenient spot for water near the trail and I went to get some. The others pressed on, knowing that I would catch up soon. I sat on a mostly flat rock, sucking the river water through my bottle's filter, enjoying the break. I lingered, somewhat guiltily, watching the guys get further and further ahead. Eventually the mosquitoes started biting, so I gathered my pack and resumed the trail.

At this point, I sauntered leisurely along in solitude, knowing that Bruce and Chris were still far behind. I started thinking about the length of this trip, and my new home in Seattle, my new friends, Sara, and starting work at Amazon in less than 2 weeks. For the first time on this trip, I felt homesick.

The feeling passed however, as I set my mind to new tasks, for example, where to put my pack while taking a dump. Everywhere I looked there were anthills riddling the ground. Eventually I resigned myself to shaking my pack after doing my business, knowing that I was surely far behind the others. I wasn't even sure if I was following the right trail. Soon enough, though, I caught them on the uphill. It saddens me that one day, I, too, will be old and slow.

After a short hike into a denser area, I shouted a greeting to 4 hikers lunching on a nice sunny rock as I passed them by. I carefully waded across the rushing but relatively shallow creek, walked a bit, and then found my own sunny rock to rest on while munching some snacks and waiting for the rest of the group.

The sun beat down angrily. I wondered if I would burn, having forgotten sunscreen that day. Soon, Dad, Mike and Paul showed up, and eventually, so did Bruce and Chris. We discussed how far we planned to go today, but were interrupted when a single, large ball of hail landed on my pack. There was a brief moment of hesitation, and then everyone burst from their relaxing position into action mode, getting out raincoats and tarps to protect packs from taking on water.

It sprinkled for less than a minute. I was sourly disappointed. But 10 minutes later, the sky opened up in a deluge of rain and hail. It was amazing how fast the trail turned into a stream of icy water. The storm kept up for at least an hour. Instead of accidentally getting pebbles stuck in my sandals, I was getting hail. Amazingly, my feet kept warm.

Finally the group gathered. It was still raining, and there were no dry camp sites for 8 miles. As it was already 5 o'clock, we opted to try to find a decent place on the hill. We found 3 soggy but not swampy places and quickly pitched tents.

Mosquito myth: Mosquitoes don't come out while it's raining. False, false, false, false, false! Oh, here's another bite. False!

Most of Dad's and my stuff is wet. I went out while Bruce was heating water and made dinner for Dad and myself, bringing it to him in the tent.

I'm getting cold. I'm going to try to get warmed up and go to sleep. We have a hard day tomorrow and have to get up early.

Day 7 - Muir Pass

July 30, 2011

Although Dad and I were wet all night long, we were warm. By morning, the inside of my sleeping bag was dry; the moisture had steamed out while my body heated it up.

The others packed up quickly, wanting to hit the trail as soon as possible. I was the slow poke. While Mike, Bruce, and Chris took off as soon as they packed, Paul and Dad waited 5 minutes for me.

Today I prepared to hike a long way without stopping. I carried no water, to save weight, and instead kept my bottle in my shorts pocket to drink when I crossed a stream. In my other pocket, I carried deshelled sunflower seeds.

As we climbed, we came to an empty trash bag sitting in the trail. What luck! I sorely missed this last night - it would have kept my bag dry. Dad fell behind a bit. He was about .25 miles back when Paul and I caught up to the rest of the group. Mike and Paul, "planners," as Bruce calls them, were concerned about our pace. There was talk about ending the entire trip. Eventually, we decided to break into groups of two - the people who shared tents - and meet at the end of the day.

With that, Mike and Paul were off. They moved fast, concerned about the thunderhead clouds ominously forming above the peak behind us. I took the opportunity to rearrange my pack, getting out pepper jerky and trail mix.

When Dad arrived I informed him of the plan and we were off, leaving behind Bruce and Chris, who were still messing with their packs. I still hiked in wool socks and sandals, even though the terrain was snowy. We trudged endlessly through the snow fields, wondering when we would finally be over the pass. The snow had settled in a most unfortunate ruffle pattern, making missteps and sliding backwards common and exhausting.

The looming clouds which had been threatening us all day finally started to deliver. After a few drops of rain, dad and I put our packs down to dig out our rain coats, and I switched from sandals to boots. What a difference! I seemed to power right through the snow with those bad boys.

Yet the rain and the slope increased. Icy wind bit at my ears. I led; Dad trudging along slowly behind me. We passed several hikers going the other way, one of them pointing out a triangular shaped rock hut marking the top of the pass.

Just when the rain was getting violent, we made it to the door of the hut at the top of the pass. I tried the handle. The door slowly spun open. I couldn't see a thing, but it was warm, and I was greeted with a chorus of "hello"s. After my eyes adjusted, I saw about 8 hikers and a lot of gear seated on and around the bench that lined the wall.

Dad ate a snack and we waited out the rain. The hikers were friendly and talkative. The gentleman to my left, who I later found out was named Jaffey, was interested in a party of two to my right; a father heating water for "hot chocolate mixed with cider" and his 17-year-old daughter.

As soon as the rain let up I was out the door and down the other side of the pass. This side was steep. It was possible, however, to sort of run-ski down, making for quick and effortless travel. After a ways I looked back and was happy to see Dad already on the trail. As I watched, he slipped and fell, pausing a bit before getting up.

Eventually, he caught up and we hiked along the snowy mountainside together. At one point, the trail led straight downhill, and there was a perfect butt-shaped halfpipe in the snow alongside it. Dad did not hesitate. He plunked down, pushed off, and then gracefully and effortlessly sped down the causeway. I, of course, followed, getting snow in my pants and dropping my bottle toward the end.

We crossed a few streams and hiked down another mile or two. The terrain went from snowy to lush and green. We took a break to swap footwear and so that I could drop a dookie. Just when I was doing my business, it started to rain. We had no idea how far ahead Mike and Paul were, nor how far behind Bruce and Chris were. We decided to pitch the tent and wait out the rain.

Minutes after we finished the tent, Bruce and Chris came by, decked out in rain gear. They considered waiting it out with us, but ultimately wanted to keep moving.

So we waited. The rain pounded relentlessly. It was nice knowing my entire pack was in the tent and out of the rain. Dad and I snoozed for several hours. The rain just wasn't letting up. Finally, I woke to the sound of Dad putting shoes on. The rain had stopped, and it was sunset. Dad and I decided to quickly pack up and hoof it down the trail in hopes of catching the others. In two minutes we were packed and making headway.

Walking on fresh legs at sunset after a rainstorm is beautiful. The colors all change, the smell is different, and there is a spookiness to it. Dad pointed out a large rock to the right that looked exactly like a whale's face, and somebody had lined up rocks on the bottom layer to make it look like teeth.

Not even an hour after we left, we spotted a campfire and Paul. We were very glad to see each other, thinking that we had been split up. At this campsite, 12 hikers in total have gathered, including Jaffey and his friend. It was fun to sit around the fire and swap stories. Apparently everybody knows Kevin, the hiker who stayed a night with us. Bruce heated up some water for Dad and me - we ate stuffing and broccoli flavored rice.

Paul and Mike are still concerned about our pace; they want to get up very early and try to get over the next pass. Paul says it would ensure our success on this trip. I say let the planners plan. I'll follow along.

I'm running low on lunch food.

Day 8 - Mather Pass, Almost

July 31, 2011

Dad started getting ready next to me while it was still completely dark. I saw headlamps on in Paul and Mike's tent, and Bruce and Chris's tent. Thinking it was close to 3am, I was slow waking up. Finally I willed myself to begin putting on warm clothes and then tried to remember what to do next.

Eventually, I had my pack together, ate oatmeal, and brushed my teeth. Turns out it was actually 5am. Examining my bear cannister, I became increasingly alarmed at my lack of food. Nevertheless, I needed energy for the long hike ahead, so I got out lunch - a bag of rice and a bag of pecans - and stuffed it into my gym shorts pocket to eat on the trail later.

I couldn't find my nice long wool socks, dirty as they were, so I wore one of the two remaining pairs of cotton socks with my sandals.

The talk back at the camp was that we had to make it over Mather Pass today to ensure the trip's completion. If Dad and I hadn't made it to camp last night we would have had to cut off the trip early. On top of that, we learned from a sign by a ranger station that there was a rock slide blocking Whitney Portal. Dad thinks that "they" will clean it up by the time we get there. Mike doesn't, but thinks we can walk to town if we have to.

I was determined to not be a part of the problem. I kept up a brisk pace, uphill, flat, and downhill. There were several creek crossings with no bridge across. While others tried to keep their shoes dry, I stomped right on through in my sandals. Soon I had a large lead on the others, except Mike, who was close behind. The trail became overgrown with dense lush plants and grass, and I jumped when something with a furry tail scampered under my legs. I saw Kelley Flowers that Paul had pointed out to me - yellow, upside down pedals with brown spots. Also I was surrounded by Jeffrey Pines. Dad must have been having a heyday.

The trail began going steeply uphill, and I ate the bag of not-very-good rice, worrying about running out of food in a few days.

Fun mosquito fact: Mosquitoes can fly faster than you can walk.

Mike and I kept climbing, watching the foggy sky ahead forming ill-boding clouds. We did not like the look of that at all. Sure enough, after another mile it started sprinkling. I heeded the warning and stopped to put on a sweater, rain coat, and switched to boots.

The rain started to pour.

I found Mike just ahead, huddling under a patch of awkwardly shaped pine trees. He looked miserable. I waited there with him for about 30 minutes until Paul showed up. There was no evidence of our failed illegal fire attempt. Both my lighters were out of fluid, and the sticks we had tried to burn were soggy.

The three of us continued walking in the rain until we realized that there were no more camp sites ahead, and the relatively flat spot we were at presently was where we needed to wait for Dad, Bruce, and Chris. We used Mike's fly and sat on my ground cloth to shelter ourselves and our packs from the rain.

Finally, the rest of the guys showed up. Dad came under the tarp for a bit, but it was uncomfortable; when the rain let up just a little, we scrambled out and pitched our tent as fast as we could.

At this point I informed Dad of my food shortage issue. He looked concerned and offered me some crackers which I gladly accepted. Not that it would be enough. I decided to skip dinner that night.

Inside our tent, Dad and I examined our belongings. Surprisingly, our stuff got only a little wet. That garbage bag I picked up yesterday had worked wonders helping my clothes and sleeping bag stay dry.

With not much else to do, we snoozed for several hours. Finally it stopped raining, and Mike yelled at us to come out and "be social."

We played several rounds of Hearts while I ate the tuna that I turned down at the ranch but Dad had packed anyway. Bruce served a dish of Fritos and hummus. So delicious. Paul gave me a bag of cereal and pumpkin seeds. I feel pretty guilty about not packing enough food at the John Muir Ranch and mooching off of everybody. Not guilty enough to turn down their self-sacrificing offers and go hungry, though.

Mosquito myth: mosquitoes can't survive at high altitudes. False!

There has been much speculation about where, how far, and how long the pass is ahead. I think we were probably over-estimating its difficulty. That being said, we have to do 2 passes tomorrow or cut the trip short. I think it all depends on the weather.

Day 9 - The "Matherhorn" Pass and Pinchot Pass

August 1, 2011

High up in the thin air, the temperature dropped below freezing during the night. Dew formed on the clothes I laid out to dry and then froze. Inside my dry down-feather sleeping bag, I slept, warm and cozy.

That is, until 5am when I felt Dad waking up and putting on his warm clothes to start the morning routine. I could clearly hear the bridge of "Mermaid" by Anamanaguchi, playing in my head. Our performance today determined whether or not we would have to end the trip early. If we could get over both Mather Pass and Pinchot Pass, we would ensure success. The guys got ready especially fast; I was still shivering and crunching on frozen dry oatmeal when Paul and Mike took off. Not wanting to fall too far behind, I skipped brushing my teeth.

Not that falling behind was anything to worry about. The conditions of the pass were perfect - the snow had solidified and become very hard overnight. With food in my belly and plenty of sleep, I practically pranced up the steep treacherous pass. For some strange reason, I had the Jeopardy theme song with "I'm a little teapot" lyrics stuck in my head.

Dad and I made it up to the top of the pass first and had a celebratory piss off the edge. While waiting for the others I rearranged my pack a bit, changing into lighter clothes now that the glorious sun was shining on me. When I finished, Dad, Paul, and Mike had started down and Chris was still waiting for Bruce. Luckily I noticed Dad's camera he had forgotten. Yelling at him to turn around, I snapped a picture of his face when he realized he'd forgotten it.

I started down the steep trail, planning to eventually catch up with Mike and Paul in front. With the camera, though, I found myself looking for picture opportunities. I kept up a brisk pace, despite that, and soon enough gave Dad his camera back. I switched from boots to sandals and focused on catching up again.

I had this sharp pain in my left foot, in the tendons when I lifted my toes. I think it may be a lipoma causing problems. I want to have a doctor look at it.

Despite this weird new pain, I hurried and soon passed Dad again while he switched shoes to cross a creek. With my sandals I saved time by simply splashing through without changing footwear, but not without first dipping my bottle and having a drink. At one point Mike found a dead deer just upstream in a creek while looking for a dry crossing. I hope I didn't drink from that one.

I stopped putting on sunscreen two days ago, and I have not been burned. My only conclusion is that I have reached Level 10 Hiker and am now immune to the sun. I can't wait until I gain intrinsic mosquito repellence.

I had to run to keep up with Mike and his stupid long legs. After a while we came upon Paul filming a nearby doe. The three of us walked along until Mike stopped to change shoes for a crossing, while Paul and I waded carefully across without changing footwear.

Paul and I were pleased at the good weather, and the good time we were making. At some point Paul stopped to change into lighter clothes and I went on alone. I walked for some time, getting hungry and thinking about my rations. Finally I stopped for lunch: one measly packet of crackers, and a medium stack of pepperoni. Delicious. I snuck another small stack of pepperoni. I wanted more, but I knew I had to ration.

After that I kept on. I tend to go slowly when alone and in the lead. Sure enough, Paul and Mike caught up after a mile or two, and I followed them, switching into my boots for the upcoming Pinchot Pass. There was a bit of uphill climbing, but then we were there, along with two hikers who went by Major Upchuck and Bounce Box. It was a couple who had met on the trail a month ago. They were friendly, and they know Stick Man. At one point Upchuck asked me if I had a bowl, which seemed a queer question, until I realized it was another euphemism for Marijuana.

After a short break at the top of Pinchot Pass, Paul, Mike, and I headed down. We walked for a long way. We could see some clouds boiling in the mountains ahead, threatening more nasty weather.

Sure enough, after another long section of hiking, the rain started to come down. We stopped to get out our rain gear and then pressed on. By this time, I was starting to get ravenously hungry. It was hard to take my mind off food, and I began to feel somewhat weak. Still, we kept walking and walking, until finally I had to sit down and have a snack. Paul and Mike walked on. Before I even had my cannister unpacked, Paul shouted my name. I put my pack back on and followed. A suspension bridge led from the trail over a roaring river to what Major Upchuck had called a "tit perfect" campsite. Hardly any mosquitoes, plenty of room, more than one fire pit, and a toilet.

By this time the rain had calmed down, but it was about to pick up again, so Mike and Paul pitched their tent. I had to wait for Dad, who had half the tent, so I used the time to relax and take inventory of my food. 1 breakfast short, 1.5 lunches short, and 2 dinners extra. I ate a bag of not-very-good rice that Dad had given me and that I had just found was not sealed properly.

I worried that Dad was too far behind and would not make it today. Just when I went to nervously check the bridge, he was there, waving at me. I was amazed. He must have been in terrible pain. I grabbed the tent poles from him as soon as possible and began pitching the tent. Not a moment too soon. Once the fly was up it started raining hard and we took shelter within. It even hailed a bit. Not too long though, and the rain stopped and the sun came out, so I ventured out.

There were about 10 tents in this site. Some folks were playing Frisbee. Bruce and Chris's tent was not one of them, however. They were nowhere to be seen. Dad came out of the tent at Paul's coaxing, wanting to take his picture in front of a giant Jeffrey Pine in the sunset. After that we played Hearts, Dad sharing salmon with me.

After round 3, Bruce and Chris finally showed up - 2 hours after Dad. They had gotten lost and followed a wrong trail. Instead of the 20 mile day we had had, theirs was 25.

Bruce, being the generous man he is, immediately started heating water. I split with Dad the chicken noodle soup I got from the hiker's box back at John Muir Ranch, which was filling and delicious.

It was getting dark, and finally a family started the fire. They were friendly and happy to have me burn my trash. I sat with them and talked. They are taking 7 days to hike 40 miles, a much more leisurely and fun trip. We had fun swapping short stories about our home town climates, and a teenager showed me a video of him jumping off the roof of his house into a snow bank on his iPhone. We all headed for bed when the fire started to die. Gotta wake at 5am again tomorrow.

Day 10 - Glen Pass - 15 miles

August 2, 2011

I slept deeply, without waking or stirring the entire night. I dreamt that I was the protagonist in a Rube Goldberg Machine, and then that I was doing covert ops with David Hayden on the Death Star.

"Wake up," Mike intoned, from outside the tent Dad and I were sleeping in. I did better at waking up this time, almost immediately sitting up to put my warm clothes on. I didn't really understand why we were getting up at 5am again - we were no longer in a hurry - but I felt okay, so I saw no reason to fuss.

By the time I had eaten breakfast, brushed my teeth, and found the rumored toilet on the hill, the guys had all left, leaving me to play catch-up. Not that I minded. Bruce had left me a full package of freeze-dried strawberries on my bear cannister, claiming they were "extra." Hard to believe, but even harder to turn down.

As I was packing, one of the nice ladies from the campfire the previous night came out of the tent and gave me a handful of almonds, citing "motherly instinct." I gave her warm thanks and ate them all right then, continuing packing.

As I left and bid farewell, she asked, "Is that your jacket hanging up right there?" I mentally kicked myself.

"Why, yes it is," I replied, again thanking her profusely. Oh the misery she has likely saved me.

Hiking in my shorts and skintight t-shirt, it was cold. The valley's mountainous sides were so tall that the sun wouldn't come out for several more hours. Hiking kept my core body warm, but my arms and hands became numb.

Finally, the trail went so far as to escape the giant valley walls, and the glorious sunshine fell upon my face. I soon passed Dad and Bruce, who were moving slowly. Dad had finally used moleskin, but it didn't appear to be helping much.

In a few miles I passed a large party of elderly looking hikers, a few campers just waking up and having breakfast, and a ranger station. I considered going up there to ask for a weather forecast, and any news about the rock slide at Whitney Portal, but I wanted to catch Mike and Paul.

I soon did after hiking around a lake. They were preparing to start the real climb that was Glen Pass. They left just as I arrived, while I stayed a few minutes to swap footwear. The boots do much better in snow.

The pass was very steep, and very cold. I shivered at the unceasing icy wind. It seemed to get stronger the higher up I went. I paused to look down and saw a half-frozen lake with beautiful blue ice. Just as I was about to turn and keep walking up the switchbacks, I heard someone calling out. Spotting Dad and Bruce far below, I waved and hollered back.

Once over the pass, there was a nice downhill dirt slope that was perfect for speed-walking. As I gained on Mike and Paul, I passed about 25 hikers going up the pass the way I came down. They must be doing the 40-mile something-loop trail the campers last night told us about.

I caught up to Paul and Mike resting in a small, sunny, grassy area, where the wind died down. I joined them, sitting on the ground and leaning my back against a rock, which felt wonderful. Paul gave me a handful of M&M's and quite a few deshelled sunflower seeds. M&M is one of my least favorite candies, but as I munched that sugary snack right then, it was just about the best thing I had ever tasted.

Paul and Mike ended their break and took off, but I lingered. The spot I had chosen to sit was surprisingly comfortable, and there were no mosquitoes. I finished off my pepperoni and the strawberries Bruce had left me, with a somewhat guilty conscience but a mostly contented stomach.

I caught Dad but then changed back to sandals. I caught him again but then had to do business again, dumping my load into some poor groundhog's hole. Once I caught Dad a third time, I walked with him until camp. He was really hurting, his giant blisters throbbing with each step. We were very happy when we finally saw Bruce's tent.

The two guys we had crossed paths with countless times today, Yuri and David, were making supper at our camp. They cooked a delicious-smelling hot dish that made my grumbling stomach jealous. Dad showed everybody his battered feet, and Yuri gave him Neosporin, blue blister fixer pad things, and tape to fix up his feet. Here's hoping it works.

Dad and I had potatoes and rice for dinner, but then Bruce made a delectable vegetable stew, complete with onions he picked along the trail, and shared it with everybody. Then he made juevos rancheros and also shared that. If there is anybody who lives up to the "love your neighbor" principle, it is Bruce. Did I mention the juevos rancheros was to die for?

The end is so close, everybody can taste it. We're already fantasizing about the hot restaurant food and beer we're going to get when we're done, and we have the mileage of the next few days planned out. We have 36 total miles left. Our last day, Mt. Whitney, is going to be 14 and it's not flexible. So that leaves 22 to split between tomorrow and the next day. Currently the plan is 14 tomorrow and 8 the next. That way we can rest up for the many-thousand-foot climb that is Mt. Whitney. Also we plan to wake again at 5am, for some reason unknown to mankind.

Day 11 - Forrester Pass - 13 miles

August 3, 2011

The last pass.

I struggled to find motivation to get out of bed. It was cold and dark, and we didn't have to go very far to stay on schedule - not to mention that we had an entire day to kill tomorrow.

I ate my oatmeal breakfast hungrily. Food rationing is a terrible thing. It leaves me with a constant element of stress, and makes me think about food more often than I would if I were not rationing, causing me to feel hungrier. I'm eating plenty, especially with Paul graciously and consistently giving me some of his own precious food. But still, the very knowledge that I must ration gives me constant hunger and uneasiness.

I finished packing just before Dad. He had gone to bed early and slept well. His blistering feet had amazingly healed almost completely overnight. He changed the moleskin and secured it with tape that Yuri had given him last night. We were last to hit the trail.

Walking behind Dad, I could see the ease with which he stepped. Compared to yesterday, he looked pain free. He said as much, too.

We caught up to and passed Paul on the steep, frozen switchbacks leading up to Forrester Pass. We passed a group that had camped at the frozen lake. It must have been a cold night for them.

Eventually, the trail was covered in snow and we followed footprints instead. Those lead laterally across the ice-hard snow mountain side, then directly up the mountain toward the top of the pass. I would not say that I had altitude sickness, but I could tell my body was uneasy with being that high up. I did notice that I became short of breath much more easily.

Our group took a brief respite at the top. We all tried our cell phones but without success. Mike claimed that he got a text-message through, but I doubt it. I don't think the text message protocol includes a confirmation. I'll have to double check that when I get back. [Note: it depends on the network. It is possible to include confirmation, but it is not perfect, and it is not part of the SMS protocol itself.]

Our group continued on down the other side of the pass, but I lingered to eat a snack - some pepper jack cheese and a few almonds and peanuts. Dad shouted up - he had forgotten his camera again, so I grabbed it. Then Yuri and David made it up. I learned that they had run out of food and were trying to make it out today. I gave them the 700-calorie bag of pumpkin seeds Paul had given me that I had been looking forward to eating. This partly eased my guilty conscience about mooching. They thanked me, and then pointed out a black sleeping pad that someone left. Chances were decent that it was either Bruce's, Chris's, or Mike's, so I decided to take it. Then Yuri asked if I might send them pictures of the hike since their camera died. I got their email addresses and snapped a nice photo of them with the mountains in the background. Then someone made it up the other side and brought the good news that the Whitney Portal rock slide was cleaned up and passable again.

By the time I was descending the pass, the others were far ahead of me, and I had forgotten the sleeping pad. I dawdled along, not caring too much about the speed I was going on this short day. At one point I lost the trail to the snow, but saw it way in the distance and simply walked directly toward it.

Finally I got intolerably hungry again and stopped to have instant potatoes, and to change clothes and shoes. Dad whistled for me but I had already unpacked. Turns out he was waiting for me just ahead, simply because he enjoyed walking with me.

We walked together for the rest of the day. We saw Mt. Whitney looming ahead, and even spotted the hut at the summit. This was the first time we actually saw the "finish line" of our trip. This short day - about 13 miles - felt long to me. I can definitely say that at this point, I would much rather be done and home. That is not to say, however, that I don't want to finish the trip. I will thoroughly enjoy the sense of accomplishment.

I think it's funny that we are considering tomorrow - 8 miles uphill - a day of "rest."

When Dad and I finally arrived at the spot where the guys were waiting, we had the biggest interpersonal conflict we have had yet. Everyone except Dad wanted to continue hiking today, so that we could hike out tomorrow and finish the trip a day early. Dad was simply not up to the task, with his feet they way they were. Nobody wanted to be a jerk about it, but there was definitely a sense of disgruntlement. We ended up camping though, and after some time, spirits lifted and everyone warmed up to the plan. Mostly.

Dad and I took food inventory together. He gave me a bunch of lunch and breakfast food, and I will be providing a nice big dinner for tomorrow. Bruce gave us noodle-and-vegetable soup for tonight. We mixed it with our minestrone soup packages and it turned out delicious. I couldn't get enough though - I'm still hungry.

Zeo made it! We thought we wouldn't see him again, but sure enough, he came strolling by at 7pm. Ironically, he's probably going to make it out a day before us. Slow and steady wins the race.

Paul, Mike, Dad, and I played a full game of Hearts. I won, going for a run 3 times in a row and succeeding twice. When I retreated to the tent to hide from mosquitoes, Dad, Bruce, Mike, and Paul engaged in a religion/philosophy discussion.

The mosquitoes here are thick - possibly the thickest of any campsite so far. Yet I'm the only one who wears his mosquito net. I don't understand.

Dad had been calling Chris "sunshine" for several days because he likes to sleep in. Chris doesn't seem to mind - he's a good sport.

Almost every night we go to sleep with the soothing sound of a rushing nearby creek. Dad wanted me to mention that.

Day 12 - Mt. Whitney - 22 miles

August 4, 2011

I slept long into the morning. Dew had formed on Dad's and my sleeping bags and frozen into ice - a testament to our high altitude. Dad got up first, drinking hot coffee for the first time on the trip, thanks to Bruce and his stove. Dad enticed me out of bed with warm noodle soup and some oatmeal.

Our lazy morning quickly turned frantic as the mosquitoes swarmed in. It was not gradual; one moment we were peacefully minding our own business, and the next we were being attacked by 7 swarms. One for each of us, and another for Zeo.

Soon only Dad and I remained, trying to pack without being eaten. Brushing my teeth with a mosquito net on probably looked silly. Eventually we, too, were on our way.

Dad was in a chipper mood, feeling good and knowing he had another day of rest before tackling Mt. Whitney. Or so he thought. After a quick 6 miles, we came upon the guys all waiting around a junction.

"We need to have a discussion," Mike said. I laughed, because I had predicted this. Everyone but Dad really wanted to summit Whitney today, and finish the hike a day early. The alternative would be to finish in just 1 hour, by 9am, and have nothing to do for the rest of the day.

Dad saw that he had to either go along, or be the "stick in the mud" as he called it. As I watched, his facial expression and demeanor quickly flashed from horror of betrayal, to quiet acceptance, to grim determination.

Once it was "settled" that we were going to summit Whitney today and make it out a day early, I opened my bear cannister and scarfed down 2 days' worth of lunch food. Oh sweet calories.

We took off down the trail, knowing a long day was ahead of us. Dad's mood had visibly changed. He knew the last 10 miles would be misery. As if fate decided to add insult to injury, Dad stumbled as he partly tore a muscle on his leg. It was not enough to stop him from walking, but it did give him extra pain with each step. Paul gave him an Excedrin dose which seemed to help after a bit.

By the time we made it to Guitar Lake, which looks more like a bottle of Jagermeister than a guitar if you ask me, Dad's determination and grit had completely overcome his fear of pain and inability to make it today. He kept a strong and steady pace all the way to the junction.

It was a long way up. The summit of Mt Whitney is at 14,500 feet - today I would climb my first "fourteener." Although the sun beat down strongly and unopposed by clouds, it was downright cold in the thin air with the strong upward wind.

Finally we made it to the junction. One way would take us down to Whitney Portal, our exit point, and the other way, up to the summit. We left our packs at the junction, dressed warmly, and ascended. I felt like I could just flap my wings and fly to the summit without my 35 pound pack weighing me down. We made it up to the tippy top, took the obligatory picture, and made it back down to the junction in less than 90 minutes. That is a guess. I don't have a watch.

Dad was able to get cell reception on the summit and talk to Mom a bit. His mood was obviously improved after that. Dad seemed to walk with a vigor on the way down. I followed happily. The sooner we made it down, the more likely we would be able to have hot greasy restaurant food, and beer for the guys who liked it.

The switchbacks and steep down-steps were brutal. Still, we passed many people on the way down from Whitney, and were passed by none. Although we had walked far and still had far to go, we were invigorated by the finish line. I got hungry again and snacked on deshelled sunflower seeds.

It was a long walk to the portal, but it was downhill, and Dad somehow felt in good shape the entire time, keeping up his brisk pace.

Finally, we could see the portal and were on the home stretch. Someone we passed mentioned the restaurant/shop, saying that it would be open as late as 8:15pm, late enough for us to get burgers and beer. I let out a whoop and Dad and I high-fived.

When we finally got there, Dad was hurting, but we had made great time, arriving not too long after Paul and Mike. As we ordered food, Bruce and Chris arrived. We had all hauled down that mountain, eager to finish.

When the food was ready, I asked for barbecue sauce and doused my chicken sandwich and fries. It tasted like just about the best thing that I ever had. I even ate the pickle on my plate. The guys enjoyed a few beers too, clinking bottles together in merry celebration of completion. Paul responsibly contacted the van service guy and had him come pick us up and take us to Lone Pine village.

Here we had McDonalds milkshakes, showered, and are sleeping in a hostel. Tomorrow we have breakfast and then a bus ride to a bigger town that has a rental car place so that we can get to Los Angeles airport for our flights on Saturday.

I connected my phone to WiFi and saw all the stuff I missed while hiking. Looks like my rent check didn't get sent out correctly. On the other hand, Sara didn't forget about me! I invited her over for dinner Sunday, and she offered to give me a ride from the airport. I'm also really looking forward to my first day of work at Amazon on Monday.

PyWeek #12 challenge starts in 2 days, 16 hours, and 11 minutes.

And I have 23 hours of homework to do in order to clear the Week of Py for development.

I've plotted out THIS week in Google Calendar (the week before PyWeek) - an entry for homework time, break time, and even sleep time. I'm following a strict schedule for optimum time management during PyWeek. The start date is 5pm on Saturday for me, so I have rotated my sleep schedule so that I will go to sleep at 9am on Saturday in order to wake up just as PyWeek starts.

I'm getting pumped up. I can't wait to start!

End of Day 1

Status: Tired, Hungry, Have to Pee

Idea: Side scrolling platformer. You're an odd-shaped lemming-like creature leading a stampede of 9 clones. When you die control is instantly transferred to the next one behind you. Your death affects the environment, so a large part of the gameplay will be suiciding yourself in order to get the next guy behind you to safety. Dying should be fun and comedic.

Screenshot:

Video

Source code on GitHub

Lines of code so far: 612

Decided to go with a tile-based level format.

I got hung up for a while on pyglet's inverted Y axis but I'm getting used to it now.

Having some trouble keeping the frame rate up. I have it up to about 24 but that's not even with a complete level. I'm hoping I won't have to mess with OpenGL directly, as I always encounter bugs that take hours upon hours to find when I do. I'm using pyglet's batch sprite API currently.

Next up on the TODO list is to create more interesting tile types and implement the physics for them. Maybe then getting a parallax background going (hard part is finding/creating the art). I think I'll save animation for later. I'd rather focus on my strengths (programming) and get some good gameplay going, and then if I have time I can make the animation happen. Really wishing I had an artist on board at this point.

Well, time for an 8-hour nap and then it's back to work!

pyglet bug

I ran into a rather unfortunate pyglet bug regarding animation.

This is going to cost me a an hour or two of dev time to use sprite sheets instead of .gif's :-(

In other news, new screenshot:

Also, when I tried to upload my character art () to Facebook, I got this:

LOL!

End of Day 2

Here I am at the end of day 2. According to git log, here's what I've done today:

6:24pm: scrolling 2-layer background 8:23pm: ability to detatch leader and give control to the next guy 10:44pm: suicide bomb control works 11:57pm: +1 dude powerup 12:06am: +infinite dudes powerup 12:21am: land mines 12:35am: spikes 9:20am: totally rewrite display engine and change level format. 9:38am: use pyglet resources rather than the skelington data.py module

Lines of code so far: 542. This is interesting because yesterdays LOC count was 612. After a full day's work and several new features, I ended up with less code. Three cheers for good design!

I'm now using Tiled for editing, which is way nicer than my own hacked up piece of crap level editor. I had fun deleting that.

Thanks to the display engine rewrite (I simply use glTranslatef instead of manually positioning every sprite), all my FPS worries are gone. I'm well above 60. However I have a new problem: weird line artifacts appearing on sprites. It seems almost as if the bottom row of pixels of every sprite is displaying on the top of the sprite instead. You can see it in the screenshot - there isn't supposed to be a flickering line on the spikes.

Video

Next up on the TODO list: use the nifty level editor to create more tiles and a fun and legit Level 1. Then begin to do whatever game upgrades necessary to make Level 1 work. This may involve doing some (shudder) animations.

I should begin to think about music and sound effects too. I am considering making the background music myself. I have been known to make some music in FL Studio. I think it would be fun if some parts of the level pulsated to the beat of the music.

End of Day 3

Video

Lines of code so far: 653

What I've done:

• 7:27pm: fixed the sub-pixel access issue with glTranslatef. thanks richard!
• 12:13am: created animation assets for the main character
• 1:05am: implement animation support in the game logic
• 3:27am: smooth camera movement - follows the character at a damped acceleration curve
• 6:50am: created level 1 in map editor with some unimplemented game ideas and tiles

Here is some new concept art:

Problem: I have a mid-term coming up in 7 hours, and I have not been to class or read the book since the last test. Also I worked through the night so I will be in sleepy-and-not-motivated mode all day while studying and test-taking. Bleh.

Overview of what my TODO list looks like right now:

• try to not fail college
• implement in game logic all the stuff I just pretended would work when I made level 1
• make another level with even more new ideas and then implement them. repeat until satisfied.
• sound effects
• bg music
• HUD
• menu system / title screen
• death / game over
• test on windows and mac

End of Day 4

Good news and bad news. The bad news is that I got at best 55% on my statistics test yesterday. The good news is that I have all kinds of new stuff done for PyWeek. I think I have my priorities in the right place :-)

Video #1

Video #2

• 3:20am: more advanced collision detection which makes walls work
• 4:18am: ramps work
• 4:33am: text boxes work. (that was dead simple, thanks to pyglet!)
• 5:40am: belly flops work.
• 6:16am: ladders work
• 6:52am: decorations work; added house decoration
• 7:18am: add ladder animation
• 8:15am: add monster animations
• 8:51am: added monster who throws you
• 9:23am: explosions will break breakable stuff
• 9:39am: you can explode monsters
• 9:54am: add some tiles to make the level a little nicer
• 11:33am: implement buttons and bridges. level 1 works completely
• 12:21am: support reverse direction for monsters

I talked to my friend Tyler and he wants to make the music for the game, if only he can whip his computing situation into shape in time. I'm crossing my fingers.

I've formulated a plan for the rest of the days I have left:

Day 5:

• sound effects
• HUD
• menu system and title screen
• game over

Day 6:

• implement as many gameplay ideas as possible
• create as many levels as possible exploring those concepts
• bg music if it doesn't work out with tyler

Day 7:

• Dinner with family
• Party with friends
• Test on Mac and Windows and smooth out any glitches/bugs/problems with the experience.
• bug fixing only. no new features

End of Day 5

I'm really fighting sleep-deprivation right now. Going to crash like a rock as soon as I'm done with this diary entry. Like a rock? Is this how a person crashes? I can't remember. Anyways:

• 1:42pm (yes I cheated and worked more after yesterday's diary entry before sleeping): maps support background music and the game engine will stream, play, and loop the background music. I'm really liking how simple this stuff is in pyglet.
• 2:11pm: test on Windows with py2exe. It worked after I added another entry to the resource search path. It's not safe to rely on the .py file location - with py2exe it moves. I better warn people who used the skellington.
• 12:46am (after a nap and Poker Night at Tyler's across town): record, create, and find sound effects
• 2:04am: create a short background music track
• 3:58am: support sound effects in game
• 4:13am: if you don't belly flop, you don't cover as many spikes up
• 4:35am: a HUD which gives you hints as to what buttons you can press
• 7:07am: slim down tiledtmxloader and make it use pyglet.resource
• 8:57am: huge code refactor enabling me to switch scenes from gameplay to title screen or restart gameplay when you die. I took my own advice from Hugoagogo's stack overflow help post and it worked pretty nicely. Hopefully there isn't a memory leak.
• 9:47am: title screen works and has sound
• 9:53am: FPS display is a command line argument, off by default
• 10:10am: on game over, starts level over
• 10:28am: if you beat the level you go on to the next one
• 10:41am: automatically save progress so player can continue from where they left off.
• 11:04am: credits scene when you win

Video

Music

I'm waiting on confirmation to make sure I don't have to swap out my explosion graphic. That would suck.

TODO:

Day 6:

• implement as many gameplay ideas as possible
• create as many levels as possible exploring those concepts
• fix some bugs/nice-to-haves

Day 7 (half-day):

• test on windows and mac
• complete the README and make sure all files are in place on pyweek.org
• download the final submission and play through it on each operating system

Bugs / nice to haves that I probably don't have time for:

• credits song
• physics bugs
• the "beat level" jingle is obnoxious. re-do it
• don't call game over until belly flop is done. he might pick up a +1
• animate the title screen

End of Day 6

Lines of code: 1939

I'm not going to spoil the surprise by posting any more commit log, videos, pictures, or sound. You'll just have to play it to find out!

Things are progressing on track. I already have something playable and working right now, all that needs doing is some more levels and bug fixes.

Good news - Tyler managed to get his computer working and make an awesome track.

I really need sleep but I'm going to be out and about for another 10 hours. I can do this.

TODO:

Day 7 (half-day):

• create at least 3 more levels
• critical bug: if you die several times, memory keeps expanding and sound stops working
• important bug: belly flop stuck in wall, and don't call it game over until all belly flops are turned to dead bodies
• test on windows and mac

I'd really like to, but probably not going to get to these, unfortunately:

• add a gory explosion of bloody organs when you die on spikes
• bad physics glitch when you jump against a vertical wall
• physics glitch with belly flops and spikes
• death shouldn't interrupt the music flow
• the sound that plays when you win is obnoxious and horrible

End of Day 7

Well here I am. 32 minutes to go and I am submitted. That was a gruelling but satisfying experience.

I tried to make as many levels as possible and ironically ended up with 9.

2021 lines of python.

Tyler's having fun with the level editor - I think we may continue to work on this past the competition.

Also I'm going to contribute to Tiled. Having relied on it heavily, I know how to make it better.

I also discovered a few pyglet bugs. I'll submit patches to fix them.

Postmortem

A lot of people are getting stuck on Level 2.

And 3, and 4, and 5, and...

I realize now that I made the levels way too hard. I should have very slowly increased the difficulty.

Each new level shows off more of the game's gameplay elements, so it would be a shame for you to give up and judge without at least checking out all the levels.

If you edit a file called "save_game" (this will be created if you beat level 1) in the directory you run the game from, it has the number of the level you start on when you press Continue from the title screen. You can edit this to skip around and see all 9 levels.

Alternately, there is a YouTube video of me speedrunning through the entire game (only 7 minutes long):