Magic isn't real

Any sufficiently advanced technology is indistinguishable from magic.

Arthur C. Clarke

This quote applies just as much to developers as it does non-tech people, sometimes more. I remember towards the beginning of my programming journey (both the first time I learned 18+ years ago, and again ~15 years later), the root cause of the feeling responsible for what they call tutorial hell (I personally loathe tutorials, and always chose to instead try to build things myself, and I attribute a great deal of the relative success I have achieved to this).

The situation:

You feel like you understand perfectly how to properly swing a hammer, lay brick, frame drywall, and you learned the right way to measure and cut beams with a saw, yet you still look at buildings and architecture and stand completely baffled that those tools you have learned were the same ones used to build these great structures. With no idea where to start, you stare at your tools, supplies and materials wondering if they must have some kind of special equipment or secret freemason knowledge that you don’t have access to. You don’t know how someone ended up with that result, using the same tools you see in front of you, and you definitely cannot imagine cutting the first board or laying the first brick.

Many know that this is the exact feeling of learning how to program, and fully grasping the concepts of loops, variables, data structures, trees, stacks, linked-lists, arrays, control flow, etc, etc, etc… then looking at a compiler, a video game, an operating system, web browser and thinking yeah right.... Those devs must all have started programming C and x86 assembly while they were in diapers, and all attended Stanford where they were taught secret knowledge passed down from Ken Thompson, by Brian Kernaghan himself.

Assuming you don’t take the strict path of the JS frameworker vercel user: eventually after enough time, you start to recognize patterns. You ‘go to definition’ on enough methods from libraries you use to see how they are implemented and you build enough side projects and watch enough ‘tsoding daily’, ‘sphaerophoria’, and ‘awesomekling’ to begin to demystify at least how things like web/network protocols, or image/video encodings, syscalls/file IO operations work at some level. You no longer would feel completely lost if you had to write a shell or a lisp interpreter: you would at the very least know that to begin, you would probably have to read the source file into memory and break it up into tokens before trying to parse it to build the syntax tree needed so you can traverse and analyze it before stepping through it to execute the code. Previously, what now feels so obvious to you, would have seemed some kind of sorcery reserved only for the aforementioned programming elite.

I’m sure I’m not alone, in that each time you pull the curtain off a piece of ‘magic’, you have the same thought:

Oooooh yeah. I mean, well duh.. how else would you do that? I can't believe I couldn't see it.

As time goes on, there are less and less things I run into where I cannot mentally parse at least from a very broad and high level, what an implementation might look like. Now I definitely don’t claim to know how kernel internals, 3d rendering, or GPU drivers work, but what I mean is most things have lost the shadowy mystique, and feel more like something I can get excited to learn about, rather than a scary forbidden knowledge I will never be allowed to possess. Although for those things, that may as well be the case ;)

The other day, after a long day’s work managing synchronizing different environments/k8s clusters, I decided to browse HN as I normally do at that time. I ran into a post referencing comptime for Go, that linked to a github repo. It immediately caught my attention, as although I have not written Zig myself, Andrew Kelly is one of my programming idols and I definitely follow zig’s development. Comptime is one of Zig’s most envied language features, and although it is achievable via metaprogramming or constexpr in other languages, zig’s straightforward procedural approach/API makes it particularly unique and admired.

This was when I came upon that familiar feeling:

^^^^^^ Me if you had told me I had to implement comptime in go without touching the compiler

So I decided that I had to know how this was done, and I had a few hours to spare so I decided I would maybe try to contribute, or at least add some kind of feature of any level of value, just to force myself to understand what was going on here.

Then after a brief peruse through the code…

Turns out, you can use the source file information you get through this flag you can pass at build time in Go called -toolexec which allows you to invoke toolchain programs, in this case the prep binary, which is called with the absolute path of the program and by using a combination of another one of the author’s packages, goinject, and the yaegi: (yet another elegant Go interpreter) library: you can get the AST, file decorator and import restorer, by implementing Modifier, which then allows you to collect the variables from the relevant function in the tree, output them each to a temporary file, on which you run the interpreter, giving you the computed results of foo in prep.Comptime(foo()), which you then use to replace the values in the DST by the Modify pass. viola, you have achieved compile time computations.

Oh, well yeah. That makes perfect sense. I mean how else did I think it was gonna work?

After a couple hours, I had added variable scoping, and global variable/const support which I decided was actually not a useful enough of a feature to actually contribute, because each function is evaluated on it’s own, and there is essentially a 0% chance of there being naming/scoping conflicts. But the point is, I didn’t discover that until I had finished the implementation and some tests, and although the feature is useless, the whole process was a very valuable use of my time.

This is just a reminder to everyone at different levels of their developer journey, that the “magic” is not real and the overwhelming majority of the time, you are simply lacking the necessary context and it will likely make perfect sense to you as soon as you have it.

It’s always worth your time to learn parts of the stack that you might not work in daily. As you build your fundamental understanding, it demystifies other pieces of the puzzle that you would never would have put together otherwise. Even if it doesn’t feel important now, I guarantee the knowledge pays off at some point in the future.

Keep learning every day, strive for deeper understanding, and spend time building or hacking on even things that are considered ‘solved problems’. Even if you are only paid to write React, it is very much of value to you and your career to understand how the internals work, or how your one-click ‘serverless’ auto-scaling deployments work…

(hint: servers)