How Compilers Work: From Source Code to Machine Code

You write total = price * quantity and a processor eventually performs a multiply and stores a result in memory. Between those two things is a surprising amount of work, and the program that does it is a compiler. It is one of the most studied pieces of software in computer science, and its design is a clean pipeline: a series of stages that each take the program one step further^[1] from human text and one step closer to machine instructions. Knowing the stages makes error messages less mysterious and performance behavior less like luck.

The pipeline at a glance

A compiler does not translate source to machine code in one leap. It passes the program through a sequence of representations, each simpler and more explicit than the last. The front end figures out what your code means. The middle improves it. The back end turns it into instructions for a specific processor. Breaking the job into stages is what lets compilers support many languages and many processors without rewriting everything each time.

Lexing: text into tokens

The first stage, the lexer, reads your source as a raw stream of characters and groups it into tokens^[2], the smallest meaningful pieces. Whitespace and comments are discarded here.

source:  total = price * 2
tokens:  IDENT(total)  EQUALS  IDENT(price)  STAR  NUMBER(2)

The lexer does not care whether the statement makes sense. It only recognizes that total is an identifier, = is an assignment, 2 is a number, and so on. It turns a flat line of text into a tidy list of labeled pieces for the next stage to work with.

Parsing: tokens into a tree

The parser takes that flat list of tokens and builds structure from it, following the grammar of the language. The result is an abstract syntax tree, or AST^[3], which captures how the pieces relate.

        =
       / \
   total   *
          / \
      price   2

The tree makes precedence and grouping explicit. It knows the multiplication happens before the assignment, and it knows the operands of each operation. This is also the stage that catches syntax errors. If you forget a closing parenthesis or write something the grammar does not allow, the parser is what reports it, because the tokens cannot be assembled into a valid tree.

Semantic analysis: does it actually make sense

A program can be grammatically correct and still be nonsense, like adding a number to a function or using a variable that was never declared. Semantic analysis is the stage that checks meaning.^[4] It resolves names to the things they refer to, tracks scopes so it knows which x you mean, and checks types to make sure operations are valid.

This is where most of the helpful errors come from. "Undefined variable," "type mismatch," and "cannot call a value that is not a function" are semantic errors. The code parsed fine. It just does not add up.

Intermediate representation: a simpler middle language

Rather than translate the checked AST straight to one processor's instructions, most compilers lower it into an intermediate representation, an IR^[5]. The IR is a simpler, more uniform language that is independent of both the source language and the target hardware.

This middle layer is one of the most useful ideas in compiler design. Because optimizations work on the IR, they can be written once and reused for any source language that compiles down to it. Because the final translation starts from the IR, support for a new processor can be added without touching the front end. LLVM is the best known example of this approach, and it is why so many languages, including Rust and Swift, could reuse a mature optimizer and backend instead of building their own.

Optimization: making it faster and smaller

With the program in IR form, the compiler improves it without changing what it does. Constant folding computes expressions that are known ahead of time, so 2 * 60 becomes 120 during compilation instead of at runtime. Dead code elimination removes work whose result is never used. Inlining pastes a small function's body into its caller to avoid the cost of the call. There are dozens more, and they run in passes over the IR. This stage is a large part of why the same code can run much faster when compiled with optimizations turned on.

Code generation: down to the metal

Finally the back end turns the optimized IR into instructions for a specific processor family, such as x86 or ARM. This stage selects the actual machine instructions, and it handles register allocation, deciding which values live in the processor's small set of fast registers and which must be kept in slower memory. The output is machine code, the binary instructions the hardware runs directly.

Interpreters and JITs, by contrast

Not every language takes this whole path to a standalone binary. An interpreter walks the AST or a compiled bytecode and executes it directly, step by step, without producing a separate executable. This makes the edit-and-run loop fast and the program portable, at the cost of some runtime speed.

A just-in-time compiler blends the two approaches. It starts by interpreting, watches which parts of the program run most often, and compiles those hot paths to machine code while the program is running. The JavaScript engines in your browser and the Java Virtual Machine both work this way, which is how a language people once dismissed as slow can run close to native speed on the code that matters.

Every language you use runs some version of this pipeline, whether fully ahead of time, on the fly, or a mix of both. Once you can picture lexing, parsing, checking, optimizing, and generating as separate steps, a compiler stops being a black box. The error you got came from a specific stage, and the speed you saw came from another, and both make a lot more sense when you know where in the line they happened.