Matthew Berger

Build your own ECS (part 3), change detection, events, tags, and commands

2026-08-09T00:00:00+00:00

This is part 3 of 3 of a series. ← Structural change and queries

Part 2 finished with a working archetype ECS. Spawn into an archetype, add and remove components at runtime, query and iterate by component combination, despawn safely. The storage is solid, the routing is fast, and you could build a small game on it. What you could not do was build a frame loop on it. The missing pieces are the things systems use to talk to each other and to the world structurally.

This post adds four pieces. Change detection records what moved so other systems can do incremental work instead of touching everything each frame. Events let one system message another across the schedule without coupling them. Sparse-set tags carry markers that flip too often to live in the archetype mask. Command buffers queue mutations during iteration so the loop does not invalidate itself. A small system schedule at the end runs the four in order.

By the end of the post you have an ECS kernel that closely mirrors what production libraries expose, in about 825 lines of Rust.

Start from the file at the end of part 2.

#Change detection by watermark

The render system wants to push transforms to the GPU only for entities that moved this frame. Recording "which slot was written" needs to cost less than the work that prompted the write, or the bookkeeping becomes more expensive than the thing it is tracking. A HashSet<Entity> of dirty entities would work but every mutation would touch a hashmap, which fails the cost test.

The cheaper shape is a parallel Vec<u32> of tick stamps next to each component vec, indexed in lockstep with entities. Every mutation writes the current frame's tick into the slot. To find what changed since last frame, walk the stamps and ask whose value is greater than the watermark from the end of the previous frame. No clearing pass between frames. The watermark moves and the old stamps are stale relative to it without ever being touched.

#[derive(Default)]
pub struct ComponentArrays {
    pub mask: u64,
    pub entities: Vec<Entity>,
    pub positions: Vec<Position>,
    pub positions_changed: Vec<u32>,
    pub velocities: Vec<Velocity>,
    pub velocities_changed: Vec<u32>,
}

For every component vec we add a parallel Vec<u32>. The convention is positions[i], positions_changed[i], and entities[i] always describe the same entity slot.

The World grows two counters.

#[derive(Default)]
pub struct World {
    pub allocator: EntityAllocator,
    pub entity_locations: EntityLocations,
    pub tables: Vec<ComponentArrays>,
    pub table_lookup: HashMap<u64, usize>,
    pub table_edges: Vec<TableEdges>,
    pub query_cache: HashMap<u64, Vec<usize>>,
    pub current_tick: u32,
    pub last_tick: u32,
}

current_tick is the value we stamp into a slot when it is modified. last_tick is the watermark. Slots stamped after last_tick are considered changed. The frame-advance function moves both.

impl World {
    pub fn step(&mut self) {
        self.last_tick = self.current_tick;
        self.current_tick = self.current_tick.wrapping_add(1);
    }
}

After step(), last_tick is what current_tick was a moment ago, and current_tick is one higher. Slots stamped during the just-finished frame have ticks == last_tick, which fails the > last_tick check, so they no longer count as changed. Slots modified in the new frame will get the new current_tick, which is > last_tick, so they will. No clearing.

The wrapping_add here is a corner this design papers over. After 2^32 frames the tick rolls back to zero, and a slot stamped at the old u32::MAX suddenly looks ancient when last_tick is also u32::MAX. At 60 frames per second that is about 2.3 years of continuous runtime, fine for a game and not fine for a long-running server. A production engine either widens the tick to u64 or runs a periodic rebasing pass that subtracts the watermark from every stamp.

Every place that pushes to a component vec also needs to push to its _changed vec, and every place that modifies an existing slot needs to stamp the current tick. Spawn first.

impl World {
    pub fn spawn(&mut self, mask: u64) -> Entity {
        let entity = self.allocator.allocate();
        let table_index = self.get_or_create_table(mask);
        let current_tick = self.current_tick;
        let table = &mut self.tables[table_index];
        let array_index = table.entities.len();

        table.entities.push(entity);
        if mask & POSITION != 0 {
            table.positions.push(Position::default());
            table.positions_changed.push(current_tick);
        }
        if mask & VELOCITY != 0 {
            table.velocities.push(Velocity::default());
            table.velocities_changed.push(current_tick);
        }

        self.entity_locations.set(entity, table_index, array_index);
        entity
    }
}

The mutable getters stamp the slot when handed out.

impl World {
    pub fn get_position_mut(&mut self, entity: Entity) -> Option<&mut Position> {
        let (table_index, array_index) = self.entity_locations.get(entity)?;
        let current_tick = self.current_tick;
        let table = &mut self.tables[table_index];
        if table.mask & POSITION == 0 {
            return None;
        }
        table.positions_changed[array_index] = current_tick;
        Some(&mut table.positions[array_index])
    }

    pub fn get_velocity_mut(&mut self, entity: Entity) -> Option<&mut Velocity> {
        let (table_index, array_index) = self.entity_locations.get(entity)?;
        let current_tick = self.current_tick;
        let table = &mut self.tables[table_index];
        if table.mask & VELOCITY == 0 {
            return None;
        }
        table.velocities_changed[array_index] = current_tick;
        Some(&mut table.velocities[array_index])
    }
}

This is conservative. We stamp the slot whether or not the caller actually modifies the data. The alternative (only stamp on actual write) would require wrapping the returned reference in a guard type that stamps in its Drop, which is more machinery than the savings justify.

Several functions need updating to keep the _changed vecs in lockstep with their component vec counterparts. despawn, move_entity, set_position, and set_velocity all need the right pushes, swap_removes, and tick stamps. Every push to positions gets a matching push to positions_changed. Every swap_remove from positions gets a matching swap_remove from positions_changed. Every direct write to positions[index] gets a positions_changed[index] = current_tick next to it. The full file at the end of the post shows every site updated.

One small redundancy lives in set_position's migration path. It calls add_components, which calls move_entity, which already pushes positions_changed.push(current_tick) for the new slot. set_position then re-fetches the location and writes positions_changed[array_index] = current_tick over the just-pushed value. The store is idempotent and the redundancy keeps the post-migration write path identical to the fast no-migration path. If you want to optimize it, you can branch on whether migration happened. The save is not worth the branch.

There is a real catch the inner-loop authors need to know about. The for_each_mut callback from part 2 hands the caller a raw &mut ComponentArrays, direct access to the component vecs, no auto-stamping. A system that writes through that callback does not trigger change detection. The slot's tick stays at whatever value it had before. This is the speed-versus-bookkeeping trade-off. Ergonomic single-entity accessors (set_position, get_position_mut) stamp for you. Hot inner loops keep the raw access and stamp manually when they actually change something. We will see both shapes in the demo.

The new query, "iterate over entities whose components changed since last frame", is the same as for_each_mut but with an extra check.

impl World {
    pub fn for_each_mut_changed<F>(&mut self, include: u64, exclude: u64, mut f: F)
    where
        F: FnMut(Entity, &mut ComponentArrays, usize),
    {
        let since_tick = self.last_tick;
        let table_indices: Vec<usize> = self.cached_tables(include).to_vec();
        for table_index in table_indices {
            let table = &mut self.tables[table_index];
            if table.mask & exclude != 0 {
                continue;
            }
            for array_index in 0..table.entities.len() {
                let mut changed = false;
                if include & POSITION != 0
                    && table.mask & POSITION != 0
                    && table.positions_changed[array_index] > since_tick
                {
                    changed = true;
                }
                if include & VELOCITY != 0
                    && table.mask & VELOCITY != 0
                    && table.velocities_changed[array_index] > since_tick
                {
                    changed = true;
                }
                if changed {
                    let entity = table.entities[array_index];
                    f(entity, table, array_index);
                }
            }
        }
    }
}

A slot is considered changed if any of the queried components have a tick newer than the watermark. For a renderer that wants "every entity whose position changed", this fires only on the slots with fresh position stamps. For a system that wants "every entity whose position OR velocity changed", it fires when either is fresh. Combining changed-ness across components with OR rather than AND is the natural fit, because the typical use case is "redraw this entity if anything about its visual representation moved."

step() runs at the end of the frame. Until you call step(), every modification this frame still counts as changed. Call step() once per frame at the end, after every system has had a chance to read the changes from this frame.

#Events

A collision system finds two entities that overlap and the damage system needs to know. Calling damage methods from inside the collision loop couples the two together. Scribbling a pending-damage component on one of the entities works for one-off cases and turns into a mess when ten systems want to broadcast. The clean shape is a queue. The collision system writes CollisionEvents without naming a receiver. The damage system reads them without naming a sender.

The tricky part is lifetime. The queue cannot empty itself the moment an event is written because a system reading later in the same frame would miss it. It cannot keep events forever because nothing would ever drain. The compromise here is a two-frame rule. An event sent on frame N is readable through the end of frame N+1 and gone by the start of frame N+2. Every system gets a full frame to react regardless of schedule order, and memory is bounded at twice the per-frame volume.

Implementation, double-buffered.

#[derive(Clone)]
pub struct EventQueue<T> {
    pub current: Vec<T>,
    pub previous: Vec<T>,
}

impl<T> Default for EventQueue<T> {
    fn default() -> Self {
        Self {
            current: Vec::new(),
            previous: Vec::new(),
        }
    }
}

impl<T> EventQueue<T> {
    pub fn send(&mut self, event: T) {
        self.current.push(event);
    }

    pub fn read(&self) -> impl Iterator<Item = &T> {
        self.previous.iter().chain(self.current.iter())
    }

    pub fn drain(&mut self) -> impl Iterator<Item = T> + '_ {
        self.previous.drain(..).chain(self.current.drain(..))
    }

    pub fn update(&mut self) {
        self.previous.clear();
        std::mem::swap(&mut self.current, &mut self.previous);
    }

    pub fn len(&self) -> usize {
        self.current.len() + self.previous.len()
    }

    pub fn is_empty(&self) -> bool {
        self.current.is_empty() && self.previous.is_empty()
    }
}

send pushes into the current buffer. read yields everything in both buffers (previous first, then current, the order events were sent). update clears the previous buffer and swaps the current into its place, so the next call starts with previous holding what was just sent and current empty. Two update calls between a send and a read will lose the event.

Each event type gets its own queue, stored as a field on the World.

#[derive(Debug, Clone)]
pub struct CollisionEvent {
    pub entity_a: Entity,
    pub entity_b: Entity,
}

#[derive(Default)]
pub struct World {
    pub allocator: EntityAllocator,
    pub entity_locations: EntityLocations,
    pub tables: Vec<ComponentArrays>,
    pub table_lookup: HashMap<u64, usize>,
    pub table_edges: Vec<TableEdges>,
    pub query_cache: HashMap<u64, Vec<usize>>,
    pub current_tick: u32,
    pub last_tick: u32,
    pub collisions: EventQueue<CollisionEvent>,
}

impl World {
    pub fn send_collision(&mut self, event: CollisionEvent) {
        self.collisions.send(event);
    }

    pub fn read_collisions(&self) -> impl Iterator<Item = &CollisionEvent> {
        self.collisions.read()
    }

    pub fn drain_collisions(&mut self) -> impl Iterator<Item = CollisionEvent> + '_ {
        self.collisions.drain()
    }
}

The familiar fan-out. One set of methods per event type, hand-written here. A macro would generate them. The cost is the same as for component accessors.

step() advances each event queue. It is the same step() that advances the tick.

impl World {
    pub fn step(&mut self) {
        self.collisions.update();
        self.last_tick = self.current_tick;
        self.current_tick = self.current_tick.wrapping_add(1);
    }
}

The order is intentional. Update events first, then advance the tick. By the time the new frame begins, both have rolled over.

#Sparse-set tags

A tag is a HashSet<Entity>. Insert, remove, and membership check are all O(1). Iteration over "all entities with this tag" yields directly from the set. The reason a tag is a hash set instead of a bit in the archetype mask is the migration cost. A bit in the mask means flipping the tag triggers move_entity, which pulls every other component off the entity, pushes them into a new table, and compacts the old slot. For markers that flip often (the "selected" tag in an RTS, a frame-local "took damage this frame" flag, an enemy alertness state), the migration is pure waste compared to a hash insert.

The masking approach is not wrong for tags that rarely change. Player and Enemy are reasonable as archetype bits if those identities are set once at spawn and never updated. The reason this kernel puts them in a HashSet anyway is uniformity. Code that wants to ask "is this entity a player" should not need to know whether the answer comes from a mask check or a hash lookup.

#[derive(Default)]
pub struct World {
    pub allocator: EntityAllocator,
    pub entity_locations: EntityLocations,
    pub tables: Vec<ComponentArrays>,
    pub table_lookup: HashMap<u64, usize>,
    pub table_edges: Vec<TableEdges>,
    pub query_cache: HashMap<u64, Vec<usize>>,
    pub current_tick: u32,
    pub last_tick: u32,
    pub collisions: EventQueue<CollisionEvent>,
    pub players: std::collections::HashSet<Entity>,
    pub enemies: std::collections::HashSet<Entity>,
}

impl World {
    pub fn add_player(&mut self, entity: Entity) {
        if self.entity_locations.get(entity).is_some() {
            self.players.insert(entity);
        }
    }

    pub fn remove_player(&mut self, entity: Entity) -> bool {
        self.players.remove(&entity)
    }

    pub fn has_player(&self, entity: Entity) -> bool {
        self.players.contains(&entity)
    }

    pub fn query_players(&self) -> impl Iterator<Item = Entity> + '_ {
        self.players.iter().copied()
    }

    pub fn add_enemy(&mut self, entity: Entity) {
        if self.entity_locations.get(entity).is_some() {
            self.enemies.insert(entity);
        }
    }

    pub fn remove_enemy(&mut self, entity: Entity) -> bool {
        self.enemies.remove(&entity)
    }

    pub fn has_enemy(&self, entity: Entity) -> bool {
        self.enemies.contains(&entity)
    }

    pub fn query_enemies(&self) -> impl Iterator<Item = Entity> + '_ {
        self.enemies.iter().copied()
    }
}

Insertion and removal are O(1) hash operations with no archetype touch. Querying "all players" yields directly from the set. Combined queries (a for_each over POSITION | VELOCITY plus a has_player filter) walk the tables and check the tag set per entity, which is a cheap hash lookup.

despawn needs to clear the entity out of every tag set, otherwise stale entity handles will accumulate in the sets. Add these lines to the existing despawn.

self.players.remove(&entity);
self.enemies.remove(&entity);

The name "sparse set" comes from the data structure traditionally used here. An array of slot pointers indexed by entity id (so checking membership is an array access), plus a packed list of present entities (so iteration is dense). A HashSet is the simplified version. For a real engine with millions of entities and many tag types, the sparse-set proper is faster, but it has the same external API.

#Command buffers

Iterating over entities while spawning, despawning, or restructuring them is impossible to do safely with direct calls. The iteration borrows the world, so any mutation requires giving up the borrow first. The standard workaround is to queue the operations and apply them later.

The queue is an enum of every operation the world supports.

pub enum Command {
    Spawn { mask: u64 },
    Despawn { entity: Entity },
    AddComponents { entity: Entity, mask: u64 },
    RemoveComponents { entity: Entity, mask: u64 },
    SetPosition { entity: Entity, value: Position },
    SetVelocity { entity: Entity, value: Velocity },
    AddPlayer { entity: Entity },
    RemovePlayer { entity: Entity },
    AddEnemy { entity: Entity },
    RemoveEnemy { entity: Entity },
}

Each operation that can be deferred has a variant. The component-specific Set* variants carry the typed payload, because that information would otherwise be erased.

The buffer is a Vec<Command> on the world, populated by queue methods and drained by apply_commands.

#[derive(Default)]
pub struct World {
    pub allocator: EntityAllocator,
    pub entity_locations: EntityLocations,
    pub tables: Vec<ComponentArrays>,
    pub table_lookup: HashMap<u64, usize>,
    pub table_edges: Vec<TableEdges>,
    pub query_cache: HashMap<u64, Vec<usize>>,
    pub current_tick: u32,
    pub last_tick: u32,
    pub collisions: EventQueue<CollisionEvent>,
    pub players: std::collections::HashSet<Entity>,
    pub enemies: std::collections::HashSet<Entity>,
    pub command_buffer: Vec<Command>,
}

impl World {
    pub fn queue_spawn(&mut self, mask: u64) {
        self.command_buffer.push(Command::Spawn { mask });
    }

    pub fn queue_despawn(&mut self, entity: Entity) {
        self.command_buffer.push(Command::Despawn { entity });
    }

    pub fn queue_add_components(&mut self, entity: Entity, mask: u64) {
        self.command_buffer
            .push(Command::AddComponents { entity, mask });
    }

    pub fn queue_remove_components(&mut self, entity: Entity, mask: u64) {
        self.command_buffer
            .push(Command::RemoveComponents { entity, mask });
    }

    pub fn queue_set_position(&mut self, entity: Entity, value: Position) {
        self.command_buffer
            .push(Command::SetPosition { entity, value });
    }

    pub fn queue_set_velocity(&mut self, entity: Entity, value: Velocity) {
        self.command_buffer
            .push(Command::SetVelocity { entity, value });
    }

    pub fn queue_add_player(&mut self, entity: Entity) {
        self.command_buffer.push(Command::AddPlayer { entity });
    }

    pub fn queue_remove_player(&mut self, entity: Entity) {
        self.command_buffer.push(Command::RemovePlayer { entity });
    }

    pub fn queue_add_enemy(&mut self, entity: Entity) {
        self.command_buffer.push(Command::AddEnemy { entity });
    }

    pub fn queue_remove_enemy(&mut self, entity: Entity) {
        self.command_buffer.push(Command::RemoveEnemy { entity });
    }

    pub fn apply_commands(&mut self) {
        let commands = std::mem::take(&mut self.command_buffer);
        for command in commands {
            match command {
                Command::Spawn { mask } => {
                    self.spawn(mask);
                }
                Command::Despawn { entity } => {
                    self.despawn(entity);
                }
                Command::AddComponents { entity, mask } => {
                    self.add_components(entity, mask);
                }
                Command::RemoveComponents { entity, mask } => {
                    self.remove_components(entity, mask);
                }
                Command::SetPosition { entity, value } => {
                    self.set_position(entity, value);
                }
                Command::SetVelocity { entity, value } => {
                    self.set_velocity(entity, value);
                }
                Command::AddPlayer { entity } => {
                    self.add_player(entity);
                }
                Command::RemovePlayer { entity } => {
                    self.remove_player(entity);
                }
                Command::AddEnemy { entity } => {
                    self.add_enemy(entity);
                }
                Command::RemoveEnemy { entity } => {
                    self.remove_enemy(entity);
                }
            }
        }
    }
}

mem::take swaps the buffer out so the loop iterates over an owned Vec<Command> while leaving an empty buffer in place. This matters because some commands invoke methods that might themselves enqueue more commands. If we held a borrow into the buffer while iterating, those nested enqueues would not compile. With mem::take, nested commands are appended to the new buffer and will be picked up by the next apply_commands call.

The dispatch is a giant match, typed at compile time, no dyn, no allocation per command beyond the vec push. The enum variants are the documentation of what can be deferred.

#Resources

Some state belongs to the world, not to any specific entity. Delta time. The input snapshot for this frame. The time of day. The score. None of these have a sensible owner among the entities, but most systems in the schedule need to read or write at least one of them. The shape that holds this kind of state is a Resources struct attached to the world.

#[derive(Default)]
pub struct Resources {
    pub delta_time: f32,
    pub game_time: f32,
}

#[derive(Default)]
pub struct World {
    pub allocator: EntityAllocator,
    pub entity_locations: EntityLocations,
    pub tables: Vec<ComponentArrays>,
    pub table_lookup: HashMap<u64, usize>,
    pub table_edges: Vec<TableEdges>,
    pub query_cache: HashMap<u64, Vec<usize>>,
    pub current_tick: u32,
    pub last_tick: u32,
    pub collisions: EventQueue<CollisionEvent>,
    pub players: HashSet<Entity>,
    pub enemies: HashSet<Entity>,
    pub command_buffer: Vec<Command>,
    pub resources: Resources,
}

A system reads from resources by accessing world.resources.delta_time and writes by assigning to world.resources.game_time = .... No accessor functions, no per-resource fan-out. The freecs macro shown later in the post generates the same shape from a Resources { delta_time: f32 } block at the macro site. We are building the hand-written equivalent.

#A trivial schedule

Systems are functions that take &mut World. A schedule is a list of named systems run in order each frame. For an introductory build we will not need anything more complicated.

pub type SystemFn = Box<dyn FnMut(&mut World)>;

#[derive(Default)]
pub struct Schedule {
    pub systems: Vec<(&'static str, SystemFn)>,
}

impl Schedule {
    pub fn add<F>(&mut self, name: &'static str, system: F) -> &mut Self
    where
        F: FnMut(&mut World) + 'static,
    {
        self.systems.push((name, Box::new(system)));
        self
    }

    pub fn run(&mut self, world: &mut World) {
        for (_, system) in &mut self.systems {
            system(world);
        }
    }
}

The named entries are for introspection. You can print the system list, find a system by name, swap one out. A production schedule grows hooks for ordering constraints, parallel execution, and conditional running, but for the purpose of this series, an ordered list is enough to drive a frame loop.

#What we built

World  (new fields)
├── current_tick: u32                       stamped on writes
├── last_tick: u32                          watermark for changed-since checks
├── collisions: EventQueue<CollisionEvent>  double-buffered messages
├── players: HashSet<Entity>                sparse-set tag, no archetype touch
├── enemies: HashSet<Entity>                sparse-set tag
├── command_buffer: Vec<Command>            deferred structural changes
└── resources: Resources                    global state, not per-entity

ComponentArrays  (new fields)
├── positions_changed: Vec<u32>             parallel tick array
└── velocities_changed: Vec<u32>            parallel tick array

New operations on World. step to advance the frame, for_each_mut_changed to iterate only the slots touched since last step, send_collision/read_collisions/drain_collisions for cross-system messaging, the tag set with add_player/remove_player/has_player/query_players (and the same for enemy), the command-buffer methods queue_spawn/queue_despawn/queue_set_position/... and apply_commands to flush them, plus direct field access on world.resources for global state. A Schedule struct that runs systems in order each frame.

#Where the abstractions stop being free

We have been doing fan-out by hand for three posts now. Every component type adds a long list of edits.

a field on ComponentArrays
a parallel _changed field
a mask constant
a match arm in component_index
a bit-position constant inside COMPONENT_COUNT
a push site in spawn
two push sites in move_entity (take from source, push to dest)
two swap_remove sites (one in despawn, one in move_entity)
two component bits in the edge-graph loop inside get_or_create_table
four accessor functions (get, get_mut, set, entity_has_*)
a branch in for_each_mut_changed

Adding a tenth component is editing thirty-something call sites, and any one of them being wrong is a silent correctness bug. This is the reason every production ECS in Rust ships with a macro layer.

freecs is what these three posts scale to. Same data layout, same archetype graph, same query cache, same watermark change detection. The difference is a single declarative macro_rules! macro on top that takes one component declaration and writes the entire fan-out for you. The whole equivalent of what we built collapses to one block.

use freecs::{ecs, Entity};

#[derive(Default, Clone)] pub struct Position { pub x: f32, pub y: f32 }
#[derive(Default, Clone)] pub struct Velocity { pub x: f32, pub y: f32 }

#[derive(Debug, Clone)]
pub struct CollisionEvent {
    pub entity_a: Entity,
    pub entity_b: Entity,
}

ecs! {
    World {
        position: Position => POSITION,
        velocity: Velocity => VELOCITY,
    }
    Tags {
        player => PLAYER,
        enemy => ENEMY,
    }
    Events {
        collision: CollisionEvent,
    }
    Resources {
        delta_time: f32,
    }
}

That declaration generates the World struct, the ComponentArrays, every typed accessor (get_position, set_position, get_position_mut, modify_position for closure-style mutation, add_position for adding a defaulted component, entity_has_position, and others), add_components and remove_components with the typed mask helpers, the tag set with its own add_player/has_player/query_player, the double-buffered event queue with send_collision/drain_collision/read_collision, the command buffer with typed queue_set_position variants, the table-edge cache, the query cache, and the change-detection tick stamps. The Schedule type is a separate piece of the crate and works the same way as the one we built.

The scaling answer the hand-built version does not give is right here. Adding health: Health => HEALTH, to the declaration writes the entire per-component fan-out for Health automatically, every accessor, every storage site, every cache update. Adding a new event type or a new tag is one line. The eleven edits per component become zero edits, and a kernel that handles two component types and one event type handles fifty of each the same way.

Using it from a system looks like the following.

fn physics_system(world: &mut World) {
    let dt = world.resources.delta_time;
    world.query_mut()
        .with(POSITION | VELOCITY)
        .iter(|_entity, table, index| {
            table.position[index].x += table.velocity[index].x * dt;
            table.position[index].y += table.velocity[index].y * dt;
        });
}

freecs also generates a few things this series does not build directly. modify_position(entity, f) mutates a component through a closure that releases the borrow at the end of the call, so you can write world.modify_position(entity, |p| p.x += 1.0) and access the world again on the next line without an explicit drop. spawn_batch(mask, count, init) reserves capacity for N entities and runs an init closure on each freshly-pushed slot in one call, which is the primitive a real game uses for bullet spawning, particle systems, and level loading. par_for_each_mut parallelizes iteration via Rayon when the entity count and per-entity work both justify the overhead. Slice iteration (iter_position_slices_mut) hands out &mut [Position] for SIMD-friendly inner loops. The multi-world form ecs! { Game { CoreWorld { ... } RenderWorld { ... } } } splits components across logical worlds with a shared entity allocator when the 64-component-per-world ceiling is not enough. Each of these is a mechanical extension of the kernel from this series, layered on without changing how the data is stored.

The macro design itself is not in this series. Once the kernel underneath exists, writing a macro to stamp out the per-component fan-out is mechanical, and freecs already does it. The production version is on crates.io if you want to use it rather than rebuild it. The source is at matthewjberger/freecs if you want to see how the macro arms work.

#Where the engine goes from here

The ECS is the bottom of the stack. A game engine adds rendering, asset loading, input, audio, scene serialization, scripting hooks, and so on. Each of these is most naturally expressed as components plus systems. The renderer is a system that reads Transform + Mesh + Material and submits draw calls. The asset loader is a system that watches a Pending<Texture> component and replaces it with Loaded<Texture> when the file is ready. Scene serialization is a function that walks for_each(ALL, ...) and emits the components per entity.

The ECS does not solve any of those problems. It makes each of them small and self-contained instead of tangled with everything else.

#The full file

The complete file is around 825 lines and lives as a gist. It compiles standalone in a fresh Cargo project. The main function spawns a player, an enemy, and a landmark, runs a schedule of four systems for four frames, queues a despawn from inside a system at frame two, and exercises change detection, events, tags, and the command buffer in the process.

cargo run produces four frames of output. Frames 0 and 1 print position-redraw lines for the player and enemy as they close in on each other, with no collision yet because they are still more than one unit apart. Frame 2 prints a collision line followed by both redraw lines as the two entities meet at the same x, and the main loop queues the enemy for despawn at the end of the frame. Frame 3 prints only the player's redraw, since the enemy is gone and the landmark has not moved since spawn. The render-changed system never prints the landmark because nothing has touched its position since the initial set_position write, which means its tick stamp stayed at zero while the watermark advanced past it.

One subtlety to call out in collision_system. It collects positions into a Vec<(Entity, Position)> before doing the O(N^2) check rather than reading them through world.get_position during the inner loop. The reason is the borrow checker. We cannot hold immutable borrows of world across the mutable call to world.send_collision later in the same iteration. Collecting up front releases all the read borrows before any of the writes begin. A reader copying this pattern should know that is what the up-front collect() is doing.

freecs lives at matthewjberger/freecs and on crates.io as freecs. It is what nightshade, my game engine, uses for every subsystem that touches entities. Transforms, scene graph, rendering, asset loading, input, audio, scripting, all of it goes through the same World. Once the ECS is in place, the rest of the engine is systems on top.

Build your own ECS (part 2), structural change and queries

2026-08-02T00:00:00+00:00

This is part 2 of 3 of a series. ← Archetype storage | Next → Change detection, events, tags, and commands

Part 1 built the storage. Generational entity handles, archetype tables, and the routing layer that ties them together. By the end of it you could spawn entities into a specific archetype, read and write their components, and despawn them safely. What you could not do was change an entity's components after spawn time, and you had no way to ask the world "give me every entity that has X."

This post fixes both. The core mechanic is the same as before. Archetypes are defined by their component set, so changing an entity's component set means physically moving the entity to a different table. Once that move is in place, queries are almost trivial. They are a walk over the tables whose mask satisfies the query.

The naive implementations of both operations are O(N) in the number of tables, which is fine until you have a few dozen archetypes and several hundred queries per frame. The second half of the post adds two caches, an archetype graph for migration and a query cache for iteration, that turn both operations into near-constant time without changing the surface API.

Start from the file at the end of part 1. We will add to it.

#Adding a component means moving the entity

If an entity is in a table with mask POSITION, and we want to add VELOCITY to it, the entity needs to end up in a table with mask POSITION | VELOCITY. There is no in-place option. The POSITION-only table has no velocities vec to push into, and the velocities slot is only present in tables whose mask includes the velocity bit. So adding a component is mechanically a migration.

Find or create the destination table.
Take each existing component value out of the source table at the entity's slot.
Push those values, plus defaults for the newly-added components, into the destination table.
Update the entity's location to point at the destination table.
swap_remove the entity's slot from the source table, updating the swapped-in entity's location.

Step 2 is the interesting one. We do not want to clone the components. A Position may be cheap to clone, but the same machinery has to handle Vec<Mesh> and Box<MaterialHandle> and other components that are expensive or impossible to clone. We also do not want to allocate. The standard library has the tool we need. std::mem::take swaps a value out, leaving Default::default() in its place. This is why every component must implement Default. Not because spawning needs defaults (though it does), but because migration needs them.

impl World {
    pub fn move_entity(
        &mut self,
        entity: Entity,
        from_table: usize,
        from_index: usize,
        to_table: usize,
    ) {
        let from_mask = self.tables[from_table].mask;

        let position = if from_mask & POSITION != 0 {
            Some(std::mem::take(&mut self.tables[from_table].positions[from_index]))
        } else {
            None
        };
        let velocity = if from_mask & VELOCITY != 0 {
            Some(std::mem::take(&mut self.tables[from_table].velocities[from_index]))
        } else {
            None
        };

        let to_array_index = {
            let to = &mut self.tables[to_table];
            let array_index = to.entities.len();
            to.entities.push(entity);
            if to.mask & POSITION != 0 {
                to.positions.push(position.unwrap_or_default());
            }
            if to.mask & VELOCITY != 0 {
                to.velocities.push(velocity.unwrap_or_default());
            }
            array_index
        };

        self.entity_locations.set(entity, to_table, to_array_index);

        let from = &mut self.tables[from_table];
        let last_index = from.entities.len() - 1;
        let swapped = if from_index < last_index {
            Some(from.entities[last_index])
        } else {
            None
        };

        from.entities.swap_remove(from_index);
        if from.mask & POSITION != 0 {
            from.positions.swap_remove(from_index);
        }
        if from.mask & VELOCITY != 0 {
            from.velocities.swap_remove(from_index);
        }

        if let Some(moved) = swapped {
            self.entity_locations.set(moved, from_table, from_index);
        }
    }
}

The first block pulls every component the source table holds out of the entity's slot, leaving Default::default() behind temporarily. The second block pushes those values into the destination table, defaulting any components that are present in the destination but were not in the source. The third block compacts the source table with the same swap-remove dance as despawn.

unwrap_or_default() is what makes adding a brand-new component to an entity work. The source table did not have a velocity, so velocity is None. The destination has velocity, so we push a default. The same line handles the case where the destination has a component the source already had. Some(value).unwrap_or_default() is value, no default involved.

One precondition move_entity does not check. from_table != to_table. The bottom swap_remove block would clobber the slot we just pushed if they were the same. The public callers add_components and remove_components short-circuit on no-op masks before they reach move_entity, so the precondition holds in practice. A reader copying move_entity somewhere else should re-establish it with an assert or an early return.

#add_components and remove_components

With move_entity in place, the public operations are short.

impl World {
    pub fn add_components(&mut self, entity: Entity, mask: u64) -> bool {
        let Some((table_index, array_index)) = self.entity_locations.get(entity) else {
            return false;
        };
        let current_mask = self.tables[table_index].mask;
        if current_mask & mask == mask {
            return true;
        }
        let new_mask = current_mask | mask;
        let new_table_index = self.get_or_create_table(new_mask);
        self.move_entity(entity, table_index, array_index, new_table_index);
        true
    }

    pub fn remove_components(&mut self, entity: Entity, mask: u64) -> bool {
        let Some((table_index, array_index)) = self.entity_locations.get(entity) else {
            return false;
        };
        let current_mask = self.tables[table_index].mask;
        if current_mask & mask == 0 {
            return true;
        }
        let new_mask = current_mask & !mask;
        let new_table_index = self.get_or_create_table(new_mask);
        self.move_entity(entity, table_index, array_index, new_table_index);
        true
    }
}

Both short-circuit when the operation would be a no-op. Adding components that the entity already has does nothing. Removing components the entity does not have does nothing. The mask test happens before we do any work, so an idempotent call costs a vec lookup and a bitwise check.

Both accept a multi-component mask. If we had a third component HEALTH, then add_components(entity, VELOCITY | HEALTH) would do a single migration to the table for current_mask | VELOCITY | HEALTH. Two separate calls would do two migrations and re-copy the unchanged components twice. One combined call is meaningfully faster.

While we are here, a couple of conveniences. A set_position that adds the component if it is missing makes scripting code much nicer to write. A spawn variant that takes an initial value avoids the spawn(...); world.get_position_mut(e).unwrap() = ... two-step.

impl World {
    pub fn set_position(&mut self, entity: Entity, value: Position) {
        let Some((table_index, array_index)) = self.entity_locations.get(entity) else {
            return;
        };
        if self.tables[table_index].mask & POSITION != 0 {
            self.tables[table_index].positions[array_index] = value;
            return;
        }
        self.add_components(entity, POSITION);
        if let Some((table_index, array_index)) = self.entity_locations.get(entity) {
            self.tables[table_index].positions[array_index] = value;
        }
    }

    pub fn set_velocity(&mut self, entity: Entity, value: Velocity) {
        let Some((table_index, array_index)) = self.entity_locations.get(entity) else {
            return;
        };
        if self.tables[table_index].mask & VELOCITY != 0 {
            self.tables[table_index].velocities[array_index] = value;
            return;
        }
        self.add_components(entity, VELOCITY);
        if let Some((table_index, array_index)) = self.entity_locations.get(entity) {
            self.tables[table_index].velocities[array_index] = value;
        }
    }
}

The structure is the same for both. Fast path if the component is already present, otherwise migrate and then write. The location must be re-fetched after migration because the entity is no longer in the same table.

A second convenience worth adding is a builder that spawns an entity with a chosen set of components in one call. So far every spawn that wants specific component values has been a two-step. Call world.spawn(POSITION | VELOCITY) to get an entity, then write world.get_position_mut(entity).unwrap() = Position { x: 1.0, y: 2.0 } and the same for velocity. That works and it reads awkwardly. The default values the spawn put in exist only to be overwritten on the next line.

A small builder collapses it.

#[derive(Default)]
pub struct EntityBuilder {
    position: Option<Position>,
    velocity: Option<Velocity>,
}

impl EntityBuilder {
    pub fn new() -> Self {
        Self::default()
    }

    pub fn with_position(mut self, value: Position) -> Self {
        self.position = Some(value);
        self
    }

    pub fn with_velocity(mut self, value: Velocity) -> Self {
        self.velocity = Some(value);
        self
    }

    pub fn spawn(self, world: &mut World) -> Entity {
        let mut mask = 0u64;
        if self.position.is_some() {
            mask |= POSITION;
        }
        if self.velocity.is_some() {
            mask |= VELOCITY;
        }
        let entity = world.spawn(mask);
        if let Some(value) = self.position {
            world.set_position(entity, value);
        }
        if let Some(value) = self.velocity {
            world.set_velocity(entity, value);
        }
        entity
    }
}

Usage:

let mover = EntityBuilder::new()
    .with_position(Position { x: 1.0, y: 2.0 })
    .with_velocity(Velocity { x: 0.5, y: 0.25 })
    .spawn(&mut world);

The builder figures out the archetype mask from which fields were set, calls world.spawn(mask) to create the entity in the right archetype directly, then writes each value through world.set_position and world.set_velocity. Going through the public setters rather than reaching into the tables directly means that any bookkeeping the setters do (the change-detection tick stamping we add in part 3, for example) the builder picks up automatically with no extra wiring.

#Queries

A query is "give me every entity that has at least this set of components." With archetype storage it is a walk over the tables. For each table, if its mask is a superset of the query mask, every entity in it satisfies the query. The mask comparison table.mask & query_mask == query_mask is one CPU instruction.

The simplest entity-yielding query is an iterator.

impl World {
    pub fn query_entities(&self, mask: u64) -> impl Iterator<Item = Entity> + '_ {
        self.tables
            .iter()
            .filter(move |table| table.mask & mask == mask)
            .flat_map(|table| table.entities.iter().copied())
    }
}

Using it.

for entity in world.query_entities(POSITION | VELOCITY) {
    let position = world.get_position(entity).unwrap();
    let velocity = world.get_velocity(entity).unwrap();
    println!("{entity:?}: ({}, {}) v=({}, {})", position.x, position.y, velocity.x, velocity.y);
}

This works, and for read-only loops it is the most ergonomic API. But it pays for each component access by going through the location map. For tight inner loops (the physics step, the AI tick) you want direct access to the underlying vecs, where the index into one vec is the index into the others. The pattern is a callback that gets called with the entity, the table, and the slot index.

impl World {
    pub fn for_each<F>(&self, include: u64, exclude: u64, mut f: F)
    where
        F: FnMut(Entity, &ComponentArrays, usize),
    {
        for table in &self.tables {
            if table.mask & include != include || table.mask & exclude != 0 {
                continue;
            }
            for array_index in 0..table.entities.len() {
                let entity = table.entities[array_index];
                f(entity, table, array_index);
            }
        }
    }

    pub fn for_each_mut<F>(&mut self, include: u64, exclude: u64, mut f: F)
    where
        F: FnMut(Entity, &mut ComponentArrays, usize),
    {
        for table in &mut self.tables {
            if table.mask & include != include || table.mask & exclude != 0 {
                continue;
            }
            for array_index in 0..table.entities.len() {
                let entity = table.entities[array_index];
                f(entity, table, array_index);
            }
        }
    }
}

The physics step then looks like the following.

world.for_each_mut(POSITION | VELOCITY, 0, |_entity, table, index| {
    table.positions[index].x += table.velocities[index].x;
    table.positions[index].y += table.velocities[index].y;
});

include is the mask of components the entity must have. exclude is the mask of components the entity must not have, useful for "every entity with POSITION that is not a PLAYER" once we have tags. Passing 0 for exclude is the common case.

The closure receives the whole ComponentArrays, not a tuple of &mut Position, &mut Velocity. The reason is the same as the earlier complaint about boilerplate. Extracting tuples of borrowed components requires either generics with associated types or a macro, and we are deliberately doing neither. Direct table access is uglier at the call site but stays trivial in the implementation.

#Making the migration fast

add_components(entity, VELOCITY) currently does this.

Look up the entity's current table via the location map (entity_locations.get).
Compute new_mask = current_mask | VELOCITY.
Look up the destination table by mask (get_or_create_table, which hits table_lookup).
Migrate.

Step 3 is the slow one. It is a hashmap lookup, and it fires on every structural change. If a thousand entities each take and release a "stunned" component each frame, that is two thousand hashmap probes per frame just for the table lookups. Step 1 is already fast (vec indexed by id). The migration itself is fundamental work.

Notice the structure of step 3. From a given source table, the destination after adding any specific component is fixed. The table for POSITION plus the velocity bit is always the table for POSITION | VELOCITY. So we can store, on each table, an array of "if you add component bit N, here is the destination table index", and another for removal. Lookup becomes an array index, no hashing.

pub const COMPONENT_COUNT: usize = 2;

#[derive(Default, Clone)]
pub struct TableEdges {
    pub add_edges: [Option<usize>; COMPONENT_COUNT],
    pub remove_edges: [Option<usize>; COMPONENT_COUNT],
}

The array is indexed by component bit position, not by mask. Bit 0 is position, bit 1 is velocity. To turn a single-bit mask into its bit position we need a small helper.

pub fn component_index(mask: u64) -> Option<usize> {
    match mask {
        POSITION => Some(0),
        VELOCITY => Some(1),
        _ => None,
    }
}

This only handles single-bit masks. Multi-bit adds (adding more than one component in a single call) cannot be represented as a single edge in the graph, because the destination depends on the combination of bits added together. We will treat those as a slow-path miss.

Storing one TableEdges per table.

#[derive(Default)]
pub struct World {
    pub allocator: EntityAllocator,
    pub entity_locations: EntityLocations,
    pub tables: Vec<ComponentArrays>,
    pub table_lookup: HashMap<u64, usize>,
    pub table_edges: Vec<TableEdges>,
}

The edges need to be populated. There are two ways, lazily on first traversal or eagerly when a new table is created. Eager is simpler because the edge information stays correct without ever needing to invalidate. The cost is a small scan over existing tables when a new table appears.

impl World {
    pub fn get_or_create_table(&mut self, mask: u64) -> usize {
        if let Some(&index) = self.table_lookup.get(&mask) {
            return index;
        }
        let new_table_index = self.tables.len();
        self.tables.push(ComponentArrays {
            mask,
            ..Default::default()
        });
        self.table_edges.push(TableEdges::default());
        self.table_lookup.insert(mask, new_table_index);

        for component_bit in [POSITION, VELOCITY] {
            let Some(component_index_value) = component_index(component_bit) else {
                continue;
            };
            for (existing_index, existing) in self.tables.iter().enumerate() {
                if existing.mask | component_bit == mask {
                    self.table_edges[existing_index].add_edges[component_index_value] =
                        Some(new_table_index);
                }
                if existing.mask & !component_bit == mask {
                    self.table_edges[existing_index].remove_edges[component_index_value] =
                        Some(new_table_index);
                }
            }
        }

        for component_bit in [POSITION, VELOCITY] {
            let Some(component_index_value) = component_index(component_bit) else {
                continue;
            };
            if let Some(&dest) = self.table_lookup.get(&(mask | component_bit)) {
                self.table_edges[new_table_index].add_edges[component_index_value] = Some(dest);
            }
            if mask & component_bit != 0
                && let Some(&dest) = self.table_lookup.get(&(mask & !component_bit))
            {
                self.table_edges[new_table_index].remove_edges[component_index_value] = Some(dest);
            }
        }

        new_table_index
    }
}

Two loops. The first walks the existing tables and sets edges into the new table from any older table that is one bit-flip away. The second walks the component bits and sets edges out of the new table to any older table that is one bit-flip away in the other direction. For each component bit, the add-side destination is mask | bit and the remove-side destination is mask & !bit (only meaningful when the bit was set in mask). The lookups are hashmap gets, but they happen once per bit at table-creation time rather than every time an entity migrates.

The second loop exists because the first one is one-directional. A future table creation will wire its own incoming edges from the older tables that were around when it appeared, but it will not reach back and fill outgoing edges on those older tables. So the new table's outgoing edges have one opportunity to be set, and that opportunity is now. Filling them on creation makes every cached migration path hot from the moment both endpoints exist.

Now add_components for a single-bit mask hits the edge cache.

impl World {
    pub fn add_components(&mut self, entity: Entity, mask: u64) -> bool {
        let Some((table_index, array_index)) = self.entity_locations.get(entity) else {
            return false;
        };
        let current_mask = self.tables[table_index].mask;
        if current_mask & mask == mask {
            return true;
        }

        let cached = if mask.count_ones() == 1 {
            component_index(mask).and_then(|index| self.table_edges[table_index].add_edges[index])
        } else {
            None
        };

        let new_table_index = match cached {
            Some(index) => index,
            None => self.get_or_create_table(current_mask | mask),
        };

        self.move_entity(entity, table_index, array_index, new_table_index);
        true
    }

    pub fn remove_components(&mut self, entity: Entity, mask: u64) -> bool {
        let Some((table_index, array_index)) = self.entity_locations.get(entity) else {
            return false;
        };
        let current_mask = self.tables[table_index].mask;
        if current_mask & mask == 0 {
            return true;
        }

        let cached = if mask.count_ones() == 1 {
            component_index(mask)
                .and_then(|index| self.table_edges[table_index].remove_edges[index])
        } else {
            None
        };

        let new_table_index = match cached {
            Some(index) => index,
            None => self.get_or_create_table(current_mask & !mask),
        };

        self.move_entity(entity, table_index, array_index, new_table_index);
        true
    }
}

For a single-component add or remove, the cached array lookup replaces the hashmap lookup. Multi-component changes still hit get_or_create_table, which is fine because they are rare and the hashmap is the only sensible structure for arbitrary masks. A real implementation would add a secondary cache per source table for common multi-bit transitions. The single-bit case is what fires every time a status effect is added or removed, so it earns the optimization.

#Making the query fast

for_each_mut(POSITION | VELOCITY, 0, ...) scans every table to find the ones whose mask matches. With a few dozen archetypes and a few hundred query invocations per frame, that scan adds up.

The fix is the same idea as the edge cache. Memoize. Map each query mask to the list of table indices that satisfy it.

#[derive(Default)]
pub struct World {
    pub allocator: EntityAllocator,
    pub entity_locations: EntityLocations,
    pub tables: Vec<ComponentArrays>,
    pub table_lookup: HashMap<u64, usize>,
    pub table_edges: Vec<TableEdges>,
    pub query_cache: HashMap<u64, Vec<usize>>,
}

The cache fills on first use.

impl World {
    pub fn cached_tables(&mut self, mask: u64) -> &[usize] {
        if !self.query_cache.contains_key(&mask) {
            let matching: Vec<usize> = self
                .tables
                .iter()
                .enumerate()
                .filter(|(_, table)| table.mask & mask == mask)
                .map(|(index, _)| index)
                .collect();
            self.query_cache.insert(mask, matching);
        }
        &self.query_cache[&mask]
    }
}

It needs to stay correct as new tables are created. The simplest invalidation strategy would be to clear the whole cache on every new table, but that is overly conservative. Most existing cache entries are still correct, and only those whose query mask is a subset of the new table's mask need updating. We can do that surgically inside get_or_create_table.

impl World {
    pub fn invalidate_query_cache_for_new_table(
        &mut self,
        new_mask: u64,
        new_table_index: usize,
    ) {
        for (query_mask, cached_tables) in self.query_cache.iter_mut() {
            if new_mask & query_mask == *query_mask {
                cached_tables.push(new_table_index);
            }
        }
    }
}

Called from get_or_create_table right after the table is pushed.

impl World {
    pub fn get_or_create_table(&mut self, mask: u64) -> usize {
        if let Some(&index) = self.table_lookup.get(&mask) {
            return index;
        }
        let new_table_index = self.tables.len();
        self.tables.push(ComponentArrays {
            mask,
            ..Default::default()
        });
        self.table_edges.push(TableEdges::default());
        self.table_lookup.insert(mask, new_table_index);

        self.invalidate_query_cache_for_new_table(mask, new_table_index);

        for component_bit in [POSITION, VELOCITY] {
            let Some(component_index_value) = component_index(component_bit) else {
                continue;
            };
            for (existing_index, existing) in self.tables.iter().enumerate() {
                if existing.mask | component_bit == mask {
                    self.table_edges[existing_index].add_edges[component_index_value] =
                        Some(new_table_index);
                }
                if existing.mask & !component_bit == mask {
                    self.table_edges[existing_index].remove_edges[component_index_value] =
                        Some(new_table_index);
                }
            }
        }

        for component_bit in [POSITION, VELOCITY] {
            let Some(component_index_value) = component_index(component_bit) else {
                continue;
            };
            if let Some(&dest) = self.table_lookup.get(&(mask | component_bit)) {
                self.table_edges[new_table_index].add_edges[component_index_value] = Some(dest);
            }
            if mask & component_bit != 0
                && let Some(&dest) = self.table_lookup.get(&(mask & !component_bit))
            {
                self.table_edges[new_table_index].remove_edges[component_index_value] = Some(dest);
            }
        }

        new_table_index
    }
}

for_each_mut consults the cache instead of scanning.

impl World {
    pub fn for_each_mut<F>(&mut self, include: u64, exclude: u64, mut f: F)
    where
        F: FnMut(Entity, &mut ComponentArrays, usize),
    {
        let table_indices: Vec<usize> = self.cached_tables(include).to_vec();

        for table_index in table_indices {
            let table = &mut self.tables[table_index];
            if table.mask & exclude != 0 {
                continue;
            }
            for array_index in 0..table.entities.len() {
                let entity = table.entities[array_index];
                f(entity, table, array_index);
            }
        }
    }
}

The .to_vec() copy is needed because cached_tables borrows the cache mutably (to fill it on miss) and then we want to borrow self.tables mutably to call f. A handful of usize copied to the stack is cheap.

The read-only for_each does not have the borrow problem, but it also cannot fill the cache, because the cache write requires &mut self. A fuller implementation would either expose a warm_query(mask) method that fills the cache eagerly during setup, or use a RefCell inside the cache so the read-only path can still write. The warm-up route is what production crates usually take because it avoids interior mutability on the hot read path. For this post the read-only path scans the tables every time.

#What we built

World  (new fields)
├── table_edges: Vec<TableEdges>           archetype graph for O(1) migration
└── query_cache: HashMap<u64, Vec<usize>>  memoized "which tables match this mask"

TableEdges  (one per table, indexed by component bit position)
├── add_edges: [Option<usize>; N]          which table to land in when adding bit N
└── remove_edges: [Option<usize>; N]       which table to land in when removing bit N

New operations on World. add_components(entity, mask), remove_components(entity, mask), set_position/set_velocity (write or add-then-write), query_entities(mask), for_each(include, exclude, ...), for_each_mut(include, exclude, ...). Migrations and queries both hit the caches on the common path and fall back to the hashmap on misses.

#What is missing

You can build a small game with what we have so far. The core loop is here. Spawn, attach components, run systems over component combinations, change components in response to events, despawn. But there are real limitations.

There is no way for a system to know which entities changed. The render system has to walk every entity with a transform every frame, even if nothing moved. The networking system has to serialize every component, not just the dirty ones. Change detection (recording when each component was last modified, and offering an "only iterate the recently-changed ones" query) is the single biggest performance win for systems that do incremental work.

There is no event system. Collision detection wants to emit "A hit B" and have damage application consume that message later in the frame. Today the only way to do this is to scribble it into a component, which is fine for one-off cases and a mess in aggregate.

There is no notion of a tag. A lightweight marker like "this entity is a player" or "this entity is selected." We could add a unit component Player; and put it in the archetype mask, but then adding the player tag to an entity migrates it to a new archetype, and removing it migrates it back. For markers that change frequently (think "selected" in an RTS), the migration cost is wasted. The standard solution is to store tags outside the archetype storage, in a side structure.

There is no way to safely make structural changes during iteration. A system iterating over enemies cannot despawn them mid-loop without invalidating the iteration. The standard solution is a command buffer. Queue the changes, apply them at the end.

Part 3 adds all four (change detection, events, sparse-set tags, and command buffers) and finishes with a small system schedule that ties everything together into a frame loop.

#The full file

Same shape as the part 1 checkpoint, dropping into src/main.rs of a fresh project. Available as a gist. cargo run should show the mover advancing while the landmark stays put, then both entities listed by the final query_entities loop. The mover went through three archetypes during execution. POSITION at spawn, POSITION | VELOCITY after the add, and back to POSITION after the remove. It kept its data the whole way.

Build your own ECS (part 1), archetype storage

2026-07-26T00:00:00+00:00

This is part 1 of 3 of a series. Next → Structural change and queries

ECS stands for Entity Component System. An entity is a handle, a small id with no data and no methods. A component is a struct of data attached to an entity, like Position or Velocity. A system is a function that reads or writes components on the entities that match a query, like "every entity with both Position and Velocity." The data lives in components, the work happens in systems, and entities are the keys that line them up.

The fast layout for an ECS groups entities by which components they have and lays out each component type as its own contiguous Vec. A query for "every entity with position and velocity" turns into a tight loop over packed memory rather than a HashMap walk. Mutating a component is a write into a known slot, not a heap indirection. Adding a component is a one-time migration to a different bucket. This is the shape almost every fast Rust ECS uses under the hood, and the next three posts build it from scratch.

End state at part 3 is around 800 lines of straightforward Rust with structural change at runtime, queries, change detection, events, sparse-set tags, deferred command buffers, and a small system schedule. No proc-macros, no unsafe, only the standard library.

Aimed at Rust developers who want to understand how an archetype ECS is shaped, well enough to write one or to read a production crate without guessing. If you want a finished version to drop in rather than build, freecs is on crates.io and saves you the work.

Part one covers spawn, despawn, and reading and writing components inside a fixed archetype. Runtime structural change is part two. Events, change detection, tags, and command buffers are part three. The complete file (just under 250 lines) sits at the end of this post.

What gets built over the series is the kernel of freecs, my Rust ECS crate. freecs wraps the same data layout in a declarative macro that writes the typed component accessors for you, and it is the foundation of nightshade, my game engine.

#Why this layout

The choice that makes the design fast is data-oriented. Object-oriented programming groups data by what kind of thing is being modeled, with the fields of a game object packed inside an instance of its class. Data-oriented design groups data by how the data gets accessed, with all the positions stored together because the physics loop iterates over positions, all the velocities stored together because it iterates them alongside. Systems become functions over arrays of data instead of methods on objects. The storage is laid out for the access pattern, not for the conceptual identity of a thing.

For ten thousand entities and a physics step that reads position and velocity from each, the difference is concrete. An OOP layout pays a cache miss per access, because each object's fields live wherever the allocator put them. The DOD layout pays one cache miss per cache line of contiguous data, which holds several positions at a time. The savings compound across every iteration of every system that runs each frame.

Inheritance also fails the modeling side. Real game objects are combinations of orthogonal capabilities. A destructible barrel, a chest that holds items and can be locked, a vendor that takes damage and runs an AI routine. Forcing a tree shape on that means committing to one combination and breaking the next. Composition over Vec<Box<dyn Component>> per entity expresses the combinations correctly but routes every access through a v-table dispatch and a heap indirection, so it solves the modeling problem by reintroducing the cache problem.

What works is to flip the storage. Group entities by which components they have, lay out each component type as its own dense Vec, and the question becomes "which buckets satisfy this query" plus "walk those buckets." Iteration is sequential reads through packed memory, which matches what the hardware prefetcher is optimized for. Adding a component triggers a one-time migration to a different bucket rather than paying a per-access cost forever.

The rest of part one is how to build that shape in Rust.

#A note on style before the code starts

Everything in this series uses plain structs with public fields. The World type has methods, but the fields underneath are pub, and the methods are not protecting any invariants. They are there because writing world.spawn(mask) is nicer than writing spawn(&mut world, mask). If the methods were not there, free functions would do the same job. Nothing is encapsulated for OOP reasons.

&mut self shows up because Rust needs it for any function that mutates the world, not because the data is private. Where direct field access is more convenient than a method, just reach in.

#Entity handles

An Entity is not an object. It owns nothing. It is a small handle the storage uses to find the components.

A bare u32 would work, but it has a hazard. If you despawn entity 42 and then spawn a new one, the allocator will reuse id 42 (otherwise ids grow without bound). Now any code holding the old handle to 42 will silently access the new entity's data. This is the ABA problem, and in a game it produces bugs that are extremely hard to diagnose because nothing crashes. Your AI just suddenly starts targeting a wall.

The fix is a generation counter. Every time an id is recycled, its generation is bumped. The handle stores both the id and the generation it was minted at. The storage refuses to honor a handle whose generation does not match the current generation for that slot. Stale handles fail loudly with None instead of returning the wrong data.

#[derive(Default, Copy, Clone, Debug, Eq, PartialEq, Hash)]
pub struct Entity {
    pub id: u32,
    pub generation: u32,
}

Copy because handles are cheap to pass around. Eq and Hash because we will want to put them in sets later.

#Allocating entities

The allocator hands out Entity handles. It needs two things. A counter for the next never-before-used id, and a freelist of recently-freed (id, next_generation) pairs to reuse.

#[derive(Default)]
pub struct EntityAllocator {
    pub next_id: u32,
    pub free: Vec<(u32, u32)>,
}

impl EntityAllocator {
    pub fn allocate(&mut self) -> Entity {
        if let Some((id, generation)) = self.free.pop() {
            Entity { id, generation }
        } else {
            let id = self.next_id;
            self.next_id += 1;
            Entity { id, generation: 0 }
        }
    }

    pub fn deallocate(&mut self, entity: Entity) {
        self.free.push((entity.id, entity.generation.wrapping_add(1)));
    }
}

On deallocate, we precompute the next generation to be handed out for that id and stash it in the freelist. wrapping_add(1) so that an id used billions of times still works (in practice this never happens, but it costs nothing). The next allocation pulls that pair off the freelist and produces a handle with the bumped generation. Old handles with the old generation will not match.

next_id itself uses regular += 1. At u32::MAX ever-spawned ids it will overflow, panicking in debug and wrapping in release. That is four billion unique entity ids across the lifetime of the program, only a real concern for long-running services that churn entities aggressively. A production allocator would either widen next_id to u64 or recycle ids more aggressively. We leave it as is.

A quick sanity check at this point.

let mut allocator = EntityAllocator::default();
let first = allocator.allocate();
assert_eq!(first.generation, 0);
allocator.deallocate(first);
let recycled = allocator.allocate();
assert_eq!(recycled.id, first.id);
assert_eq!(recycled.generation, 1);

The allocator does not know whether a handle is currently valid. That bookkeeping lives elsewhere. Its job is to mint id-and-generation pairs that are unique over the lifetime of the program.

#Components are just structs

Components are plain data. No traits, no inheritance, no marker interfaces. We will start with two.

#[derive(Default, Clone, Debug)]
pub struct Position {
    pub x: f32,
    pub y: f32,
}

#[derive(Default, Clone, Debug)]
pub struct Velocity {
    pub x: f32,
    pub y: f32,
}

Default is required because of how component migration works in part 2. We will rely on mem::take to move components between archetypes without allocation, and mem::take requires Default. Clone is convenient for tests. We are not adding any other constraint.

The "components are just structs" rule is more important than it looks. Component types are decoupled from the storage. Any struct can become a component just by declaring its existence to the world. There is no impl Component for Foo ceremony.

#Why HashMap is the wrong storage

Before we build the archetype storage, it is worth understanding why the obvious approach fails.

struct NaiveWorld {
    positions: HashMap<Entity, Position>,
    velocities: HashMap<Entity, Velocity>,
}

This is correct. It is also slow. To advance physics you iterate over every entity that has both a position and a velocity.

for (entity, position) in &mut world.positions {
    if let Some(velocity) = world.velocities.get(entity) {
        position.x += velocity.x;
        position.y += velocity.y;
    }
}

For each entity, one hash lookup. The position and velocity for the same entity live at unrelated heap addresses, so each access is a cache miss. With ten thousand entities that is ten thousand hash lookups and twenty thousand probable cache misses every frame.

The data we want to access together is not stored together. Fixing that is the whole point of archetypes.

#Grouping by component set

Group entities by which set of components they have. Every entity that has exactly Position + Velocity lives in one bucket. Every entity that has only Position lives in another. Inside each bucket, store each component type as its own Vec in slot-aligned order. The position at index 7 and the velocity at index 7 belong to the same entity.

This is "structure of arrays" instead of "array of structures." The physics loop becomes the following.

for table in tables_with(POSITION | VELOCITY) {
    for index in 0..table.entities.len() {
        table.positions[index].x += table.velocities[index].x;
        table.positions[index].y += table.velocities[index].y;
    }
}

Inside the inner loop, positions[index] and velocities[index] are stride-1 accesses on dense arrays the CPU prefetcher can predict. No hash lookups. No indirection. The component combination is identified by a bitmask, where each component type is assigned a bit.

pub const POSITION: u64 = 1 << 0;
pub const VELOCITY: u64 = 1 << 1;

A u64 mask gives us 64 component types per world. That is the practical ceiling of this design. 64 is enough for most games. If you need more, freecs has a multi-world macro form that splits components across logical worlds with a shared entity allocator, mentioned at the end of part 3.

#The archetype table

#[derive(Default)]
pub struct ComponentArrays {
    pub mask: u64,
    pub entities: Vec<Entity>,
    pub positions: Vec<Position>,
    pub velocities: Vec<Velocity>,
}

mask records which components this table holds. entities records which entity owns each slot, indexed in lockstep with the component arrays. positions and velocities get pushed to only when the corresponding bit is set in mask, otherwise they stay empty. A table with mask POSITION (bit 0 set, bit 1 clear) has entries in positions and an empty velocities.

This struct is awkward. We are hard-coding two component fields and two corresponding masks, and adding a new component means editing this struct, the spawn code, the despawn code, the read code, the write code. A real ECS solves this with code generation. A macro stamps out the per-component fields and accessors from a single declaration. We will not do that here. The whole point of the series is to see the machine working, and a macro would hide the moving parts. The macro layer that scales this kernel cleanly to arbitrary component sets is exactly what freecs is, and part 3 shows what its ecs! declaration looks like next to the hand-rolled equivalent.

#Routing entities to their slots

We have entities (handles) and we have tables (storage). We need a way to look up "given this entity, where do its components live." A HashMap<Entity, (table_index, array_index)> would work, but entity ids come out of a small counter and most of them will be densely packed. A Vec indexed by id is faster and just as compact.

#[derive(Default, Copy, Clone)]
pub struct EntityLocation {
    pub generation: u32,
    pub table_index: u32,
    pub array_index: u32,
    pub allocated: bool,
}

#[derive(Default)]
pub struct EntityLocations {
    pub locations: Vec<EntityLocation>,
}

Each slot stores the generation of the entity currently at this id, the (table, index) of its components, and an allocated flag. The flag flips between two states. true while an entity is live at this id, false once that entity has been despawned. The generation lets a recycled handle be distinguished from the entity that previously held its id. The allocated flag lets a fully-deallocated id be distinguished from anything live. Together they reject every stale handle.

EntityLocation::default() setting allocated: false is load-bearing. When ensure_slot grows the backing vec, it fills the new entries with EntityLocation::default(). A slot that has been grown into but never written looks identical to a despawned slot, and get rejects both. We get safe defaults for free from the derive.

impl EntityLocations {
    pub fn ensure_slot(&mut self, id: u32) {
        let needed = id as usize + 1;
        if needed > self.locations.len() {
            self.locations.resize(needed, EntityLocation::default());
        }
    }

    pub fn get(&self, entity: Entity) -> Option<(usize, usize)> {
        let location = self.locations.get(entity.id as usize)?;
        if !location.allocated || location.generation != entity.generation {
            return None;
        }
        Some((location.table_index as usize, location.array_index as usize))
    }

    pub fn set(&mut self, entity: Entity, table_index: usize, array_index: usize) {
        self.ensure_slot(entity.id);
        self.locations[entity.id as usize] = EntityLocation {
            generation: entity.generation,
            table_index: table_index as u32,
            array_index: array_index as u32,
            allocated: true,
        };
    }

    pub fn mark_deallocated(&mut self, id: u32) {
        if let Some(location) = self.locations.get_mut(id as usize) {
            location.allocated = false;
        }
    }
}

get is the gatekeeper. It refuses to return a location if the slot is unallocated or if the requested entity's generation does not match. Stale handles die here. The function returns Option<(usize, usize)> rather than Option<&EntityLocation> because the caller only ever needs the table index and the array index inside it.

ensure_slot resizes the backing vec to fit the requested id. We use resize, not doubling, because the allocator hands out ids monotonically when the freelist is empty, so growing on demand by exact size is fine.

#The World

use std::collections::HashMap;

#[derive(Default)]
pub struct World {
    pub allocator: EntityAllocator,
    pub entity_locations: EntityLocations,
    pub tables: Vec<ComponentArrays>,
    pub table_lookup: HashMap<u64, usize>,
}

Four fields. The allocator (so it can mint entities), the location map (so it can route), the tables (so it can store components), and a mask to table_index lookup so we can find the right table for a given component combination without scanning.

Finding or creating a table.

impl World {
    pub fn get_or_create_table(&mut self, mask: u64) -> usize {
        if let Some(&index) = self.table_lookup.get(&mask) {
            return index;
        }
        let index = self.tables.len();
        self.tables.push(ComponentArrays { mask, ..Default::default() });
        self.table_lookup.insert(mask, index);
        index
    }
}

The first time someone spawns an entity with POSITION | VELOCITY, this creates a fresh table for that mask. Every subsequent spawn with the same mask reuses it.

#Spawning

impl World {
    pub fn spawn(&mut self, mask: u64) -> Entity {
        let entity = self.allocator.allocate();
        let table_index = self.get_or_create_table(mask);
        let table = &mut self.tables[table_index];
        let array_index = table.entities.len();

        table.entities.push(entity);
        if mask & POSITION != 0 {
            table.positions.push(Position::default());
        }
        if mask & VELOCITY != 0 {
            table.velocities.push(Velocity::default());
        }

        self.entity_locations.set(entity, table_index, array_index);
        entity
    }
}

Allocate the handle, find the table, append a slot to every component vec the mask says is present, record the entity's location. The mask-checked pushes are the manual fan-out the macro would generate. In a hand-written ECS, this block grows linearly with the number of component types, which is the main reason ECS libraries use macros.

The components are initialized to Default. In part 2 we will add a way to spawn with custom values. For now, the caller writes the components after spawn.

#Reading and writing components

impl World {
    pub fn get_position(&self, entity: Entity) -> Option<&Position> {
        let (table_index, array_index) = self.entity_locations.get(entity)?;
        let table = &self.tables[table_index];
        if table.mask & POSITION == 0 {
            return None;
        }
        Some(&table.positions[array_index])
    }

    pub fn get_position_mut(&mut self, entity: Entity) -> Option<&mut Position> {
        let (table_index, array_index) = self.entity_locations.get(entity)?;
        let table = &mut self.tables[table_index];
        if table.mask & POSITION == 0 {
            return None;
        }
        Some(&mut table.positions[array_index])
    }

    pub fn get_velocity(&self, entity: Entity) -> Option<&Velocity> {
        let (table_index, array_index) = self.entity_locations.get(entity)?;
        let table = &self.tables[table_index];
        if table.mask & VELOCITY == 0 {
            return None;
        }
        Some(&table.velocities[array_index])
    }

    pub fn get_velocity_mut(&mut self, entity: Entity) -> Option<&mut Velocity> {
        let (table_index, array_index) = self.entity_locations.get(entity)?;
        let table = &mut self.tables[table_index];
        if table.mask & VELOCITY == 0 {
            return None;
        }
        Some(&mut table.velocities[array_index])
    }
}

Two reasons a get can fail. The entity is invalid (stale handle, despawned, never existed), or the entity is valid but its archetype does not contain this component. The first case bails out at entity_locations.get. The second is the mask & POSITION == 0 check.

This is fan-out again. Four near-identical functions per component type. A macro would write them. We are not.

#Despawning

Despawn has to do three things. Invalidate the entity so future lookups fail, remove its components from the table so they do not leak, and keep the table compact so iteration stays fast.

The compactness requirement is the source of all the complexity. We could mark the slot as a tombstone and skip it during iteration, but that wastes memory and slows iteration with branching. The standard answer is swap_remove. Take the last entry in the vec and move it into the deleted slot. That keeps the vec contiguous. The cost is that the entity formerly at the last slot now lives at a different array index, so its EntityLocation needs updating.

impl World {
    pub fn despawn(&mut self, entity: Entity) -> bool {
        let Some((table_index, array_index)) = self.entity_locations.get(entity) else {
            return false;
        };

        self.entity_locations.mark_deallocated(entity.id);
        self.allocator.deallocate(entity);

        let table = &mut self.tables[table_index];
        let last_index = table.entities.len() - 1;
        let swapped = if array_index < last_index {
            Some(table.entities[last_index])
        } else {
            None
        };

        table.entities.swap_remove(array_index);
        if table.mask & POSITION != 0 {
            table.positions.swap_remove(array_index);
        }
        if table.mask & VELOCITY != 0 {
            table.velocities.swap_remove(array_index);
        }

        if let Some(moved) = swapped {
            self.entity_locations.set(moved, table_index, array_index);
        }

        true
    }
}

Order matters here. We capture which entity is about to be moved before we do the swap_remove, because afterwards the last index is gone and we cannot recover it. We do all the swap_removes on the component vecs in lockstep, since entities[i], positions[i], and velocities[i] must always describe the same entity. Then we update the moved entity's location.

If the despawned entity was the last one in its table, there is no swap to track.

#What we built

World
├── allocator: EntityAllocator          mints generational handles
├── entity_locations: EntityLocations   maps entity id to (table, slot)
├── tables: Vec<ComponentArrays>        one per archetype
└── table_lookup: HashMap<u64, usize>   mask to table index

ComponentArrays  (one per unique component combination)
├── mask: u64                           which components this table holds
├── entities: Vec<Entity>               which entity owns each slot
├── positions: Vec<Position>            populated iff mask & POSITION
└── velocities: Vec<Velocity>           populated iff mask & VELOCITY

EntityLocation  (one per entity id)
├── generation: u32                     ABA guard
├── table_index: u32
├── array_index: u32
└── allocated: bool                     stale-handle guard

Operations on World. spawn(mask), get_position/get_position_mut, get_velocity/get_velocity_mut, despawn. Despawned ids get recycled with a bumped generation, and stale handles fail closed with None.

#What is missing

There are several gaps in what we built. None of them are accidents.

You cannot add a component to an entity that did not have it at spawn time. There is no add_velocity(entity). The entity is locked into the archetype it was spawned into. Real games need to add and remove components at runtime (a "stunned" effect, equipment, AI states), and the way archetypes handle this is the central trick of part 2. The entity has to be physically moved to a different table.

There is no way to query. You can ask "what is this entity's position?" but you cannot ask "give me every entity that has both a position and a velocity." That is also part 2.

There is no set_position that adds the component if it is missing. There is no batch spawn. There is no parallelism. There is no change tracking, no events, no tags, no commands. Those are part 3.

Part 2 picks up structural change and queries. Making components addable and removable at runtime, walking the tables to find entities that match a component combination, and the two caches (archetype graph for migration, query cache for iteration) that turn the naive O(N) scans into near-constant time without changing the surface API.

#The full file

The complete file is around 250 lines and lives as a gist. Run cargo new ecs && cd ecs, replace src/main.rs with its contents, then cargo run. It should print the mover's state and then ok. The final block of main exercises the generational guard. After we despawn mover, the next spawn reuses its id but bumps the generation, and the original handle no longer resolves.

Building a static site generator in Rust

2026-05-16T00:00:00+00:00

This site is generated by bamboo, the static site generator I wrote in Rust and use to build matthewberger.dev/articles. Every page you read here starts as a markdown file with a TOML header on disk, gets turned into HTML by bamboo, and lands in a dist/ directory that gets uploaded to GitHub Pages. There is no server doing work at request time, no database, no framework. The deployed site is a few hundred plain files.

This post is about how that machine is shaped. The pieces, why they ended up that way, and what the awkward parts cost. I will walk through the code in the order data flows through it: read the config, walk the content tree, render markdown, expand shortcodes, run templates, drop files, post-process.

The whole codebase is in the matthewjberger/bamboo repo. Under ten thousand lines of Rust including tests, no unsafe, no proc macros, edition 2024. The library crate is on crates.io as bamboo-ssg and the CLI as bamboo-cli.

#What a static site generator actually is

A static site generator's input is a tree of markdown and configuration. Its output is a directory of HTML, CSS, and assets, laid out so a dumb file server can serve them at the right URLs. "Static" means the work happens at build time. Once dist/ is written, the site has no runtime dependencies. No PHP, no Node process, no database. You hand the directory to GitHub Pages or a CDN bucket and that is the whole deployment.

I wrote my own because the existing ones do not compose the way I want. Zola, Hugo, Jekyll, Eleventy, and mdBook are all fine. Each has a paint-yourself-into-a-corner moment for some thing I wanted to do (multi-collection sites with their own templates, fingerprinted assets and Sass without a Node toolchain, a search index I controlled the shape of, an embedded default theme that needs no separate clone step). bamboo started as a small tool to host one site and accreted features until it could host the four I care about.

#Shape: builder, renderer, CLI

bamboo splits cleanly into three pieces.

SiteBuilder reads a source directory and produces an in-memory Site. ThemeEngine consumes that Site and renders to disk. The CLI ties them together with clap for argument parsing and a notify watcher for live reload.

The split matters because it makes the library usable as a library. You can build the Site, mutate it (drop drafts, reorder posts, attach extra metadata), then hand it to a different renderer. The Site type is the IR:

pub struct Site {
    pub config: SiteConfig,
    pub home: Option<Page>,
    pub pages: Vec<Page>,
    pub posts: Vec<Post>,
    pub collections: HashMap<String, Collection>,
    pub data: HashMap<String, Value>,
    pub assets: Vec<Asset>,
}

Every field is pub. The Page/Post/Collection types all flatten a Content substruct in via #[serde(flatten)], so templates see the inner fields directly instead of having to walk through content.title. That flattening turns the struct nesting into a flat shape in the Tera context, which is what authors actually want when they write {{ post.title }}.

#Reading the content tree

The content walker is walkdir with a filter chain: keep files only, keep .md extension only, skip filenames starting with _ except _index.md, skip the posts/ directory (handled separately), skip any directory holding a _collection.toml. Sort the result lexically for deterministic output. Parse each file in parallel with rayon:

let parsed_pages: Vec<(Page, PathBuf, PathBuf)> = file_entries
    .par_iter()
    .map(|(path, relative)| {
        let page = self.parse_page(path, relative)?;
        Ok((page, path.clone(), relative.clone()))
    })
    .collect::<Result<Vec<_>>>()?;

For this site (a few dozen posts) the parallel parse is barely measurable, but it scales to hundreds of files without effort and the call site stays trivial.

The same module finds collections (every top-level directory under content/ with a _collection.toml is one), data files (data/**/*.{toml,yaml,json}, nested directories become nested maps), and static assets (static/**/* copied verbatim).

#Frontmatter

A bamboo content file looks like this:

+++
title = "My Post"
date = "2024-01-15"
tags = ["rust"]
+++

Body markdown.

TOML between +++, or YAML between ---. The parser scans line-by-line for the closing delimiter, hands the slice to toml::from_str or serde_yml::from_str, and gets back a HashMap<String, serde_json::Value>:

pub struct Frontmatter {
    #[serde(flatten)]
    pub raw: HashMap<String, Value>,
}

The Value map preserves every field the author wrote, known or not. Templates can reach for arbitrary {{ post.frontmatter.whatever }} keys, which is the only way I have found to let authors invent new conventions without editing the schema in the SSG.

Dates can also live in the filename. A file named 2024-01-15-hello.md parses out 2024-01-15 as the date and hello as the slug. The function in parsing.rs returns Option so a short filename like about.md falls through harmlessly:

pub fn parse_date_from_filename(filename: &str) -> Option<(String, String)> {
    let name = filename.strip_suffix(".md").unwrap_or(filename);
    let date_part = name.get(..10)?;
    ...
}

name.get(..10)? matters. If name is shorter than 10 bytes the slice is None and the function returns immediately rather than panicking on out-of-range indexing.

#Markdown rendering

pulldown-cmark is the parser. It emits an event stream of Start(Tag::Heading), Text, Code, Start(Tag::CodeBlock), etc. bamboo does not use the convenience html::push_html on the entire stream because two things need custom handling: code blocks and headings.

Code blocks need syntax highlighting. The stream looks like Start(CodeBlock(Fenced(lang))) → many Text events → End(CodeBlock). bamboo accumulates the text into a buffer and hands it to syntect at the end event:

Event::End(TagEnd::CodeBlock) => {
    in_code_block = false;
    let highlighted = if let Some(ref lang) = code_block_lang {
        self.syntax_set
            .find_syntax_by_token(lang)
            .map(|syntax| {
                highlighted_html_for_string(
                    &code_block_content,
                    &self.syntax_set,
                    syntax,
                    theme,
                )
                .unwrap_or_else(|_| escape_html(&code_block_content))
            })
            ...

Headings need anchor links so URLs can point at sections. The renderer slugifies the heading text, dedupes against a HashSet<String> of already-emitted ids so two ## Setup headings produce setup and setup-1, and emits:

<h2 id="setup"><a class="anchor" href="#setup">#</a>Setup</h2>

It also pushes every heading into a Vec<TocEntry> so templates can render a table of contents without re-parsing. The TOC is just (level, id, title) triples.

#Shortcodes

Plain markdown is fine for prose, awful for "embed a youtube video" or "show a notice box." Hugo-style shortcodes fill that gap:

{{< youtube id="dQw4w9WgXcQ" >}}

Two flavors. The inline form has no body; the block form wraps markdown content between an opening and matching closing tag:

inline:  {{< name arg="value" >}}

block:   {{% name arg="value" %}}
         body markdown
         {{% /name %}}

The shortcode processor runs before pulldown-cmark, scans the text for tags, replaces each with its rendered Tera template, and hands the result forward. Block bodies are themselves rendered through markdown first, so:

{{% note type="info" %}}
This is **emphasized**.
{{% /note %}}

emits a note containing <strong>emphasized</strong>.

The processor tracks fenced code blocks (triple-backtick and triple-tilde) so a shortcode tag inside a code block stays literal. It does not track inline single-backtick code spans, which means an inline-code example of an opening shortcode delimiter in prose still gets parsed as a real shortcode and triggers a hard error. I noticed this writing the draft of this post — the first build failed on a one-liner mention of the delimiter in flowing text, and the fix was to move every shortcode example into a fenced block instead. The scanning is hand-rolled because the shortcode pass runs before pulldown-cmark, so I cannot lean on its code-fence detection. Properly handling inline backticks belongs on the same list.

One special shortcode renders no template:

{{< ref "post-name.md" >}}

It resolves a source-path to the URL its content will deploy at. A registry is built during the initial content walk that maps source paths to final URLs, including permalink overrides. Internal links survive renames and custom permalinks through this indirection.

#Templates

Tera is the templating engine. Jinja2-style: inheritance, includes, filters, macros. Every page handed to the renderer gets a Tera context containing the full Site plus the page itself plus a few extras (pagination state, related posts, prev/next post for blog entries).

The built-in default theme is embedded into the binary with include_str!:

const DEFAULT_BASE_TEMPLATE: &str = include_str!("../themes/default/templates/base.html");
const DEFAULT_INDEX_TEMPLATE: &str = include_str!("../themes/default/templates/index.html");
const DEFAULT_PAGE_TEMPLATE: &str = include_str!("../themes/default/templates/page.html");
const DEFAULT_POST_TEMPLATE: &str = include_str!("../themes/default/templates/post.html");
...

A fresh Tera::default() gets each one stuffed in with add_raw_template. The upside is cargo install bamboo-cli produces a working SSG that needs no theme clone step. The downside is templates are not user-editable without dropping override files into the site's own templates/ directory. Site-level overrides are registered after the theme, so Tera's later-registration-wins semantics let them shadow the theme without any priority machinery.

Custom themes load from a directory pattern instead. The directory layout mirrors the embedded one: templates/, templates/partials/, templates/shortcodes/, and a static/ for theme assets.

#Custom filters

Four bamboo-specific Tera filters: slugify, reading_time, word_count, toc. They live in theme.rs::register_custom_filters and are plain closures:

tera.register_filter(
    "slugify",
    |value: &tera::Value, _args: &HashMap<String, tera::Value>| {
        let text = value.as_str().unwrap_or("");
        Ok(tera::Value::String(slugify(text)))
    },
);

The toc filter takes the heading list off a post or page and emits a nested <ul> indented by heading level. It is rendered inside templates with {{ post.toc | toc | safe }} (Tera's autoescape makes you opt in with safe for trusted HTML).

#Parallel rendering

After parsing, the engine renders pages, posts, and collections in parallel with rayon's par_iter. Each task writes into its own slot in dist/, so nothing needs coordination:

site.posts.par_iter().try_for_each(|post| {
    self.render_post(site, post, prev_post, next_post, output_dir)
})?;

Templates are registered up front and never mutated afterward. The renderer captures the Tera instance by reference into each closure. Tera is Send + Sync, so the parallel pass works without locks.

#Asset pipeline

Once HTML lands on disk, an optional pass walks output_dir and runs four kinds of post-processing:

Sass/SCSS compilation (grass, a pure-Rust Sass compiler)
CSS minification (lightningcss)
JS minification (minify-js)
HTML minification (minify-html)

Plus optional content-hash fingerprinting: each .css and .js output gets renamed from style.css to style.a3f8c91d.css, and every HTML and XML file gets walked to rewrite references. Cache headers can be aggressive (Cache-Control: public, max-age=31536000, immutable) after that without breaking deployments.

The order is load-bearing. Sass → minify CSS/JS → fingerprint → minify HTML. Fingerprinting before HTML minification means the rewrite step can still find unminified href="/style.css" attributes. Fingerprinting after would require parsing minified HTML.

#Responsive images

Opt-in. If [images] is set in bamboo.toml, the engine walks the output for images, resizes each to a configured set of widths (320, 640, 1024, 1920 by default), and writes each variant as both WebP and JPEG. Then it walks every HTML file and rewrites:

<img src="/photo.jpg">

into:

<picture>
  <source type="image/webp" srcset="/photo-320w.webp 320w, /photo-640w.webp 640w, ...">
  <source type="image/jpeg" srcset="/photo-320w.jpg 320w, /photo-640w.jpg 640w, ...">
  <img src="/photo.jpg">
</picture>

so the browser picks an appropriate resolution. The image crate does the decode and resize. The webp crate handles WebP encoding. The JPEG encoder comes from image::codecs::jpeg.

The variants that the pipeline already generated have a -{width}w.{ext} suffix, which is how the scanner avoids regenerating them on subsequent builds: any file whose stem ends in -320w / -640w / etc. is skipped as already-derived.

#RSS, Atom, sitemap, search

Each output is its own module taking the in-memory Site and writing a single file:

feeds::generate_rss → rss.xml
feeds::generate_atom → atom.xml
sitemap::generate_sitemap → sitemap.xml
search::generate_search_index → search-index.json

The XML files are produced with format! and a tiny xml::escape helper for the five special characters. For five entities and a handful of fixed templates this is fine. For anything with attributes-in-elements or namespaces I would reach for a real serializer.

The search index is a JSON array of { title, url, tags, date, excerpt, content } objects, where content is the rendered HTML run through strip_html_tags and truncated to 5000 characters per entry — small enough that the client-side Fuse.js search stays responsive even on sites with hundreds of posts. The default theme's search page pulls the file at request time and feeds it to Fuse.js client-side. There is no server-side search component because there is no server.

#Internal link validation

After the build, every HTML file under output_dir is grepped for href="..." by byte scan (the templates are trusted, so a regex is overkill). Anything that looks like an internal link gets resolved against the output tree:

/about/ checks output_dir/about/index.html
/style.css checks output_dir/style.css
https://example.com/about/ (when the base URL is https://example.com) gets normalized to /about/ and checked the same way

External links and #fragment-only links are skipped. Broken links print a warning. There is a link_check_ignore config field for prefixes that share the deployment domain with sibling projects on the same host that bamboo cannot see into.

#Incremental builds for `bamboo serve`

The dev server runs bamboo serve --open, watches the project tree with notify, debounces FS events by 300ms, then rebuilds. The rebuild is incremental.

Every tracked file (content/, data/, static/, templates/, bamboo.toml) gets SHA-256 hashed at the start of each build. The hashes are compared against the cache at .bamboo-cache/build-state.json. The diff produces a ChangeClassification:

pub enum ChangeClassification {
    Full,
    Targeted { changed_files: Vec<PathBuf> },
}

A change in bamboo.toml, any templates/*, or any deletion forces a full rebuild. A change to content/posts/2024-01-15-hello.md produces a Targeted classification, which expands to a HashSet<RenderTarget>:

pub enum RenderTarget {
    Page(String),
    Post(String),
    Collection(String),
    Pagination,
    AllTaxonomies,
    Feeds,
    Sitemap,
    SearchIndex,
    All,
}

A post edit invalidates pagination, feeds, sitemap, search, taxonomies, and the index page (the index lists recent posts). A page edit only invalidates sitemap and search. The render functions accept the target set as Option<&HashSet<RenderTarget>> and consult it before doing the work, so an incremental pass writes only the few files that actually changed.

This is finicky. The win is that editing a post and saving rebuilds in ~50ms instead of ~500ms on a site this size. With tower-livereload pushing a WebSocket message after each rebuild, the browser refreshes before I have tabbed back to it.

The error path is interesting. If a build fails mid-rebuild, the server stashes the error string in an Arc<Mutex<Option<String>>> and an Axum middleware intercepts the next page request, returning a styled error overlay with the message. As soon as the next successful build completes, the overlay clears and live reload fires. Editing markdown with a misformatted frontmatter no longer produces a 404 from the dev server; it produces a readable error screen.

#What I left out, and why

A real plugin system. bamboo's "plugin" is "drop a Tera template into templates/" plus "edit bamboo.toml." That covers everything I want from a personal blog. Zola has the same answer.

A proc-macro layer for typed frontmatter. The frontmatter is a HashMap<String, Value>. Templates speak through the map. A typed struct per content type would be tidier in pure-Rust terms, but every site has a different schema, and I do not want to compile a per-site bamboo binary. The flat map wins.

Configurable markdown extensions. pulldown-cmark options are hardcoded: tables, footnotes, strikethrough, task lists, heading attributes. I have never wanted any of these off.

A theme registry. Themes are directories. The built-in is embedded. I have not built infrastructure to publish or install third-party themes because nobody has asked.

#Dependencies

A few non-obvious choices:

pulldown-cmark: event-stream API, fast. The custom code block highlighting and TOC extraction are straightforward against the event stream and awkward against any "give me HTML" wrapper.
syntect: syntax highlighter, the same one bat uses. The default theme set covers enough languages.
Tera: Jinja2-style templating with filters and macros. Performance is fine for site sizes that fit in memory, which is "every site I care about."
grass: pure-Rust Sass compiler. The reason there is no Node.js anywhere in this toolchain.
rayon: parallel iterators. Used in content parsing, page/post/collection rendering, taxonomy term rendering, image resizing, and asset minification. Each loop writes to its own file so there is nothing to coordinate.
walkdir: recursive directory walks. Filter-then-collect-then-parallel is the pattern everywhere.
lightningcss / minify-html / minify-js: the three minifiers. lightningcss in particular is faster and produces tighter output than the JS-ecosystem alternatives.
axum + tower-livereload + notify: the live-reload dev server stack. notify watches the disk, the rebuild thread fires through an mpsc::channel, a tokio::sync::broadcast fans out reload signals to connected WebSocket clients.

#Closing

The whole repo is at matthewjberger/bamboo. The source for this site is at matthewjberger/articles. If you want a ready-to-deploy starter with GitHub Pages CI, matthewjberger/bamboo-template is one gh repo create --template away.

Matthew Berger

Build your own ECS (part 3), change detection, events, tags, and commands

#Change detection by watermark

#Events

#Sparse-set tags

#Command buffers

#Resources

#A trivial schedule

#What we built

#Where the abstractions stop being free

#Where the engine goes from here

#The full file

Build your own ECS (part 2), structural change and queries

#Adding a component means moving the entity

#add_components and remove_components

#Queries

#Making the migration fast

#Making the query fast

#What we built

#What is missing

#The full file

Build your own ECS (part 1), archetype storage

#Why this layout

#A note on style before the code starts

#Entity handles

#Allocating entities

#Components are just structs

#Why HashMap is the wrong storage

#Grouping by component set

#The archetype table

#Routing entities to their slots

#The World

#Spawning

#Reading and writing components

#Despawning

#What we built

#What is missing

#The full file

Building a static site generator in Rust

#What a static site generator actually is

#Shape: builder, renderer, CLI

#Reading the content tree

#Frontmatter

#Markdown rendering

#Shortcodes

#Templates

#Custom filters

#Parallel rendering

#Asset pipeline

#Responsive images

#RSS, Atom, sitemap, search

#Internal link validation

#Incremental builds for bamboo serve

#What I left out, and why

#Dependencies

#Closing

#Incremental builds for `bamboo serve`