rin · data-oriented design · build an ECS
rin-dod
Build an archetype entity component system from nothing, in fifteen chapters you read, compile and step through: from an entity as a number to a game running on it.
Data-oriented design in rin, in two courses and a dojo:
- Build an ECS. Start here. Fifteen chapters that build an archetype entity component system from nothing, in the order of its name: entities, then components, then systems. It ends with a game running on it. Each chapter is a program that compiles on its own, the output it must print, and a guide that builds it step by step.
- Advanced: layout and SIMD. Seven lessons, for after. They cover what a loop fetches from memory, aliasing and
noalias, what the compiler vectorizes and how to check it in the assembly, and which layout suits which access pattern. - The dojo. The thirteen building blocks an ECS is made of, taken out on their own: a playground to watch each one work and break, and a kata to write it yourself, timed, against a harness, until your hands know it.
There is no renderer, no window and no library: only data, and the loops that stream it.
All of it is written in rin's safe subset. rin checks every program with its memory-safety rules on, the default on its regions branch, and every program prints its expected output and runs clean under AddressSanitizer and UBSan (tools/check.py). Memory is regions: level: region;, fields that say *vec<T> in level, lookups that return option<*T>. What the checker cannot see stays in small unsafe { } blocks and proc[unsafe] kernels: the type-erased byte columns, the noalias promises, and reading argv. The main branch keeps the earlier unsafe version of the course.
Build an ECS
| chapter | about | |
|---|---|---|
| 1 | entities | |
| 1.1 | an entity is a number | ids, components as arrays indexed by them, a system as a loop |
| 1.2 | despawning | a free list, reused numbers, and the stale reference they cause |
| 1.3 | generations | handles that know when they are stale |
| 1.4 | the entity index | slots, packed rows, swap-remove, kayan_verify |
| 2 | components | |
| 2.1 | components and signatures | component ids, a registry of reflect records, 64-bit signatures |
| 2.2 | columns | one array of any component, as bytes, viewed as a typed slice |
| 2.3 | a table | the entity index and the columns, joined |
| 2.4 | archetypes | a table for every combination, found by signature |
| 2.5 | typed access | kayan_get<T>, kayan_set<T>; keep handles, borrow pointers |
| 2.6 | adding and removing | moving between tables, and the archetype graph |
| 2.7 | inspecting | any entity, field by field, from reflection |
| 3 | systems | |
| 3.1 | queries | the tables a system runs over, matched once and kept |
| 3.2 | systems | kernels, systems, a schedule, a frame |
| 3.3 | commands | why a system cannot change structure, and the buffer that can |
| 3.4 | a game | turrets, waves, bullets and fire, on everything above |
Read each chapter's guide first. It explains the new idea and builds its code step by step, and every sample in it is copied from the chapter's program. Then open the program: it holds everything built so far, with the chapter's new code near the end, so any chapter can be opened in a debugger and stepped through as a whole ECS. Run it and read the output against the code. Each guide ends with exercises, most of them "break this on purpose and watch what happens".
Each chapter's file starts from the previous chapter's, so diffing two neighbours shows exactly what a chapter changed:
git diff --no-index ecs/src/2.components/2.5.typed_access/typed_access.rin ecs/src/2.components/2.6.add_remove/add_remove.rin
The ECS is called kayan, after the C sketch it grew out of.
The dojo
| block | what it is | |
|---|---|---|
| 1 | free list and generations | handles that are safe to keep after what they name is gone |
| 2 | swap-remove and the back-reference | values packed with no holes, found by id: a sparse set |
| 3 | the slot map | 1 and 2 together: handles into packed values |
| 4 | a type-erased column | an array of any type as bytes, viewed as a typed slice |
| 5 | a command buffer | structural changes recorded now and applied later |
| 6 | bitsets | a set of components in a u64: has, all of, none of |
| 7 | an arena | push, push, clear: memory for things that die together |
| 8 | index, not pointer | links that stay right when their storage moves |
| 9 | interning | find or make, by key: a hash table that hands out dense ids |
| 10 | memoized edges | a slow lookup done once, then remembered on the edge |
| 11 | a reflection walk | reading, writing and comparing any value through its type's record |
| 12 | a watermark | looking only at what was added since last time |
| 13 | a dispatch table | behaviour as data: the schedule as a list of functions |
python tools/dojo.py your reps, and which blocks are due
python tools/dojo.py play slot_map watch it: the playground
python tools/dojo.py start slot_map 2 write it: a kata from level 2, timed
python tools/dojo.py slot_map check it against the harness
Each block has three levels (one proc, every proc, a blank page) and a few variations, and the runner keeps your times and spaces your reps out. dojo/README.md has the rest. Each course chapter says which blocks it is built from.
Advanced: layout and SIMD
| lesson | about | |
|---|---|---|
| 0 | from kayan | what was kept from the C kayan, what changed, and why |
| 1 | aos vs soa | the same particles in three layouts, what a loop fetches, and what SIMD needs |
| 2 | archetype table | one table: columns, rows, spawn, swap-remove, systems over slices |
| 3 | many archetypes | one table type for every archetype: ids, signatures, queries |
| 4 | entity handles | generational handles, slots, moving between archetypes |
| 5 | inspector | printing and editing any entity from reflection records |
| 6 | kayan | a small game, and deferring structural changes to a command buffer |
| 7 | access patterns | four layouts under streams and lookups, and how an ECS decides |
Lessons 1 and 7 are about memory, aliasing and the code the compiler makes. Lessons 2 to 6 build an ECS again, compressed into five steps, so read them as a recap of the course. docs/README.md has more.
Build and run
Needs a rin compiler built from rin's regions branch (rin.exe with its std beside it, through RIN_HOME or PATH), CMake, Ninja, clang-cl and the Windows SDK.
rin build
build\cmake\ids.exe
Every chapter and lesson is its own program, named after its file, and so is each dojo playground. Or compile one by hand: rin writes C, and any C compiler does the rest, with rin's folder and its std on the include path:
rin compile ecs/src/2.components/2.4.archetypes/archetypes.rin -o archetypes.c
clang -std=c11 -g -I <rin> -I <rin>/std archetypes.c -o archetypes
On Windows that last line warns about fopen; -D_CRT_SECURE_NO_WARNINGS quiets it, as build.rin does.
rin compile refuses a program that breaks a safety rule, so a program that builds has been checked. rin check file.rin does the checking alone.
python tools/check.py finds rin through RIN, then RIN_HOME, then PATH, and its std beside it. It builds every program with clang and warnings as errors, runs it, and compares what it prints with its .expected file. --sanitize adds AddressSanitizer and UBSan, --bless rewrites the expected files after a deliberate change, and a word filters the programs. tools/check_snippets.py checks that every sample in the ECS guides and the dojo's READMEs is in its program:
python tools/check.py ecs
python tools/check.py kayan --sanitize
python tools/check_snippets.py
rin itself builds on Windows: its arena runtime, in extern/haikal, calls VirtualAlloc. The generated C is portable, and the whole course has passed on Linux, gcc and clang, with a test build of rin over a stand-in for those calls, and -Wno-sign-compare added to tools/check.py's flags: an enum compared with its member warns there, harmlessly.
Reading it in a browser
Open website/index.html. It has every chapter's and lesson's guide, its source with line numbers, and the output it must print, with an index on the left. It is generated from the files in this repository, so after changing a program or a guide run python tools/site.py to bring it up to date.
Layout
ecs/
src/
1.entities/
1.1.ids/
ids.rin the program
ids.expected what it prints
...
2.components/
3.systems/
README.md what the safe subset asks of each part
docs/ a guide a chapter: steps, output, exercises
README.md the course on its own
src/ the advanced lessons, one folder each
1.aos_vs_soa/
aos_vs_soa.rin the program
aos_vs_soa.expected what it prints
...
includes/
kayan.rin lesson 4's world, used by lessons 5 and 6
docs/ the advanced lessons' guides
dojo/
1.free_list/ a block: README, playground, harness, levels, solution
...
README.md how the dojo works
dojo.rin what every harness shares
website/
index.html everything as one page: guide, code, output
template.html its layout and styles
tools/
check.py build, run and compare every program
check_snippets.py check every sample in the guides and dojo READMEs is in its program
site.py rebuild website/index.html from the repo
dojo.py start, run and time the dojo's katas
build.rin the program list, for `rin build`
task-8910.md the backlog: lessons 8, 9 and 10
License
MIT. See LICENSE.
Build an ECS · part 1, entities · 1.1
An entity is a number
ecs/src/1.entities/1.1.ids/ids.rin
An ECS has three parts: Entities, Components and Systems. This chapter builds all three in the simplest way that works, about a hundred lines, so you can see the whole shape before any of it gets clever. The chapters after this one keep the shape and fix its costs, one at a time.
The idea
A game is full of things: a turret, drones, bullets. An ECS splits each thing into three parts:
- An entity names the thing. It is just a number. It has no data and no behaviour of its own.
- Components are the thing's data: a position, a velocity, health. Each component is plain data, a small struct or a number.
- Systems are the behaviour: loops that run over every entity that has the components they need. Movement runs over everything with a position and a velocity.
A turret is entity 0 with a position and health. A drone is entity 1 with a position, a velocity and health. Nothing says "turret" or "drone" anywhere; what a thing is comes from which components it has.
Step 1: the entity
kayan_entity: struct = {
index: u32;
}
An entity is a number, so why the struct? Because a bare u32 can be mixed up with any other u32. A proc that takes a kayan_entity cannot be handed a count or a row index by mistake, and the struct costs nothing at run time: it compiles to the same 4 bytes. It also gives the number room to grow, and in 1.3 it does.
Step 2: the world
Where do components live? The simplest answer: one array per component, with one slot for every possible entity, indexed by the entity's number. Beside each array, a flag per entity saying whether it has that component.
kayan_world: struct = {
count: u32;
positions: [8]vec2;
has_position: [8]b32;
velocities: [8]vec2;
has_velocity: [8]b32;
healths: [8]f32;
has_health: [8]b32;
}
Look at the shape. This is lesson 1's struct of arrays: every position together, every velocity together. An entity's data is spread across the arrays, all at the same index. Entity 1's position is positions[1] and its health is healths[1].
Step 3: spawning
A new entity is the next number:
kayan_spawn: proc(w: *kayan_world) -> kayan_entity = {
if (w*.count == 8) {
printfmt("the world is full\n");
exit(1);
}
e: kayan_entity = { .index = w*.count };
w*.count += 1;
return e;
}
It has no components yet: every flag for its number is still false.
Step 4: giving it components
Giving an entity a component is writing the value at its number and raising the flag:
set_position: proc(w: *kayan_world, e: kayan_entity, p: vec2) -> void = {
w*.positions[e.index] = p;
w*.has_position[e.index] = true;
}
set_velocity and set_health are the same, one per component. Writing one proc per component type is tedious, and Part 2 replaces all of them with a single generic kayan_set<T>. For now, plain is better: you can see exactly which bytes get written.
Building a drone is spawning a number and attaching data to it:
drone_a: kayan_entity = kayan_spawn(w&);
set_position(w&, drone_a, { .x = 10.0, .y = 0.0 });
set_velocity(w&, drone_a, { .x = -1.0, .y = 0.5 });
set_health(w&, drone_a, 100.0);
Step 5: a system
A system is a loop. Movement runs over everything with a position and a velocity:
movement_system: proc(w: *kayan_world, dt: f32) -> u32 = {
moved: u32 = 0;
for (i: u32 = 0; i < w*.count; i += 1) {
if (w*.has_position[i] && w*.has_velocity[i]) {
w*.positions[i].x += w*.velocities[i].x * dt;
w*.positions[i].y += w*.velocities[i].y * dt;
moved += 1;
}
}
return moved;
}
It doesn't know about turrets, drones or bullets. It knows about positions and velocities, and anything that has both moves, including bullets that didn't exist when the system was written. That's the ECS idea in one proc: behaviour attaches to data, not to types of thing.
Checked as written
This course is written in rin's safe subset, which is what rin checks by default, and nothing in this chapter needed anything special to pass. The world is a value on main's stack. Procs borrow it as *kayan_world, and a pointer parameter can't be kept anywhere that outlives the call. Every positions[i] is bounds checked, in every build. Plain data in fixed arrays is the easy case for the checker, and it stays this easy until Part 2 needs memory that grows.
Run it
spawned turret 0, drones 1 and 2, bullet 3
entity 0 position ( 0.00, 0.00) velocity - health 500
entity 1 position (10.00, 0.00) velocity (-1.00, 0.50) health 100
entity 2 position ( 0.00, 10.00) velocity ( 0.50, -1.00) health 100
entity 3 position ( 0.00, 0.00) velocity ( 4.00, 4.00) health -
step 1: the movement system looked at 4 entities and moved 3
step 2: the movement system looked at 4 entities and moved 3
...
every component array has room for 8; 4 entities exist
positions hold 4 values, velocities 3, healths 3
The turret has no velocity, so movement skips it. The bullet has no health, and nothing cares.
What this costs
It works, and every later chapter keeps this shape: entities are numbers, components are arrays of plain data, systems are loops. But look at the last two lines of the output and at the loop:
- Every array is as long as the most entities the world can hold. Velocity has room for 8 and holds 3. A game with 10,000 entities and 50 component types would have 50 arrays of 10,000, mostly empty.
- Every system visits every entity and checks flags to skip the ones that don't apply. Movement looked at 4 and moved 3. With 10,000 entities of which 50 move, it would look at 10,000 to move 50.
- Nothing can be destroyed. There's no despawn, and numbers only go up.
The next chapter fixes 3. Chapter 1.4 packs the data so systems stop skipping, and Part 2 fixes 1 and 2 for good with tables.
Try
- Give the turret a velocity and run it. Nothing else changes: the turret moves because it now has the data movement wants.
- Add a
lifetime: [8]f32component with its flags, and anaging_systemthat counts it down. How many places did you have to touch? - Spawn a ninth entity. Read the error, then think about what a game would need instead of a fixed 8.
1// 1.1 An entity is a number2//3// An entity names one thing in the game: a drone, a bullet, the turret. It has4// no data of its own and no behaviour. It is a number, and everything about5// the thing it names is stored somewhere else, in arrays, at that number.6//7// This chapter builds the simplest ECS there is: one array per component, as8// long as the most entities the world can hold, and a flag beside each array9// saying which entities have that component. A system is a loop over every10// entity that checks the flags it needs.11//12// It works, and it already has the shape every later chapter keeps: entities13// are ids, components are plain data in arrays, systems are loops. It also14// wastes memory and time in ways the next chapters remove one at a time.15 16import "std/print.rin"17 18vec2: struct = {19 x: f32;20 y: f32;21}22 23// An entity is a number. Wrapping it in a struct costs nothing at run time and24// keeps it from being mixed up with any other u32: a proc that takes a25// kayan_entity cannot be handed a count or a row by mistake.26kayan_entity: struct = {27 index: u32;28}29 30// The world: how many entities exist, and for each component an array of31// values and an array of flags, both indexed by the entity's number. Every32// array has room for 8, the most entities this world can hold.33kayan_world: struct = {34 count: u32;35 positions: [8]vec2;36 has_position: [8]b32;37 velocities: [8]vec2;38 has_velocity: [8]b32;39 healths: [8]f32;40 has_health: [8]b32;41}42 43// A new entity is the next number. It has no components yet.44kayan_spawn: proc(w: *kayan_world) -> kayan_entity = {45 if (w*.count == 8) {46 printfmt("the world is full\n");47 exit(1);48 }49 e: kayan_entity = { .index = w*.count };50 w*.count += 1;51 return e;52}53 54// Giving an entity a component is writing the value at the entity's number and55// raising the flag beside it.56set_position: proc(w: *kayan_world, e: kayan_entity, p: vec2) -> void = {57 w*.positions[e.index] = p;58 w*.has_position[e.index] = true;59}60 61set_velocity: proc(w: *kayan_world, e: kayan_entity, v: vec2) -> void = {62 w*.velocities[e.index] = v;63 w*.has_velocity[e.index] = true;64}65 66set_health: proc(w: *kayan_world, e: kayan_entity, points: f32) -> void = {67 w*.healths[e.index] = points;68 w*.has_health[e.index] = true;69}70 71// A system: everything with a position and a velocity moves. It visits every72// entity in the world and skips the ones that lack either component. Returns73// how many it moved.74movement_system: proc(w: *kayan_world, dt: f32) -> u32 = {75 moved: u32 = 0;76 for (i: u32 = 0; i < w*.count; i += 1) {77 if (w*.has_position[i] && w*.has_velocity[i]) {78 w*.positions[i].x += w*.velocities[i].x * dt;79 w*.positions[i].y += w*.velocities[i].y * dt;80 moved += 1;81 }82 }83 return moved;84}85 86print_world: proc(w: *kayan_world) -> void = {87 for (i: u32 = 0; i < w*.count; i += 1) {88 printfmt(" entity {}", i);89 if (w*.has_position[i]) {90 printfmt(" position ({:5.2}, {:5.2})", cast(w*.positions[i].x, f64), cast(w*.positions[i].y, f64));91 } else {92 printfmt(" position - ");93 }94 if (w*.has_velocity[i]) {95 printfmt(" velocity ({:5.2}, {:5.2})", cast(w*.velocities[i].x, f64), cast(w*.velocities[i].y, f64));96 } else {97 printfmt(" velocity - ");98 }99 if (w*.has_health[i]) {100 printfmt(" health {:.0}", cast(w*.healths[i], f64));101 } else {102 printfmt(" health -");103 }104 printfmt("\n");105 }106}107 108// How many of the 8 slots of one component's array hold a value.109used: proc(flags: [8]b32) -> u32 = {110 n: u32 = 0;111 for (i: u32 = 0; i < 8; i += 1) {112 if (flags[i]) {113 n += 1;114 }115 }116 return n;117}118 119main: proc() -> i32 = {120 w: kayan_world = {};121 122 turret: kayan_entity = kayan_spawn(w&);123 set_position(w&, turret, { .x = 0.0, .y = 0.0 });124 set_health(w&, turret, 500.0);125 126 drone_a: kayan_entity = kayan_spawn(w&);127 set_position(w&, drone_a, { .x = 10.0, .y = 0.0 });128 set_velocity(w&, drone_a, { .x = -1.0, .y = 0.5 });129 set_health(w&, drone_a, 100.0);130 131 drone_b: kayan_entity = kayan_spawn(w&);132 set_position(w&, drone_b, { .x = 0.0, .y = 10.0 });133 set_velocity(w&, drone_b, { .x = 0.5, .y = -1.0 });134 set_health(w&, drone_b, 100.0);135 136 bullet: kayan_entity = kayan_spawn(w&);137 set_position(w&, bullet, { .x = 0.0, .y = 0.0 });138 set_velocity(w&, bullet, { .x = 4.0, .y = 4.0 });139 140 printfmt("spawned turret {}, drones {} and {}, bullet {}\n",141 turret.index, drone_a.index, drone_b.index, bullet.index);142 print_world(w&);143 144 for (step: u32 = 1; step <= 2; step += 1) {145 moved: u32 = movement_system(w&, 0.5);146 printfmt("step {}: the movement system looked at {} entities and moved {}\n", step, w.count, moved);147 }148 print_world(w&);149 150 printfmt("every component array has room for 8; {} entities exist\n", w.count);151 printfmt(" positions hold {} values, velocities {}, healths {}\n",152 used(w.has_position), used(w.has_velocity), used(w.has_health));153 return 0;154}What ids prints. tools/check.py fails the chapter if a single byte differs.
spawned turret 0, drones 1 and 2, bullet 3
entity 0 position ( 0.00, 0.00) velocity - health 500
entity 1 position (10.00, 0.00) velocity (-1.00, 0.50) health 100
entity 2 position ( 0.00, 10.00) velocity ( 0.50, -1.00) health 100
entity 3 position ( 0.00, 0.00) velocity ( 4.00, 4.00) health -
step 1: the movement system looked at 4 entities and moved 3
step 2: the movement system looked at 4 entities and moved 3
entity 0 position ( 0.00, 0.00) velocity - health 500
entity 1 position ( 9.00, 0.50) velocity (-1.00, 0.50) health 100
entity 2 position ( 0.50, 9.00) velocity ( 0.50, -1.00) health 100
entity 3 position ( 4.00, 4.00) velocity ( 4.00, 4.00) health -
every component array has room for 8; 4 entities exist
positions hold 4 values, velocities 3, healths 3
Build an ECS · part 1, entities · 1.2
Despawning, and reusing numbers
ecs/src/1.entities/1.2.despawn/despawn.rin
In the dojo: 1. free list and generations, on its own, to watch and to practise.
Games destroy things constantly: a bullet hits, a drone dies, an effect fades. 1.1 could only add. This chapter adds kayan_despawn, and with it the first real problem in ECS design: what happens to an entity's number when the entity is gone?
Step 1: a free list
If despawned numbers were never reused, a game firing one bullet a frame would use up 8 numbers in 8 frames and 4 billion in a couple of years of play. More to the point, every array would keep growing. So despawned numbers go on a free list, and spawning takes from it first:
kayan_world: struct = {
count: u32;
alive: [8]b32;
free: [8]u32;
free_count: u32;
count is now how many numbers have ever been handed out, not how many entities exist; the numbers on the free list are the gaps. alive says which numbers are in use.
Step 2: spawning reuses
kayan_spawn: proc(w: *kayan_world) -> kayan_entity = {
e: kayan_entity = {};
if (w*.free_count > 0) {
w*.free_count -= 1;
e.index = w*.free[w*.free_count];
} else {
if (w*.count == 8) {
printfmt("the world is full\n");
exit(1);
}
e.index = w*.count;
w*.count += 1;
}
w*.alive[e.index] = true;
return e;
}
The free list is a stack: the last number freed is the first reused.
Step 3: despawning
kayan_despawn: proc(w: *kayan_world, e: kayan_entity) -> void = {
w*.alive[e.index] = false;
w*.has_position[e.index] = false;
w*.has_velocity[e.index] = false;
w*.has_health[e.index] = false;
w*.free[w*.free_count] = e.index;
w*.free_count += 1;
}
Despawning lowers every flag, so the entity's old components can't leak into whatever reuses the number. The values stay in the arrays, but with the flags down nobody reads them. Systems need no change: a despawned entity has no flags, so movement skips it without knowing anything about despawning.
Step 4: the bug
Now the problem. Things in a game refer to other things: a turret aims at a drone, a missile chases a target, a health bar follows a unit. With entities as numbers, a reference is a number:
target: kayan_entity = drone_a;
The program then destroys the drone, and the turret fires. The bullet is a new entity, and it takes the first free number, the drone's:
the turret targets entity 1, a drone
...
entity 1 is destroyed and despawned
entity 1 (free)
the turret fires: the bullet is entity 1
...
the turret aims at entity 1, at (2.00, 2.00), moving (4.00, 4.00)
that is the turret's own bullet, not the drone: the number was reused
Nothing told the turret its target died, and nothing could: references can be anywhere, in any number of places. The number in target is still a valid number, it just names a different entity now. No crash, no error, just a turret shooting at its own bullet. In a real game this is the bug where a homing missile turns around mid-flight, or a health bar jumps to another unit.
Notice that rin's checker has no complaint about any of this, and it shouldn't. Every read is of a real element of a real array, so the program is memory safe. It is also wrong. Memory safety says a reference can't reach bad memory. It says nothing about whether the reference still means what you think it means. That part is a design problem, and the rest of Part 1 solves it with data.
Run it
the turret targets entity 1, a drone
entity 0 position ( 0.00, 0.00) velocity - health 500
entity 1 position (10.00, 0.00) velocity (-1.00, 0.50) health 100
entity 2 position ( 0.00, 10.00) velocity ( 0.50, -1.00) health 100
entity 1 is destroyed and despawned
entity 0 position ( 0.00, 0.00) velocity - health 500
entity 1 (free)
entity 2 position ( 0.00, 10.00) velocity ( 0.50, -1.00) health 100
the turret fires: the bullet is entity 1
...
Why not just...
- ...never reuse numbers? The arrays would grow forever, and every system would walk an ever-longer list of dead numbers.
- ...find and clear every reference on despawn? You'd need to know where every reference lives. That's exactly what an ECS is trying to avoid: things refer to each other loosely, by number.
- **...check
alive[target.index]?** It saystrue: the number is alive. It's alive as a different entity, and nothing can tell the two apart.
The next chapter tells them apart with one extra number per entity.
Try
- Despawn the bullet too, then spawn two more entities. Which numbers do they get, and in what order? Why?
- Comment out the lines that lower the flags in
kayan_despawnand run it. What does the new bullet have that it shouldn't? - Despawn
drone_atwice. What does the free list look like after? Spawn twice more and see why that is a problem.
1// 1.2 Despawning, and reusing numbers2//3// Games destroy things all the time: a bullet hits, a drone dies. 1.1 could4// only add. Here an entity can be despawned, and its number goes on a free5// list, so the next spawn takes it instead of growing the world. Without that,6// a game that fires a bullet a frame would run out of numbers in seconds.7//8// Reuse has a cost, and this program walks straight into it. Something that9// kept an entity's number, the turret's target, still holds it after the10// entity is gone. When the number is handed to a new entity, the old11// reference silently names the new one. 1.3 fixes that.12 13import "std/print.rin"14 15vec2: struct = {16 x: f32;17 y: f32;18}19 20kayan_entity: struct = {21 index: u32;22}23 24// `alive` says which numbers are in use. `free` holds the numbers of despawned25// entities, `free_count` of them, ready to be handed out again.26kayan_world: struct = {27 count: u32;28 alive: [8]b32;29 free: [8]u32;30 free_count: u32;31 positions: [8]vec2;32 has_position: [8]b32;33 velocities: [8]vec2;34 has_velocity: [8]b32;35 healths: [8]f32;36 has_health: [8]b32;37}38 39// A free number is reused before a new one is made. A reused number must start40// with no components: whatever the last entity at this number had was cleared41// when it was despawned.42kayan_spawn: proc(w: *kayan_world) -> kayan_entity = {43 e: kayan_entity = {};44 if (w*.free_count > 0) {45 w*.free_count -= 1;46 e.index = w*.free[w*.free_count];47 } else {48 if (w*.count == 8) {49 printfmt("the world is full\n");50 exit(1);51 }52 e.index = w*.count;53 w*.count += 1;54 }55 w*.alive[e.index] = true;56 return e;57}58 59// The entity's components are dropped by lowering their flags, and its number60// goes on the free list. The values stay in the arrays, unread.61kayan_despawn: proc(w: *kayan_world, e: kayan_entity) -> void = {62 w*.alive[e.index] = false;63 w*.has_position[e.index] = false;64 w*.has_velocity[e.index] = false;65 w*.has_health[e.index] = false;66 w*.free[w*.free_count] = e.index;67 w*.free_count += 1;68}69 70set_position: proc(w: *kayan_world, e: kayan_entity, p: vec2) -> void = {71 w*.positions[e.index] = p;72 w*.has_position[e.index] = true;73}74 75set_velocity: proc(w: *kayan_world, e: kayan_entity, v: vec2) -> void = {76 w*.velocities[e.index] = v;77 w*.has_velocity[e.index] = true;78}79 80set_health: proc(w: *kayan_world, e: kayan_entity, points: f32) -> void = {81 w*.healths[e.index] = points;82 w*.has_health[e.index] = true;83}84 85// Despawned entities have no flags left, so the system skips them without86// knowing anything about despawning.87movement_system: proc(w: *kayan_world, dt: f32) -> u32 = {88 moved: u32 = 0;89 for (i: u32 = 0; i < w*.count; i += 1) {90 if (w*.has_position[i] && w*.has_velocity[i]) {91 w*.positions[i].x += w*.velocities[i].x * dt;92 w*.positions[i].y += w*.velocities[i].y * dt;93 moved += 1;94 }95 }96 return moved;97}98 99print_world: proc(w: *kayan_world) -> void = {100 for (i: u32 = 0; i < w*.count; i += 1) {101 if (!w*.alive[i]) {102 printfmt(" entity {} (free)\n", i);103 continue;104 }105 printfmt(" entity {}", i);106 if (w*.has_position[i]) {107 printfmt(" position ({:5.2}, {:5.2})", cast(w*.positions[i].x, f64), cast(w*.positions[i].y, f64));108 } else {109 printfmt(" position - ");110 }111 if (w*.has_velocity[i]) {112 printfmt(" velocity ({:5.2}, {:5.2})", cast(w*.velocities[i].x, f64), cast(w*.velocities[i].y, f64));113 } else {114 printfmt(" velocity - ");115 }116 if (w*.has_health[i]) {117 printfmt(" health {:.0}", cast(w*.healths[i], f64));118 } else {119 printfmt(" health -");120 }121 printfmt("\n");122 }123}124 125main: proc() -> i32 = {126 w: kayan_world = {};127 128 turret: kayan_entity = kayan_spawn(w&);129 set_position(w&, turret, { .x = 0.0, .y = 0.0 });130 set_health(w&, turret, 500.0);131 132 drone_a: kayan_entity = kayan_spawn(w&);133 set_position(w&, drone_a, { .x = 10.0, .y = 0.0 });134 set_velocity(w&, drone_a, { .x = -1.0, .y = 0.5 });135 set_health(w&, drone_a, 100.0);136 137 drone_b: kayan_entity = kayan_spawn(w&);138 set_position(w&, drone_b, { .x = 0.0, .y = 10.0 });139 set_velocity(w&, drone_b, { .x = 0.5, .y = -1.0 });140 set_health(w&, drone_b, 100.0);141 142 // The turret remembers what it is aiming at, by number.143 target: kayan_entity = drone_a;144 printfmt("the turret targets entity {}, a drone\n", target.index);145 print_world(w&);146 147 printfmt("entity {} is destroyed and despawned\n", drone_a.index);148 kayan_despawn(w&, drone_a);149 print_world(w&);150 151 // The turret fires. The bullet takes the first free number.152 bullet: kayan_entity = kayan_spawn(w&);153 set_position(w&, bullet, { .x = 0.0, .y = 0.0 });154 set_velocity(w&, bullet, { .x = 4.0, .y = 4.0 });155 printfmt("the turret fires: the bullet is entity {}\n", bullet.index);156 _ = movement_system(w&, 0.5);157 print_world(w&);158 159 // Nothing told the turret its target died. Its number now names the bullet.160 printfmt("the turret aims at entity {}, at ({:.2}, {:.2}), moving ({:.2}, {:.2})\n", target.index,161 cast(w.positions[target.index].x, f64), cast(w.positions[target.index].y, f64),162 cast(w.velocities[target.index].x, f64), cast(w.velocities[target.index].y, f64));163 printfmt(" that is the turret's own bullet, not the drone: the number was reused\n");164 return 0;165}What despawn prints. tools/check.py fails the chapter if a single byte differs.
the turret targets entity 1, a drone
entity 0 position ( 0.00, 0.00) velocity - health 500
entity 1 position (10.00, 0.00) velocity (-1.00, 0.50) health 100
entity 2 position ( 0.00, 10.00) velocity ( 0.50, -1.00) health 100
entity 1 is destroyed and despawned
entity 0 position ( 0.00, 0.00) velocity - health 500
entity 1 (free)
entity 2 position ( 0.00, 10.00) velocity ( 0.50, -1.00) health 100
the turret fires: the bullet is entity 1
entity 0 position ( 0.00, 0.00) velocity - health 500
entity 1 position ( 2.00, 2.00) velocity ( 4.00, 4.00) health -
entity 2 position ( 0.25, 9.50) velocity ( 0.50, -1.00) health 100
the turret aims at entity 1, at (2.00, 2.00), moving (4.00, 4.00)
that is the turret's own bullet, not the drone: the number was reused
Build an ECS · part 1, entities · 1.3
Generations
ecs/src/1.entities/1.3.generations/generations.rin
In the dojo: 1. free list and generations, on its own, to watch and to practise.
1.2 ended with a turret aiming at its own bullet. The number it remembered was reused, and nothing could tell the old entity from the new one. The fix is small and used by nearly every ECS: count each number's lives.
The idea
Give each number a generation. Entity 1 in its first life is 1:1. When it's despawned, number 1's generation goes up to 2, so the next entity to reuse it is 1:2. A reference carries both numbers:
kayan_entity: struct = {
index: u32;
generation: u32;
}
A reference with a generation is called a handle. The turret's target is the handle 1:1. After the drone dies and the bullet becomes 1:2, the target still says 1:1, and 1:1 != 1:2. One comparison catches the bug.
Step 1: the world keeps generations
kayan_world: struct = {
count: u32;
generations: [8]u32;
free: [8]u32;
free_count: u32;
generations[i] is the life number i is in now, or the life it'll be in next if i is free. 1.2's alive array is gone, because the generations already say it:
kayan_alive: proc(w: *kayan_world, e: kayan_entity) -> b32 = {
return e.index < w*.count && w*.generations[e.index] == e.generation;
}
A handle is alive when its generation is the one its number is in now.
Step 2: despawn bumps the generation
kayan_despawn: proc(w: *kayan_world, e: kayan_entity) -> kayan_status = {
if (!kayan_alive(w, e)) {
return kayan_status.dead;
}
w*.has_position[e.index] = false;
w*.has_velocity[e.index] = false;
w*.has_health[e.index] = false;
w*.generations[e.index] += 1;
w*.free[w*.free_count] = e.index;
w*.free_count += 1;
return kayan_status.ok;
}
The generation goes up at despawn, not at the next spawn. So a free number already holds a generation that no handle has been given yet, and no handle can match it. That's why there's no separate alive flag: "free" and "every existing handle is stale" are the same fact.
Despawning a dead handle now returns dead and changes nothing. In 1.2, despawning twice put the same number on the free list twice, and two later entities would have shared it.
Step 3: a status you can't ignore
kayan_status: enum[must_use] = {
ok,
dead,
}
enum[must_use] makes ignoring the result a compile error. When you mean to ignore it, you say so with _ = kayan_despawn(...). The program does that where it knows the handle is alive.
Step 4: generations start at 1
e.index = w*.count;
w*.count += 1;
w*.generations[e.index] = 1;
A brand-new number starts at generation 1, never 0. So the zeroed handle, 0:0, can never be alive, and {} is a free "no entity" value. A turret with nothing to aim at holds {}, and checking it needs no special case.
Step 5: every access checks
get_position: proc(w: *kayan_world, e: kayan_entity) -> option<*vec2> = {
if (!kayan_alive(w, e) || !w*.has_position[e.index]) {
return null;
}
return w*.positions[e.index]&;
}
Reading through a handle checks it first. A dead handle gives null instead of whatever data now sits at the number.
The return type says so. A plain *vec2 is a promise that the pointer is never null, and the checker holds this proc to it: with that signature, return null is an error. option<*vec2> says it may be null, and then the checker holds every caller to it: reading through one before testing it doesn't compile.
type error: this pointer may be null: test it first, as 'if (p) { ... }', or take a '*T'
So the turret checks its target every time it aims, picks a new one when the old one is gone, and has to say what happens if that one is gone too:
aim: proc(w: *kayan_world, target: *kayan_entity, fallback: kayan_entity) -> void = {
p: option<*vec2> = get_position(w, target*);
if (!p) {
printfmt("the turret's target {}:{} is dead; it picks {}:{} instead\n",
target*.index, target*.generation, fallback.index, fallback.generation);
target* = fallback;
p = get_position(w, target*);
}
// The fallback could be dead too; the checker insists this is said.
if (!p) {
printfmt("the turret has nothing to aim at\n");
return;
}
printfmt("the turret aims at {}:{}, at ({:.2}, {:.2})\n", target*.index, target*.generation,
cast(p*.x, f64), cast(p*.y, f64));
}
Past a test like if (!p) { ...; return; }, the checker knows p isn't null. Assigning to p takes that knowledge away, which is why the second test is there. This program never reaches it, since the fallback is alive. But a dead fallback is a real case, and the checker won't let the code assume it away.
The pointer names no region, and that means something too: it points into the world the caller passed. Parts 2 and 3 lean on this.
Run it
the turret aims at 1:1, at (10.00, 0.00)
drone 1:1 is destroyed and despawned
the turret fires: the bullet is 1:2, the same number in its next life
entity 0:1 position ( 0.00, 0.00) velocity - health 500
entity 1:2 position ( 2.00, 2.00) velocity ( 4.00, 4.00) health -
entity 2:1 position ( 0.25, 9.50) velocity ( 0.50, -1.00) health 100
the turret's target 1:1 is dead; it picks 2:1 instead
the turret aims at 2:1, at (0.25, 9.50)
despawning 1:1 again: dead
the zeroed handle 0:0 is alive: no
The same moment that ended 1.2, and this time the handle knows.
Notes
- Cost: a handle is 8 bytes instead of 4, and checking one is one load and one compare. Cheap enough to do on every access.
- Wrapping: a
u32generation wraps after 4 billion reuses of the same number. A stale handle would have to survive that long to be fooled. Some ECSs use fewer bits for the generation, packing index and generation into oneu32oru64. The idea is the same. - What it doesn't fix: the arrays are still as long as the most entities, and systems still skip holes. That's next.
Try
- Keep a copy of the bullet's handle, despawn the bullet, spawn something new, and check the old handle with
kayan_alive. - Remove
_ =from one of thekayan_despawncalls and compile. Read the error. - Make
kayan_alivecompare only the index. Which line of the output goes wrong first? - Delete the second
if (!p)inaimand compile. The fallback is alive in this program, so why does the checker still object?
1// 1.3 Generations2//3// 1.2's bug: a number outlives the entity it named, and when the number is4// reused the old reference names the new entity. The fix is to count lives.5// Each number gets a generation, and a reference carries both: entity 1 in its6// first life is 1:1, and the entity that reuses the number is 1:2. Despawning7// bumps the generation, so every reference to the dead entity stops matching8// at once, wherever it is kept.9//10// A reference with a generation is called a handle. Checking one is a single11// comparison, so every access can afford it.12 13import "std/print.rin"14 15vec2: struct = {16 x: f32;17 y: f32;18}19 20// A handle: which number, and which life of that number.21kayan_entity: struct = {22 index: u32;23 generation: u32;24}25 26// What despawning reports. `dead` means the handle no longer names a live27// entity, so nothing was changed. must_use makes ignoring it a compile error.28kayan_status: enum[must_use] = {29 ok,30 dead,31}32 33// `generations[i]` is the life number `i` is in, or will be in next if `i` is34// free. The `alive` array from 1.2 is gone: the generations say it.35kayan_world: struct = {36 count: u32;37 generations: [8]u32;38 free: [8]u32;39 free_count: u32;40 positions: [8]vec2;41 has_position: [8]b32;42 velocities: [8]vec2;43 has_velocity: [8]b32;44 healths: [8]f32;45 has_health: [8]b32;46}47 48// A handle is alive when its generation is the one its number is in now.49// Despawning bumps the number's generation straight away, so a free number50// already holds a generation no handle has been given yet, and no handle51// matches it.52kayan_alive: proc(w: *kayan_world, e: kayan_entity) -> b32 = {53 return e.index < w*.count && w*.generations[e.index] == e.generation;54}55 56// Generations start at 1. So the zeroed handle, 0:0, never names a live57// entity, and `{}` can stand for "no entity".58kayan_spawn: proc(w: *kayan_world) -> kayan_entity = {59 e: kayan_entity = {};60 if (w*.free_count > 0) {61 w*.free_count -= 1;62 e.index = w*.free[w*.free_count];63 } else {64 if (w*.count == 8) {65 printfmt("the world is full\n");66 exit(1);67 }68 e.index = w*.count;69 w*.count += 1;70 w*.generations[e.index] = 1;71 }72 e.generation = w*.generations[e.index];73 return e;74}75 76// A dead handle reports `dead` and changes nothing, so despawning twice is77// harmless.78kayan_despawn: proc(w: *kayan_world, e: kayan_entity) -> kayan_status = {79 if (!kayan_alive(w, e)) {80 return kayan_status.dead;81 }82 w*.has_position[e.index] = false;83 w*.has_velocity[e.index] = false;84 w*.has_health[e.index] = false;85 w*.generations[e.index] += 1;86 w*.free[w*.free_count] = e.index;87 w*.free_count += 1;88 return kayan_status.ok;89}90 91set_position: proc(w: *kayan_world, e: kayan_entity, p: vec2) -> void = {92 w*.positions[e.index] = p;93 w*.has_position[e.index] = true;94}95 96set_velocity: proc(w: *kayan_world, e: kayan_entity, v: vec2) -> void = {97 w*.velocities[e.index] = v;98 w*.has_velocity[e.index] = true;99}100 101set_health: proc(w: *kayan_world, e: kayan_entity, points: f32) -> void = {102 w*.healths[e.index] = points;103 w*.has_health[e.index] = true;104}105 106// Reading through a handle checks it first. A dead handle, or an entity107// without a position, gives null instead of whatever is at the number. The108// result is `option<*vec2>`: it may be null, and the caller has to test it before109// reading through it. The pointer names no region: it points into the world110// the caller passed, and is good until the caller changes the world.111get_position: proc(w: *kayan_world, e: kayan_entity) -> option<*vec2> = {112 if (!kayan_alive(w, e) || !w*.has_position[e.index]) {113 return null;114 }115 return w*.positions[e.index]&;116}117 118movement_system: proc(w: *kayan_world, dt: f32) -> u32 = {119 moved: u32 = 0;120 for (i: u32 = 0; i < w*.count; i += 1) {121 if (w*.has_position[i] && w*.has_velocity[i]) {122 w*.positions[i].x += w*.velocities[i].x * dt;123 w*.positions[i].y += w*.velocities[i].y * dt;124 moved += 1;125 }126 }127 return moved;128}129 130print_world: proc(w: *kayan_world) -> void = {131 for (i: u32 = 0; i < w*.count; i += 1) {132 if (!w*.has_position[i] && !w*.has_velocity[i] && !w*.has_health[i]) {133 printfmt(" entity {} (free, next generation {})\n", i, w*.generations[i]);134 continue;135 }136 printfmt(" entity {}:{}", i, w*.generations[i]);137 if (w*.has_position[i]) {138 printfmt(" position ({:5.2}, {:5.2})", cast(w*.positions[i].x, f64), cast(w*.positions[i].y, f64));139 } else {140 printfmt(" position - ");141 }142 if (w*.has_velocity[i]) {143 printfmt(" velocity ({:5.2}, {:5.2})", cast(w*.velocities[i].x, f64), cast(w*.velocities[i].y, f64));144 } else {145 printfmt(" velocity - ");146 }147 if (w*.has_health[i]) {148 printfmt(" health {:.0}", cast(w*.healths[i], f64));149 } else {150 printfmt(" health -");151 }152 printfmt("\n");153 }154}155 156// The turret's aim: the target's position, or a new target if it is gone.157aim: proc(w: *kayan_world, target: *kayan_entity, fallback: kayan_entity) -> void = {158 p: option<*vec2> = get_position(w, target*);159 if (!p) {160 printfmt("the turret's target {}:{} is dead; it picks {}:{} instead\n",161 target*.index, target*.generation, fallback.index, fallback.generation);162 target* = fallback;163 p = get_position(w, target*);164 }165 // The fallback could be dead too; the checker insists this is said.166 if (!p) {167 printfmt("the turret has nothing to aim at\n");168 return;169 }170 printfmt("the turret aims at {}:{}, at ({:.2}, {:.2})\n", target*.index, target*.generation,171 cast(p*.x, f64), cast(p*.y, f64));172}173 174main: proc() -> i32 = {175 w: kayan_world = {};176 177 turret: kayan_entity = kayan_spawn(w&);178 set_position(w&, turret, { .x = 0.0, .y = 0.0 });179 set_health(w&, turret, 500.0);180 181 drone_a: kayan_entity = kayan_spawn(w&);182 set_position(w&, drone_a, { .x = 10.0, .y = 0.0 });183 set_velocity(w&, drone_a, { .x = -1.0, .y = 0.5 });184 set_health(w&, drone_a, 100.0);185 186 drone_b: kayan_entity = kayan_spawn(w&);187 set_position(w&, drone_b, { .x = 0.0, .y = 10.0 });188 set_velocity(w&, drone_b, { .x = 0.5, .y = -1.0 });189 set_health(w&, drone_b, 100.0);190 191 target: kayan_entity = drone_a;192 print_world(w&);193 aim(w&, target&, drone_b);194 195 printfmt("drone {}:{} is destroyed and despawned\n", drone_a.index, drone_a.generation);196 _ = kayan_despawn(w&, drone_a);197 198 bullet: kayan_entity = kayan_spawn(w&);199 set_position(w&, bullet, { .x = 0.0, .y = 0.0 });200 set_velocity(w&, bullet, { .x = 4.0, .y = 4.0 });201 printfmt("the turret fires: the bullet is {}:{}, the same number in its next life\n",202 bullet.index, bullet.generation);203 _ = movement_system(w&, 0.5);204 print_world(w&);205 206 // The same moment as the end of 1.2, and this time the handle knows.207 aim(w&, target&, drone_b);208 209 status: kayan_status = kayan_despawn(w&, drone_a);210 printfmt("despawning {}:{} again: {}\n", drone_a.index, drone_a.generation, status);211 212 none: kayan_entity = {};213 printfmt("the zeroed handle {}:{} is alive: {}\n", none.index, none.generation,214 kayan_alive(w&, none) ? "yes" : "no");215 return 0;216}What generations prints. tools/check.py fails the chapter if a single byte differs.
entity 0:1 position ( 0.00, 0.00) velocity - health 500
entity 1:1 position (10.00, 0.00) velocity (-1.00, 0.50) health 100
entity 2:1 position ( 0.00, 10.00) velocity ( 0.50, -1.00) health 100
the turret aims at 1:1, at (10.00, 0.00)
drone 1:1 is destroyed and despawned
the turret fires: the bullet is 1:2, the same number in its next life
entity 0:1 position ( 0.00, 0.00) velocity - health 500
entity 1:2 position ( 2.00, 2.00) velocity ( 4.00, 4.00) health -
entity 2:1 position ( 0.25, 9.50) velocity ( 0.50, -1.00) health 100
the turret's target 1:1 is dead; it picks 2:1 instead
the turret aims at 2:1, at (0.25, 9.50)
despawning 1:1 again: dead
the zeroed handle 0:0 is alive: no
Build an ECS · part 1, entities · 1.4
The entity index
ecs/src/1.entities/1.4.entity_index/entity_index.rin
In the dojo: 2. swap-remove and the back-reference, 3. the slot map, each on its own, to watch and to practise.
Handles fixed identity. Two costs from 1.1 remain: every array is as long as the most entities, and every system walks all of it, skipping holes. This chapter packs the data: live entities in rows 0 to count - 1, no holes, so a system is a straight loop over the rows. That's the stream lesson 1 measured as the fast case.
Packing has a consequence that the rest of the course is built around: an entity's data no longer sits at its number, and it moves.
To keep this chapter about the index, every entity is the same kind of thing: a particle with a position and a velocity. Part 2 brings back different kinds, each packed in a table of its own.
Step 1: two halves
kayan_slot: struct = {
generation: u32;
row: u32;
}
kayan_world: struct = {
slot_count: u32;
slots: [8]kayan_slot;
free: [8]u32;
free_count: u32;
count: u32;
entities: [8]kayan_entity;
positions: [8]vec2;
velocities: [8]vec2;
}
The world now has two halves:
- The slots, indexed by the entity's number. A number never moves, so this is where a handle looks first. A slot holds the generation from 1.3, and now also the row the entity's data is in.
- The rows, packed.
countof them, each with its components side by side in the arrays, andentities[row]saying which entity owns the row.
The two point at each other: slot to row, row to entity. That two-way link is called the entity index.
Step 2: spawning appends a row
row: u32 = w*.count;
w*.count += 1;
w*.entities[row] = e;
w*.positions[row] = p;
w*.velocities[row] = v;
w*.slots[e.index].row = row;
return e;
The number comes from the free list or the end, exactly as in 1.3. The data goes in the next row, which is always the end of the packed rows. The slot records where.
Step 3: despawning swap-removes
To keep the rows packed, a hole can't be left behind. The last row moves into it, called swap-remove:
kayan_despawn: proc(w: *kayan_world, e: kayan_entity) -> kayan_status = {
if (!kayan_alive(w, e)) {
return kayan_status.dead;
}
row: u32 = w*.slots[e.index].row;
last: u32 = w*.count - 1;
if (row != last) {
moved: kayan_entity = w*.entities[last];
w*.entities[row] = moved;
w*.positions[row] = w*.positions[last];
w*.velocities[row] = w*.velocities[last];
w*.slots[moved.index].row = row;
}
w*.count -= 1;
w*.slots[e.index].generation += 1;
w*.free[w*.free_count] = e.index;
w*.free_count += 1;
return kayan_status.ok;
}
The line that matters is this one:
w*.slots[moved.index].row = row;
The entity that owned the last row now lives somewhere else, and its slot must say so. entities[last] is how we know whose slot to fix: without that column, a moved row would be anonymous. Forget the line and that entity's handle still finds a row, just the wrong one, and nothing crashes.
Swap-remove costs one row copy, however many rows there are. Removing by shifting everything after the hole down would keep the order, but cost a copy of every later row. ECSs don't promise order, so they take the cheap one.
Step 4: reading through the index
get_position: proc(w: *kayan_world, e: kayan_entity) -> option<*vec2> = {
if (!kayan_alive(w, e)) {
return null;
}
return w*.positions[w*.slots[e.index].row]&;
}
Handle to slot to row. That's one extra lookup compared with 1.3, and it's the price of packing. As in 1.3, the result is an option<*vec2>, so main reads it inside a test:
at: option<*vec2> = get_position(w&, p[2]);
if (at) {
printfmt("{}:{} is at ({:.2}, {:.2}), wherever its row is\n", p[2].index, p[2].generation,
cast(at*.x, f64), cast(at*.y, f64));
}
Now that rows move, a pointer like at has a shelf life. These arrays are fixed, so the memory it points at stays good, and the checker has no reason to object if you keep it across a despawn. But a despawn can move another entity's row into that memory, and then the pointer reads the wrong entity. So use it right away and keep the handle: 2.5 turns this into a rule.
Systems don't pay for the lookup, because they don't go through handles:
movement_system: proc(w: *kayan_world, dt: f32) -> void = {
for (row: u32 = 0; row < w*.count; row += 1) {
w*.positions[row].x += w*.velocities[row].x * dt;
w*.positions[row].y += w*.velocities[row].y * dt;
}
}
No flags, no holes, no handles. Every row is a live particle. This is the loop the rest of the course works to keep.
Step 5: checking the invariant
Rows move all the time, and one missed slot update corrupts the world silently. So the world can check itself:
kayan_verify: proc(w: *kayan_world) -> u32 = {
problems: u32 = 0;
for (row: u32 = 0; row < w*.count; row += 1) {
e: kayan_entity = w*.entities[row];
if (!kayan_alive(w, e)) {
printfmt("kayan_verify: row {} holds dead entity {}:{}\n", row, e.index, e.generation);
problems += 1;
} else if (w*.slots[e.index].row != row) {
printfmt("kayan_verify: {}:{} is in row {}, its slot says row {}\n",
e.index, e.generation, row, w*.slots[e.index].row);
problems += 1;
}
}
live: u32 = w*.slot_count - w*.free_count;
if (live != w*.count) {
printfmt("kayan_verify: {} rows, {} live entities\n", w*.count, live);
problems += 1;
}
return problems;
}
The invariant: every row belongs to a live entity whose slot points back at that row, and there are as many rows as live entities. Every structural change in this course has to keep it true. When something goes wrong later, run kayan_verify first.
Run it
spawned 5 particles
row 0 0:1 position ( 0.00, 0.00)
row 1 1:1 position ( 1.00, 0.00)
...
row 4 4:1 position ( 4.00, 0.00)
slots: 0 -> row 0 1 -> row 1 2 -> row 2 3 -> row 3 4 -> row 4
despawn 1:1, in row 1: the last row moves into the hole
row 0 0:1 position ( 0.00, 0.00)
row 1 4:1 position ( 4.00, 0.00)
row 2 2:1 position ( 2.00, 0.00)
row 3 3:1 position ( 3.00, 0.00)
slots: 0 -> row 0 1 free 2 -> row 2 3 -> row 3 4 -> row 1
despawn 4:1, which the last despawn moved: its handle still finds it
...
kayan_verify: 0 problems
Follow 4:1: it starts in row 4, moves to row 1 when 1:1 is removed, and its handle still finds it there because its slot was updated.
Part 1 in one picture
handle 4:1 ──> slots[4] { generation 1, row 1 } ──> rows[1] { owner 4:1, position, velocity }
- Handles never change: they're what the rest of the game keeps.
- Slots never move: one per number, holding the generation and the row.
- Rows move all the time, and stay packed so systems can stream them.
Part 2 keeps exactly this, and generalizes the one kind of row into tables, one per combination of components.
Try
- Delete the line
w*.slots[moved.index].row = row;, run it, and read whatkayan_verifyreports. Then look at whatget_positionreturns for the moved entity. - Despawn every particle, then spawn three. Which numbers and rows do they get?
- Add a
kayan_get_velocityand use it to stop one particle by handle. - Write a
print_rowsthat prints rows in handle order instead of row order. What does it have to do? - Get a position pointer for
p[4], despawnp[1]so its row moves, and read through the pointer.rin checkaccepts this. What does it read, and whose data is that?
1// 1.4 The entity index2//3// So far every array has room for every number, and systems walk all of them,4// skipping holes. This chapter packs the data instead. Live entities sit in5// rows 0 to count-1 with no holes, and a system is a straight loop over the6// rows: the stream lesson 1 measured.7//8// Packing means an entity's data no longer sits at its number, and it moves.9// Removing a row moves the last row into the hole (swap-remove), so whatever10// was last changes row. Two things keep track:11//12// * each row records which entity owns it, `entities[row]`;13// * each number has a slot recording its generation and its current row.14//15// A handle finds its data through its slot. When a row moves, the moved16// entity's slot is updated. That one line is the invariant everything else in17// this course relies on, and `kayan_verify` checks it.18//19// To keep the chapter about the index, every entity here is the same kind of20// thing: a particle with a position and a velocity. Part 2 brings back21// different kinds, each packed in a table of its own.22 23import "std/print.rin"24 25vec2: struct = {26 x: f32;27 y: f32;28}29 30kayan_entity: struct = {31 index: u32;32 generation: u32;33}34 35kayan_status: enum[must_use] = {36 ok,37 dead,38}39 40// Where a number's entity is: its generation, and the row its data is in.41kayan_slot: struct = {42 generation: u32;43 row: u32;44}45 46// Two halves. The slots, indexed by the entity's number, which never moves. The47// rows, packed, indexed by row, which moves: `count` of them, each with its48// owner and its components side by side.49kayan_world: struct = {50 slot_count: u32;51 slots: [8]kayan_slot;52 free: [8]u32;53 free_count: u32;54 55 count: u32;56 entities: [8]kayan_entity;57 positions: [8]vec2;58 velocities: [8]vec2;59}60 61kayan_alive: proc(w: *kayan_world, e: kayan_entity) -> b32 = {62 return e.index < w*.slot_count && w*.slots[e.index].generation == e.generation;63}64 65// A number, as in 1.3, and the next row at the end of the packed rows.66kayan_spawn: proc(w: *kayan_world, p: vec2, v: vec2) -> kayan_entity = {67 e: kayan_entity = {};68 if (w*.free_count > 0) {69 w*.free_count -= 1;70 e.index = w*.free[w*.free_count];71 } else {72 if (w*.slot_count == 8) {73 printfmt("the world is full\n");74 exit(1);75 }76 e.index = w*.slot_count;77 w*.slot_count += 1;78 w*.slots[e.index].generation = 1;79 }80 e.generation = w*.slots[e.index].generation;81 82 row: u32 = w*.count;83 w*.count += 1;84 w*.entities[row] = e;85 w*.positions[row] = p;86 w*.velocities[row] = v;87 w*.slots[e.index].row = row;88 return e;89}90 91// Swap-remove: the last row moves into the hole, so the rows stay packed. The92// entity that owned the last row now lives at `row`, and its slot must say so.93kayan_despawn: proc(w: *kayan_world, e: kayan_entity) -> kayan_status = {94 if (!kayan_alive(w, e)) {95 return kayan_status.dead;96 }97 row: u32 = w*.slots[e.index].row;98 last: u32 = w*.count - 1;99 if (row != last) {100 moved: kayan_entity = w*.entities[last];101 w*.entities[row] = moved;102 w*.positions[row] = w*.positions[last];103 w*.velocities[row] = w*.velocities[last];104 w*.slots[moved.index].row = row;105 }106 w*.count -= 1;107 w*.slots[e.index].generation += 1;108 w*.free[w*.free_count] = e.index;109 w*.free_count += 1;110 return kayan_status.ok;111}112 113// Handle, then slot, then row: the data is wherever the row is now. `option<*vec2>`:114// null for a dead handle, and the caller tests it before reading.115get_position: proc(w: *kayan_world, e: kayan_entity) -> option<*vec2> = {116 if (!kayan_alive(w, e)) {117 return null;118 }119 return w*.positions[w*.slots[e.index].row]&;120}121 122// No flags and no holes: every row is a live particle. This is a stream over123// two packed arrays.124movement_system: proc(w: *kayan_world, dt: f32) -> void = {125 for (row: u32 = 0; row < w*.count; row += 1) {126 w*.positions[row].x += w*.velocities[row].x * dt;127 w*.positions[row].y += w*.velocities[row].y * dt;128 }129}130 131// The invariant, checked: every row belongs to a live entity whose slot points132// back at that row, and there are as many rows as live entities. Returns how133// many checks failed, printing each one.134kayan_verify: proc(w: *kayan_world) -> u32 = {135 problems: u32 = 0;136 for (row: u32 = 0; row < w*.count; row += 1) {137 e: kayan_entity = w*.entities[row];138 if (!kayan_alive(w, e)) {139 printfmt("kayan_verify: row {} holds dead entity {}:{}\n", row, e.index, e.generation);140 problems += 1;141 } else if (w*.slots[e.index].row != row) {142 printfmt("kayan_verify: {}:{} is in row {}, its slot says row {}\n",143 e.index, e.generation, row, w*.slots[e.index].row);144 problems += 1;145 }146 }147 live: u32 = w*.slot_count - w*.free_count;148 if (live != w*.count) {149 printfmt("kayan_verify: {} rows, {} live entities\n", w*.count, live);150 problems += 1;151 }152 return problems;153}154 155print_rows: proc(w: *kayan_world) -> void = {156 for (row: u32 = 0; row < w*.count; row += 1) {157 e: kayan_entity = w*.entities[row];158 printfmt(" row {} {}:{} position ({:5.2}, {:5.2})\n", row, e.index, e.generation,159 cast(w*.positions[row].x, f64), cast(w*.positions[row].y, f64));160 }161 // A slot is in use when the row it points at is owned by its number, in its162 // current generation. A free slot still holds the row it last had.163 printfmt(" slots:");164 for (i: u32 = 0; i < w*.slot_count; i += 1) {165 s: kayan_slot = w*.slots[i];166 owner: kayan_entity = w*.entities[s.row];167 if (s.row < w*.count && owner.index == i && owner.generation == s.generation) {168 printfmt(" {} -> row {}", i, s.row);169 } else {170 printfmt(" {} free", i);171 }172 }173 printfmt("\n");174}175 176main: proc() -> i32 = {177 w: kayan_world = {};178 p: [5]kayan_entity = {};179 for (i: u32 = 0; i < 5; i += 1) {180 p[i] = kayan_spawn(w&, { .x = cast(i, f32), .y = 0.0 }, { .x = 0.0, .y = 1.0 });181 }182 printfmt("spawned 5 particles\n");183 print_rows(w&);184 185 printfmt("despawn {}:{}, in row 1: the last row moves into the hole\n", p[1].index, p[1].generation);186 _ = kayan_despawn(w&, p[1]);187 print_rows(w&);188 189 printfmt("despawn {}:{}, which the last despawn moved: its handle still finds it\n",190 p[4].index, p[4].generation);191 _ = kayan_despawn(w&, p[4]);192 print_rows(w&);193 194 q: kayan_entity = kayan_spawn(w&, { .x = 9.0, .y = 0.0 }, { .x = 0.0, .y = 1.0 });195 printfmt("spawn {}:{}: it reuses a number and takes the next row\n", q.index, q.generation);196 print_rows(w&);197 198 movement_system(w&, 0.5);199 movement_system(w&, 0.5);200 printfmt("two steps of movement, a straight loop over {} rows\n", w.count);201 print_rows(w&);202 203 at: option<*vec2> = get_position(w&, p[2]);204 if (at) {205 printfmt("{}:{} is at ({:.2}, {:.2}), wherever its row is\n", p[2].index, p[2].generation,206 cast(at*.x, f64), cast(at*.y, f64));207 }208 printfmt("{}:{} has a position: {}\n", p[4].index, p[4].generation,209 get_position(w&, p[4]) != null ? "yes" : "no, it is dead");210 printfmt("kayan_verify: {} problems\n", kayan_verify(w&));211 return 0;212}What entity_index prints. tools/check.py fails the chapter if a single byte differs.
spawned 5 particles
row 0 0:1 position ( 0.00, 0.00)
row 1 1:1 position ( 1.00, 0.00)
row 2 2:1 position ( 2.00, 0.00)
row 3 3:1 position ( 3.00, 0.00)
row 4 4:1 position ( 4.00, 0.00)
slots: 0 -> row 0 1 -> row 1 2 -> row 2 3 -> row 3 4 -> row 4
despawn 1:1, in row 1: the last row moves into the hole
row 0 0:1 position ( 0.00, 0.00)
row 1 4:1 position ( 4.00, 0.00)
row 2 2:1 position ( 2.00, 0.00)
row 3 3:1 position ( 3.00, 0.00)
slots: 0 -> row 0 1 free 2 -> row 2 3 -> row 3 4 -> row 1
despawn 4:1, which the last despawn moved: its handle still finds it
row 0 0:1 position ( 0.00, 0.00)
row 1 3:1 position ( 3.00, 0.00)
row 2 2:1 position ( 2.00, 0.00)
slots: 0 -> row 0 1 free 2 -> row 2 3 -> row 1 4 free
spawn 4:2: it reuses a number and takes the next row
row 0 0:1 position ( 0.00, 0.00)
row 1 3:1 position ( 3.00, 0.00)
row 2 2:1 position ( 2.00, 0.00)
row 3 4:2 position ( 9.00, 0.00)
slots: 0 -> row 0 1 free 2 -> row 2 3 -> row 1 4 -> row 3
two steps of movement, a straight loop over 4 rows
row 0 0:1 position ( 0.00, 1.00)
row 1 3:1 position ( 3.00, 1.00)
row 2 2:1 position ( 2.00, 1.00)
row 3 4:2 position ( 9.00, 1.00)
slots: 0 -> row 0 1 free 2 -> row 2 3 -> row 1 4 -> row 3
2:1 is at (2.00, 1.00), wherever its row is
4:1 has a position: no, it is dead
kayan_verify: 0 problems
Build an ECS · part 2, components · 2.1
Components and signatures
ecs/src/2.components/2.1.components/components.rin
In the dojo: 6. bitsets, on its own, to watch and to practise.
Part 1 hard-coded its components: a positions array, a velocities array, a flag for each. Adding a component meant editing the world, kayan_spawn, kayan_despawn and every printer. Part 2 builds a world that works for components it has never heard of. This chapter lays the ground with four ideas:
- A component is a plain struct.
- Each component type has an id, a small number fixed at compile time.
- The world keeps a registry: per id, the component's size, alignment and name, which is all it needs to store bytes it doesn't understand.
- A set of components is a signature: one bit per id in a
u64.
There are no entities in this chapter. 2.3 puts the two halves together.
Step 1: components are plain structs
position: struct = { at: vec2; }
velocity: struct = { per_second: vec2; }
health: struct = { points: f32; max: f32; }
damage: struct = { amount: f32; }
lifetime: struct = { seconds: f32; }
burning: struct = { per_second: f32; }
turret: struct = { cooldown: f32; timer: f32; }
No procs, no pointers to other entities, no idea that a world exists. That's deliberate. Behaviour belongs to systems, and data that knows nothing can be copied, moved and stored as raw bytes, which the world is about to do.
Why position { at: vec2 } rather than using vec2 directly? Because a component's type is how the world tells components apart. Position and velocity are both two floats, but they're different things, and making them different types is what lets you ask for one and not the other.
Step 2: an id per component type
component: enum = {
position,
velocity,
health,
damage,
lifetime,
burning,
turret,
}
The enum gives each component a number. Then one line per type connects the type to its number:
component_of: proc<position>() -> component = { return component.position; }
component_of: proc<velocity>() -> component = { return component.velocity; }
Each line is a specialization of the generic proc component_of<T> for one type. component_of<health>() is resolved by the compiler to component.health. Nothing is looked up at run time; the id is a constant in the generated code.
Step 3: the world speaks ids, never types
The world can't be generic over component types. Which combinations exist is only known while the game runs. So the world works in ids and bits, and a small typed layer translates:
kayan_bit: proc(id: u32) -> u64 = {
return 1'u64 shl cast(id, u64);
}
// The typed layer: from a type to the id and bit the world works in.
kayan_id_of: proc<T>() -> u32 = {
return cast(component_of<T>(), u32);
}
kayan_mask_of: proc<T>() -> u64 = {
return kayan_bit(kayan_id_of<T>());
}
Game code says kayan_mask_of<velocity>(), and the world only ever sees 2, a bit.
Step 4: the registry
To store a component as bytes, the world needs three facts about it: how many bytes, what alignment, and a name for printing. rin's compiler already wrote all three down. position<> is the reflect record for position, data the compiler puts into the program. So the registry is an array of pointers to reflect records, indexed by id:
kayan_world: struct = {
infos: [64]option<*const reflect in permanent>;
component_count: u32;
}
The slot's type says two true things:
in permanent: a reflect record is written into the program, so it lives inpermanent, the region of the whole program. A pointer intopermanentis good forever, so the world may keep it. Part 2 moves the world into memory of its own, and the slots won't need to change.option<...>: a slot nobody registered is null, so every read has to test it.
kayan_register: proc(w: *kayan_world, id: u32, info: *const reflect in permanent) -> void = {
if (id >= 64) {
printfmt("kayan: component id {} is past the 64 a signature can hold\n", id);
exit(1);
}
w*.infos[id] = info;
if (id + 1 > w*.component_count) {
w*.component_count = id + 1;
}
}
register_components: proc(w: *kayan_world) -> void = {
kayan_register(w, kayan_id_of<position>(), position<>&);
kayan_register(w, kayan_id_of<velocity>(), velocity<>&);
Each line names the type twice, once for the id and once for the record. A generic kayan_register_type<T> that took T<>& would say it once, and rin allows that; writing it is the fourth exercise below.
The printers want a component's name, and a slot may be empty, so that question is asked once, here, instead of at every print:
kayan_name: proc(w: *kayan_world, id: u32) -> ^const char = {
info: option<*const reflect in permanent> = w*.infos[id];
return info ? info*.name : "?";
}
Step 5: signatures
A set of components is a u64 with a bit set per component in the set. What a thing is is a signature:
drone: u64 = kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<health>();
Every question you'd ask about sets is a bit operation:
kayan_contains: proc(signature: u64, wanted: u64) -> b32 = {
return (signature & wanted) == wanted;
}
"Does this drone have what movement wants?" is kayan_contains(drone, moves), one AND and one compare. Adding a component is |, and removing one is & ~:
put_out: u64 = burning_drone & ~kayan_mask_of<burning>();
64 bits means at most 64 component types per world, which is plenty for learning and for most games. Larger ECSs use wider bitsets or sorted id lists; the idea is the same.
Run it
the registry: 7 components
id name size align
0 position 8 4
1 velocity 8 4
2 health 8 4
3 damage 4 4
4 lifetime 4 4
5 burning 4 4
6 turret 8 4
signatures, one bit per component, highest id first:
drone 0000111 {position velocity health}
bullet 0011011 {position velocity damage lifetime}
turret 1000001 {position turret}
burning drone 0100111 {position velocity health burning}
movement wants {position velocity}:
drone yes, bullet yes, turret no, burning drone yes
a burning drone with burning removed is a drone again: yes
Read the bits as a table: each column is a component, each row a kind of thing. That table is the next three chapters. Each distinct row will become a table of its own.
Try
- Add a
shield: struct = { points: f32; }component. What do you have to touch? (Four lines.) Print a "shielded drone" signature. - Print
kayan_contains(moves, drone)(arguments swapped). Why is the answer different? - Reorder the
componentenum. Which outputs change, and which don't? - Write
kayan_register_type<T>withT<>&, and use it inregister_components.
1// 2.1 Components and signatures2//3// Part 1 hard-coded its components: a positions array, a velocities array, a4// flag for each. Every new component meant editing the world. Part 2 makes the5// world work for components it has never heard of, and this chapter lays the6// ground: what a component is, how the world tells components apart without7// knowing their types, and how a set of components is written down.8//9// * A component is a plain struct.10// * Each component type gets an id, a small number, decided at compile time.11// * The world keeps a registry: for each id, the component's reflect record,12// which says its name, size and alignment. That is all the world needs to13// store, copy and move a component it knows nothing else about.14// * A set of components is a signature: a u64 with bit `id` set for each15// component in the set. Asking whether a set contains another is one AND.16//17// No entities in this chapter. 2.3 puts the two halves together.18//19// The registry's slots are `option<*const reflect in permanent>`. A reflect record is20// data the compiler wrote into the program, so it lives in `permanent`, the21// region of the whole program; and a slot nobody registered is null, so the22// type says so and every read tests it.23 24import "std/print.rin"25 26vec2: struct = {27 x: f32;28 y: f32;29}30 31// The components. Plain data, nothing else: no procs, no pointers to other32// entities, no knowledge of the world.33position: struct = { at: vec2; }34velocity: struct = { per_second: vec2; }35health: struct = { points: f32; max: f32; }36damage: struct = { amount: f32; }37lifetime: struct = { seconds: f32; }38burning: struct = { per_second: f32; }39turret: struct = { cooldown: f32; timer: f32; }40 41// One id per component type: its position in this enum.42component: enum = {43 position,44 velocity,45 health,46 damage,47 lifetime,48 burning,49 turret,50}51 52// From a type to its id, at compile time. Each line is a specialization of the53// generic proc for one type; calling component_of<health>() is resolved by the54// compiler to `component.health`, with nothing looked up at run time.55component_of: proc<position>() -> component = { return component.position; }56component_of: proc<velocity>() -> component = { return component.velocity; }57component_of: proc<health>() -> component = { return component.health; }58component_of: proc<damage>() -> component = { return component.damage; }59component_of: proc<lifetime>() -> component = { return component.lifetime; }60component_of: proc<burning>() -> component = { return component.burning; }61component_of: proc<turret>() -> component = { return component.turret; }62 63// ---------------------------------------------------------------------------64// The world's side: ids and bits, never types65// ---------------------------------------------------------------------------66 67// The registry. The world stores components as bytes, so per id it needs to68// know how many bytes, at what alignment, and a name for printing. A reflect69// record has all three.70kayan_world: struct = {71 infos: [64]option<*const reflect in permanent>;72 component_count: u32;73}74 75// One bit per component. A signature is a u64, so a world has at most 64.76kayan_bit: proc(id: u32) -> u64 = {77 return 1'u64 shl cast(id, u64);78}79 80// The typed layer: from a type to the id and bit the world works in.81kayan_id_of: proc<T>() -> u32 = {82 return cast(component_of<T>(), u32);83}84 85kayan_mask_of: proc<T>() -> u64 = {86 return kayan_bit(kayan_id_of<T>());87}88 89kayan_register: proc(w: *kayan_world, id: u32, info: *const reflect in permanent) -> void = {90 if (id >= 64) {91 printfmt("kayan: component id {} is past the 64 a signature can hold\n", id);92 exit(1);93 }94 w*.infos[id] = info;95 if (id + 1 > w*.component_count) {96 w*.component_count = id + 1;97 }98}99 100// A component's name, for printing: a registered id's, or "?" for a slot101// nobody filled. The registry slot may be null, so the question is asked here102// once instead of at every print.103kayan_name: proc(w: *kayan_world, id: u32) -> ^const char = {104 info: option<*const reflect in permanent> = w*.infos[id];105 return info ? info*.name : "?";106}107 108// A signature as names, `{position velocity}`.109kayan_print_signature: proc(w: *kayan_world, signature: u64) -> void = {110 printfmt("{");111 first: b32 = true;112 for (id: u32 = 0; id < w*.component_count; id += 1) {113 if ((signature & kayan_bit(id)) != 0) {114 if (!first) {115 printfmt(" ");116 }117 printfmt("{}", kayan_name(w, id));118 first = false;119 }120 }121 printfmt("}");122}123 124// A signature as bits, highest component first, so each column of the output125// is one component.126print_bits: proc(w: *kayan_world, signature: u64) -> void = {127 id: u32 = w*.component_count;128 while (id > 0) {129 id -= 1;130 printfmt("{}", (signature & kayan_bit(id)) != 0 ? "1" : "0");131 }132}133 134// Does a set contain every component of another? One AND and one compare.135kayan_contains: proc(signature: u64, wanted: u64) -> b32 = {136 return (signature & wanted) == wanted;137}138 139// `position<>` is the compiler's reflect record for position: its name, size140// and alignment, written into the program as data. Registering hands the world141// that record under position's id.142register_components: proc(w: *kayan_world) -> void = {143 kayan_register(w, kayan_id_of<position>(), position<>&);144 kayan_register(w, kayan_id_of<velocity>(), velocity<>&);145 kayan_register(w, kayan_id_of<health>(), health<>&);146 kayan_register(w, kayan_id_of<damage>(), damage<>&);147 kayan_register(w, kayan_id_of<lifetime>(), lifetime<>&);148 kayan_register(w, kayan_id_of<burning>(), burning<>&);149 kayan_register(w, kayan_id_of<turret>(), turret<>&);150}151 152main: proc() -> i32 = {153 w: kayan_world = {};154 register_components(w&);155 156 printfmt("the registry: {} components\n", w.component_count);157 printfmt(" id name size align\n");158 for (id: u32 = 0; id < w.component_count; id += 1) {159 info: option<*const reflect in permanent> = w.infos[id];160 if (info) {161 printfmt(" {:2} {:-9} {:4} {:5}\n", id, info*.name, cast(info*.size, u32), cast(info*.align, u32));162 }163 }164 165 // What a thing is, written as the set of its components.166 drone: u64 = kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<health>();167 bullet: u64 = kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<damage>()168 | kayan_mask_of<lifetime>();169 gun: u64 = kayan_mask_of<position>() | kayan_mask_of<turret>();170 burning_drone: u64 = drone | kayan_mask_of<burning>();171 172 printfmt("signatures, one bit per component, highest id first:\n");173 printfmt(" drone "); print_bits(w&, drone); printfmt(" "); kayan_print_signature(w&, drone); printfmt("\n");174 printfmt(" bullet "); print_bits(w&, bullet); printfmt(" "); kayan_print_signature(w&, bullet); printfmt("\n");175 printfmt(" turret "); print_bits(w&, gun); printfmt(" "); kayan_print_signature(w&, gun); printfmt("\n");176 printfmt(" burning drone "); print_bits(w&, burning_drone); printfmt(" "); kayan_print_signature(w&, burning_drone); printfmt("\n");177 178 // Which of them would a movement system want? Everything that contains179 // {position velocity}.180 moves: u64 = kayan_mask_of<position>() | kayan_mask_of<velocity>();181 printfmt("movement wants "); kayan_print_signature(w&, moves); printfmt(":\n");182 printfmt(" drone {}, bullet {}, turret {}, burning drone {}\n",183 kayan_contains(drone, moves) ? "yes" : "no", kayan_contains(bullet, moves) ? "yes" : "no",184 kayan_contains(gun, moves) ? "yes" : "no", kayan_contains(burning_drone, moves) ? "yes" : "no");185 186 // Adding and removing a component is setting and clearing its bit.187 put_out: u64 = burning_drone & ~kayan_mask_of<burning>();188 printfmt("a burning drone with burning removed is a drone again: {}\n", put_out == drone ? "yes" : "no");189 return 0;190}What components prints. tools/check.py fails the chapter if a single byte differs.
the registry: 7 components
id name size align
0 position 8 4
1 velocity 8 4
2 health 8 4
3 damage 4 4
4 lifetime 4 4
5 burning 4 4
6 turret 8 4
signatures, one bit per component, highest id first:
drone 0000111 {position velocity health}
bullet 0011011 {position velocity damage lifetime}
turret 1000001 {position turret}
burning drone 0100111 {position velocity health burning}
movement wants {position velocity}:
drone yes, bullet yes, turret no, burning drone yes
a burning drone with burning removed is a drone again: yes
Build an ECS · part 2, components · 2.2
Columns
ecs/src/2.components/2.2.columns/columns.rin
In the dojo: 4. a type-erased column, 8. index, not pointer, each on its own, to watch and to practise.
1.4 kept components in typed arrays: positions: [8]vec2. A world that works for any component can't, because it doesn't know the types. It knows, from the registry, each component's size and alignment, and that's enough.
A column is one component's array, kept as bytes. Row r is the size bytes starting at r * size. Everything a table does with its data is a byte operation on columns: grow, copy one row over another, zero a new row. None of it depends on the type. The type comes back only at the edge, where game code reads a column as []T.
This chapter builds the column and exercises it by hand. 2.3 puts columns in a table. It's also the first chapter whose memory can grow, so it's the first that needs a region, and the first where the checker can't follow everything we do. Both of those show up in what a column looks like.
Step 1: what a column is
A region is memory that ends all at once, an arena with a name the checker can reason about. The file names one at the top:
level: region;
and main opens it around everything that uses it:
level: region = {
When that block ends, everything allocated in level is gone together. There's nothing to free one piece at a time, and nothing that can be freed too early: the checker won't let a pointer into level outlive it.
kayan_column: struct = {
data: []u8 in level;
size: u64;
align: u64;
}
Bytes, plus how many bytes make a row, plus the alignment to allocate with. No count and no capacity: every column of a table has the same number of rows, so the table keeps those once (2.3).
data is a slice with a length, every byte the column owns, and in level says where those bytes are. That's a rule, not a comment: a pointer or slice kept in memory has to name the region it points into. Only a local or a parameter may leave it off.
A column for a component is made from its reflect record:
kayan_column_init: proc(c: *kayan_column, info: *const reflect) -> void = {
c*.size = info*.size;
c*.align = info*.align;
}
Its bytes will be in the region, so the column is too:
positions: *kayan_column in level = new<kayan_column>(level);
healths: *kayan_column in level = new<kayan_column>(level);
kayan_column_init(positions, position<>&);
kayan_column_init(healths, health<>&);
Step 2: rows are byte offsets
kayan_column_row: proc(c: *kayan_column, row: u64) -> []u8 in level = {
return c*.data[row * c*.size..(row + 1) * c*.size];
}
That's the whole addressing scheme: row * size. With it, copying and zeroing rows are plain loops over bytes:
kayan_column_copy_row: proc(c: *kayan_column, to: u64, from: u64) -> void = {
for (i: u64 = 0; i < c*.size; i += 1) {
c*.data[to * c*.size + i] = c*.data[from * c*.size + i];
}
}
kayan_column_zero_row: proc(c: *kayan_column, row: u64) -> void = {
for (i: u64 = 0; i < c*.size; i += 1) {
c*.data[row * c*.size + i] = 0;
}
}
kayan_column_copy_row is swap-remove's copy from 1.4, for any component. These are loops rather than memcpy and memset because those two take raw pointers and nothing checks what they do with them. Because data has a length, these loops are ordinary checked code: a row past the end stops the program, in every build, instead of writing over a neighbour.
Step 3: growing
kayan_column_grow: proc(r: region level, c: *kayan_column, count: u64, capacity: u64) -> void = {
unsafe {
raw: ^u8 = cast(memops_arena_push_zero(r, c*.size * capacity, c*.align), ^u8);
bytes: []u8 in level = cast(raw[0..c*.size * capacity], []u8 in level);
// Slicing checks both ends: a count past either block stops the program.
kept: u64 = c*.size * count;
slice<u8>copy(bytes[0..kept], c*.data[0..kept]);
c*.data = bytes;
}
}
A new block for capacity rows, with the rows in use copied over. The old block stays in the region, unused: a region frees everything at once, never one allocation. Doubling the capacity each time keeps the waste to at most the size of what's in use, and the copies to a few per table over a whole game.
r: region level is how a proc asks for the right to allocate: a region parameter says which region, so you can read off a signature whether a proc can allocate at all, and where.
This is the first unsafe { } in the course, and it's here because of what a column is. Checked code can allocate a []T with new_slice<T>, which knows T. A column doesn't know T. It has a size and an alignment as run-time numbers, and only the arena can hand back bytes aligned for them. So this proc goes under the checker: it calls the arena directly, and casts what comes back. An unsafe { } block is the author saying "I keep the rules here myself", and everything outside it takes the proc's signature at its word. The course calls a proc like this trusted. Its signature has to be honest: this one says the bytes end up in level, and they do.
Notice what the block does straight after it gets its bytes: it turns them into a slice, and copies with std's checked slice<u8>copy rather than memcpy. Why bother, when the rows in use are right there in count? Step 4 answers that.
This is also why pointers into a column go stale: after a grow, the data is somewhere else. Notice what stale means here, because it's what makes the shape safe. The old block is still in the region. A pointer into it reads good memory holding old values. It can be wrong, but it can't be dangling, since no block in level goes away before level does. 2.5 shows it happening.
Step 4: the typed view
kayan_column_slice: proc<T>(c: *const kayan_column, count: u64) -> []T in level = {
if (c*.size != sizeof(T)) {
printfmt("kayan: a column of {}-byte rows read as a {}-byte type\n", c*.size, sizeof(T));
exit(1);
}
if (!reflect_is_plain(T<>&)) {
printfmt("kayan: a column holds plain data, and this {}-byte type is not\n", sizeof(T));
exit(1);
}
if (count > c*.data.length / sizeof(T)) {
printfmt("kayan: a view of {} rows on a column with room for {}\n", count, c*.data.length / sizeof(T));
exit(1);
}
unsafe {
if (cast(c*.data.data, usize) % alignof(T) != 0) {
printfmt("kayan: a column's bytes are not aligned to {}\n", alignof(T));
exit(1);
}
raw: ^T = cast(c*.data.data, ^T);
return cast(raw[0..count], []T in level);
}
}
This is the untyped/typed split in one proc. Below it, bytes. Above it, a []T that game code indexes like any slice, bounds checked in every build. The cast from bytes to T is the reinterpretation itself, the second thing a checker can't see through, so this is the chapter's second unsafe { }. It's four checks and a cast, and only the last check and the cast are inside the block.
To see why so many checks, ask what an unsafe block is allowed to believe.
rin has no private fields. Any checked code holding a *kayan_column can write c*.size, or cut c*.data short, and any caller can pass any count. Nothing stops them, and nothing should have to. So an unsafe block can't believe a field or an argument just because the code that normally sets it is correct. It can only believe a value that carries its own bound, which is what a slice does: it knows its length, and checked code can only get one by allocating it or cutting another. So here's what the view checks, rather than believes:
- Room.
countcomes from a table, but the view measures it against the slice, the one thing no field can lie about. It divides rather than multiplies, so a hugecountcan't overflow its way past the test. - Plain data.
reflect_is_plainlooks atT's record: numbers, enums, and structs and arrays of those, with no pointer anywhere. For such aT, any bytes are a validT. That's what makes this safe. It also meansThas to be a struct or an enum, sinceT<>has no record for a bare number. - Alignment. The bytes came from the arena at
c*.align, butc*.alignandc*.dataare both just fields, so the block tests the address itself. - Size. This one catches a mistake. It isn't what keeps the view safe:
positionandhealthare both 8 bytes, so reading one as the other passes every check. You get wrong numbers, and never a bad pointer, because neither has a pointer in it. Which is the same promise 2.5 is about to make for a stale pointer: a wrong answer, never a bad address.
Now look back at the grow. It takes its count from the caller and its sizes from fields, so it doesn't believe those either. It cuts both blocks to the rows it was told to keep, and a count past either one stops the program before a byte moves.
Those two blocks are the whole unsafe core of the type-erased column, and they're small on purpose. The rest of the chapter is checked, and so is everything built on it for the rest of the course. And since the blocks check what they're handed, a mistake in the checked code above them stops the program instead of corrupting memory.
Writing goes through the typed view too:
ps: []position = kayan_column_slice<position>(positions, count);
hs: []health = kayan_column_slice<health>(healths, count);
ps[row] = { .at = { .x = cast(i, f32), .y = 1.0 } };
hs[row] = { .points = 50.0 + 10.0 * cast(i, f32), .max = 100.0 };
(Writing kayan_column_slice<position>(...)[row] = ... in one line would be nicer. rin doesn't allow indexing a call's result directly yet, because the bounds check would evaluate the call twice. Hence the locals.)
Step 5: rows stay rows across columns
The program keeps two columns side by side, positions and healths, and moves them together:
last: u64 = count - 1;
kayan_column_copy_row(positions, 1, last);
kayan_column_copy_row(healths, 1, last);
count -= 1;
Same operation, each column using its own row size. Row 1 is still one entity's data across both columns. That's a table in all but name.
Run it
a position column has 8-byte rows, a health column 8-byte rows
grew to room for 2 rows
grew to room for 4 rows
grew to room for 8 rows
five rows, written through the typed view:
row 0 position ( 0.00, 1.00) health 50 / 100
row 1 position ( 1.00, 1.00) health 60 / 100
...
row 1's health as the column stores it, bytes: 00 00 70 42 00 00 c8 42
that is 60.0 and 100.0 as little-endian f32s; the column never knew
swap-removed row 1: row 4 moved into it, in both columns
The byte line is the point of the chapter. 00 00 70 42 is 60.0 as an f32, and the column stored it without knowing it was a float, or a health, or anything at all.
Try
- Add
damage: struct = { amount: f32; }, read the health column as[]damage(4-byte rows) and run it. Read the message. - Start the capacity at 1 instead of 2. How many grows does it take to fit 5 rows? What if it grew by 1 each time instead of doubling?
- Print the bytes of a position row. Which bytes are
xand which arey? - Take the
unsafe { }offkayan_column_growand compile. Read each thing the checker objects to. Which of them could a column avoid, knowing only a size and an alignment? - Read the position column as
[]health. Every check passes. What are row 3'spointsandmax, and why is this wrong but not dangerous? - Add a component with a pointer in it,
holder: struct = { at: *position in level; }, and read the health column as[]holder. It's the right size. Why would it be dangerous if this view let it through? - Break the column on purpose, from checked code: after the rows are written, add
positions*.data = positions*.data[1..positions*.data.length];.rin checkhas no objection. Run it. Then trycount = 1000;, andkayan_column_zero_row(positions, 1000);. Who stops each one?
1// 2.2 Columns2//3// 1.4 kept its components in typed arrays, `positions: [8]vec2`. A world that4// works for any component cannot: it does not know the types. It knows, from5// the registry, each component's size and alignment, and that is enough. A6// column is an array of a component kept as bytes: row `r` is the `size` bytes7// at `r * size`.8//9// Everything a table needs to do with its data is a byte operation on columns:10// grow, copy a row over another (swap-remove), zero a new row. None of it11// depends on the type. The type comes back at the edge, where game code reads12// the column as `[]T`, and that typed view checks the size matches.13//14// This chapter builds the column and exercises it on its own, with the count15// and capacity kept beside it by hand. 2.3 puts columns in a table.16//17// The column's bytes are a slice into the `level` region,18// `[]u8 in level`, with a real length, so copying and zeroing rows are plain19// bounds-checked loops that the checker reads. Two things about a column are20// unsafe by nature and are the chapter's two `unsafe { }` procs: growing it,21// which asks the arena for bytes at the component's alignment, and the typed22// view, which reads those bytes as `[]T`. Everything else is checked code.23 24import "std/print.rin"25import "std/memops.rin"26import "std/slice.rin"27 28level: region;29 30vec2: struct = {31 x: f32;32 y: f32;33}34 35position: struct = { at: vec2; }36health: struct = { points: f32; max: f32; }37 38component: enum = {39 position,40 health,41}42 43component_of: proc<position>() -> component = { return component.position; }44component_of: proc<health>() -> component = { return component.health; }45 46// ---------------------------------------------------------------------------47// The column: bytes, and how many of them make one row48// ---------------------------------------------------------------------------49 50// A column does not know its count or capacity. A table keeps those once for51// all its columns, since every column of a table has the same number of rows.52// `data` is every byte the column owns, `size * capacity` of them.53kayan_column: struct = {54 data: []u8 in level;55 size: u64;56 align: u64;57}58 59// An empty column for one component. Its size and alignment come from the60// component's reflect record.61kayan_column_init: proc(c: *kayan_column, info: *const reflect) -> void = {62 c*.size = info*.size;63 c*.align = info*.align;64}65 66// Row `row`'s bytes.67kayan_column_row: proc(c: *kayan_column, row: u64) -> []u8 in level = {68 return c*.data[row * c*.size..(row + 1) * c*.size];69}70 71// A new block with room for `capacity` rows; the first `count` rows are copied72// over. The old block stays in the arena: an arena frees everything at once,73// never one allocation, and this course accepts the waste. Trusted: the block74// has to be aligned for the component, which only the arena can promise.75kayan_column_grow: proc(r: region level, c: *kayan_column, count: u64, capacity: u64) -> void = {76 unsafe {77 raw: ^u8 = cast(memops_arena_push_zero(r, c*.size * capacity, c*.align), ^u8);78 bytes: []u8 in level = cast(raw[0..c*.size * capacity], []u8 in level);79 // Slicing checks both ends: a count past either block stops the program.80 kept: u64 = c*.size * count;81 slice<u8>copy(bytes[0..kept], c*.data[0..kept]);82 c*.data = bytes;83 }84}85 86kayan_column_copy_row: proc(c: *kayan_column, to: u64, from: u64) -> void = {87 for (i: u64 = 0; i < c*.size; i += 1) {88 c*.data[to * c*.size + i] = c*.data[from * c*.size + i];89 }90}91 92kayan_column_zero_row: proc(c: *kayan_column, row: u64) -> void = {93 for (i: u64 = 0; i < c*.size; i += 1) {94 c*.data[row * c*.size + i] = 0;95 }96}97 98// The typed view: the column's first `count` rows as `[]T`. This is the one99// place bytes become a type, so it takes nothing on trust. T must be the100// column's size, and plain data: any bytes are then a valid T, so reading the101// wrong type gives wrong numbers, never a bad pointer. The rows must fit, which102// the slice knows and no field can lie about, and the bytes must be aligned for103// T. The view is `[]T in level`: the bytes are in the region, so the view is104// good memory for as long as the region is. After the column grows it is the105// old block, still readable, no longer the column: take the view again.106kayan_column_slice: proc<T>(c: *const kayan_column, count: u64) -> []T in level = {107 if (c*.size != sizeof(T)) {108 printfmt("kayan: a column of {}-byte rows read as a {}-byte type\n", c*.size, sizeof(T));109 exit(1);110 }111 if (!reflect_is_plain(T<>&)) {112 printfmt("kayan: a column holds plain data, and this {}-byte type is not\n", sizeof(T));113 exit(1);114 }115 if (count > c*.data.length / sizeof(T)) {116 printfmt("kayan: a view of {} rows on a column with room for {}\n", count, c*.data.length / sizeof(T));117 exit(1);118 }119 unsafe {120 if (cast(c*.data.data, usize) % alignof(T) != 0) {121 printfmt("kayan: a column's bytes are not aligned to {}\n", alignof(T));122 exit(1);123 }124 raw: ^T = cast(c*.data.data, ^T);125 return cast(raw[0..count], []T in level);126 }127}128 129// ---------------------------------------------------------------------------130// Using it131// ---------------------------------------------------------------------------132 133print_bytes: proc(c: *kayan_column, row: u64) -> void = {134 bytes: []u8 = kayan_column_row(c, row);135 for (i: u64 = 0; i < bytes.length; i += 1) {136 printfmt(" ");137 print_hex_byte(bytes[i]);138 }139}140 141// Two hex digits; printfmt has no `%02x`.142print_hex_byte: proc(b: u8) -> void = {143 digits: [16]^const char = { "0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "a", "b", "c", "d", "e", "f" };144 printfmt("{}{}", digits[b shr 4], digits[b & 15]);145}146 147print_rows: proc(positions: *kayan_column, healths: *kayan_column, count: u64) -> void = {148 ps: []position = kayan_column_slice<position>(positions, count);149 hs: []health = kayan_column_slice<health>(healths, count);150 for (row: u64 = 0; row < count; row += 1) {151 // Read the row into values, then print the values: the checker treats152 // what goes into a printfmt in a loop as a possible change to its source.153 p: position = ps[row];154 h: health = hs[row];155 printfmt(" row {} position ({:5.2}, {:5.2}) health {:3.0} / {:3.0}\n", cast(row, u32),156 cast(p.at.x, f64), cast(p.at.y, f64), cast(h.points, f64), cast(h.max, f64));157 }158}159 160main: proc() -> i32 = {161 level: region = {162 // Two columns, made from nothing but reflect records. They live in the163 // region, since their bytes do.164 positions: *kayan_column in level = new<kayan_column>(level);165 healths: *kayan_column in level = new<kayan_column>(level);166 kayan_column_init(positions, position<>&);167 kayan_column_init(healths, health<>&);168 printfmt("a position column has {}-byte rows, a health column {}-byte rows\n",169 positions*.size, healths*.size);170 171 // Count and capacity, kept by hand here. 2.3 moves them into a table.172 count: u64 = 0;173 capacity: u64 = 0;174 175 for (i: u64 = 0; i < 5; i += 1) {176 if (count == capacity) {177 capacity = capacity == 0 ? 2 : capacity * 2;178 kayan_column_grow(level, positions, count, capacity);179 kayan_column_grow(level, healths, count, capacity);180 printfmt(" grew to room for {} rows\n", capacity);181 }182 row: u64 = count;183 count += 1;184 kayan_column_zero_row(positions, row);185 kayan_column_zero_row(healths, row);186 // Back to types to write the values.187 ps: []position = kayan_column_slice<position>(positions, count);188 hs: []health = kayan_column_slice<health>(healths, count);189 ps[row] = { .at = { .x = cast(i, f32), .y = 1.0 } };190 hs[row] = { .points = 50.0 + 10.0 * cast(i, f32), .max = 100.0 };191 }192 printfmt("five rows, written through the typed view:\n");193 print_rows(positions, healths, count);194 195 printfmt("row 1's health as the column stores it, bytes:");196 print_bytes(healths, 1);197 printfmt("\n that is 60.0 and 100.0 as little-endian f32s; the column never knew\n");198 199 // Swap-remove row 1 from both columns: the same byte copy for both, sized by200 // each column, so a row stays a row across columns.201 last: u64 = count - 1;202 kayan_column_copy_row(positions, 1, last);203 kayan_column_copy_row(healths, 1, last);204 count -= 1;205 printfmt("swap-removed row 1: row {} moved into it, in both columns\n", last);206 print_rows(positions, healths, count);207 }208 return 0;209}What columns prints. tools/check.py fails the chapter if a single byte differs.
a position column has 8-byte rows, a health column 8-byte rows
grew to room for 2 rows
grew to room for 4 rows
grew to room for 8 rows
five rows, written through the typed view:
row 0 position ( 0.00, 1.00) health 50 / 100
row 1 position ( 1.00, 1.00) health 60 / 100
row 2 position ( 2.00, 1.00) health 70 / 100
row 3 position ( 3.00, 1.00) health 80 / 100
row 4 position ( 4.00, 1.00) health 90 / 100
row 1's health as the column stores it, bytes: 00 00 70 42 00 00 c8 42
that is 60.0 and 100.0 as little-endian f32s; the column never knew
swap-removed row 1: row 4 moved into it, in both columns
row 0 position ( 0.00, 1.00) health 50 / 100
row 1 position ( 4.00, 1.00) health 90 / 100
row 2 position ( 2.00, 1.00) health 70 / 100
row 3 position ( 3.00, 1.00) health 80 / 100
Build an ECS · part 2, components · 2.3
A table
ecs/src/2.components/2.3.tables/tables.rin
In the dojo: 2. swap-remove and the back-reference, 4. a type-erased column, each on its own, to watch and to practise.
The two halves meet. Part 1 built the entity index: handles, slots, packed rows, swap-remove. 2.1 and 2.2 built the component side: ids, a registry, columns of bytes. A table is 1.4's packed rows with the typed arrays replaced by columns, one per component in its signature:
table {position velocity health}
entities [ 0:1 | 3:1 | 2:1 | ... ] who owns each row
columns position [ ... ] velocity [ ... ] health [ ... ]
count, capacity shared by every column
Row r is entity entities[r], and its components are row r of every column. This world still has a single table, so every entity has the same components; 2.4 gives each combination its own.
Step 1: the table
kayan_table: struct = {
signature: u64;
count: u64;
capacity: u64;
entities: []kayan_entity in level;
columns: [64]kayan_column;
}
signaturesays which components the table has.columns[id]is in use when the signature has bitid. The rest stay zeroed. 64 slots indexed by id means finding a column is an index, never a search. kayan in C looked columns up by name in a hash map, which is the thing this replaces.countandcapacitybelong to the table, shared by every column and byentities, so they can't disagree.
A table is made from a signature and the registry:
kayan_table_init: proc(w: *kayan_world in level, t: *kayan_table, signature: u64) -> void = {
t*.signature = signature;
for (id: u32 = 0; id < 64; id += 1) {
if ((signature & kayan_bit(id)) != 0) {
info: option<*const reflect in permanent> = id < w*.component_count ? w*.infos[id] : null;
if (!info) {
printfmt("kayan: signature {} uses component {}, which was never registered\n", signature, id);
exit(1);
} else {
kayan_column_init(t*.columns[id]&, info);
}
}
}
}
The signature is the column list. In the C kayan, the signature and the columns were written separately and could disagree (archetype D had position columns its signature didn't mention). Here one is made from the other. A registry slot may be null (2.1), so the test isn't optional: the checker won't let info reach kayan_column_init until the empty case has been dealt with.
Step 2: rows, for every column at once
Adding a row grows every column together when the table is full, then zeroes the new row in each:
kayan_table_add_row: proc(r: region level, w: *kayan_world in level, t: *kayan_table, e: kayan_entity) -> u64 = {
if (t*.count == t*.capacity) {
kayan_table_grow(r, w, t);
}
row: u64 = t*.count;
t*.count += 1;
t*.entities[row] = e;
for (id: u32 = 0; id < w*.component_count; id += 1) {
if (kayan_table_has(t, id)) {
kayan_column_zero_row(t*.columns[id]&, row);
}
}
return row;
}
Removing a row is 1.4's swap-remove, for every column:
kayan_table_remove_row: proc(w: *kayan_world in level, t: *kayan_table, row: u64) -> void = {
last: u64 = t*.count - 1;
if (row != last) {
for (id: u32 = 0; id < w*.component_count; id += 1) {
if (kayan_table_has(t, id)) {
kayan_column_copy_row(t*.columns[id]&, row, last);
}
}
moved: kayan_entity = t*.entities[last];
t*.entities[row] = moved;
vec<kayan_slot>at(w*.slots, cast(moved.index, u64))*.row = cast(row, u32);
}
t*.count = last;
}
Compare it with 1.4's kayan_despawn: the same moves and the same slot fix, but over columns of bytes instead of named arrays. This proc works for any table that will ever exist.
Compare the two signatures, too. Adding a row can allocate, so it takes r: region level. Removing one can't, so it doesn't. Whether a proc can allocate is written where you can see it.
Step 3: the typed view of a table
kayan_slice: proc<T>(t: *kayan_table in level) -> []T in level = {
id: u32 = kayan_id_of<T>();
if (!kayan_table_has(t, id)) {
printfmt("kayan: the table has no column for component {}\n", id);
exit(1);
}
return kayan_column_slice<T>(t*.columns[id]&, t*.count);
}
kayan_slice<position>(t) is the table's position column as a []position, cut to the rows in use. Systems will live on this in Part 3: get the slices, loop over rows. Notice that this proc is checked: unsafe { } is still only in the two column procs from 2.2, and nothing above them needs it.
Step 4: the world
The world moves off main's stack and into the region, with everything it holds:
kayan_world: struct = {
infos: [64]option<*const reflect in permanent>;
component_count: u32;
slots: *vec<kayan_slot> in level;
free: *vec<u32> in level;
live: u64;
table: kayan_table;
}
The registry from 2.1, the slots and free list from Part 1 (as vecs now, so the 8-entity limit is gone), and one table.
*vec<kayan_slot> in level is a plain pointer, not an option: it may not be null, ever, so every proc can use it without a test. The price is that the world can't exist for a moment without its vecs, so it's made whole, in one expression:
kayan_world_make: proc(r: region level) -> *kayan_world in level = {
return new<kayan_world>(r, {
.slots = vec<kayan_slot>make(r, 8),
.free = vec<u32>make(r, 8),
});
}
Leave one out and it doesn't compile:
type error: field 'free' is left zero, and it holds a pointer that may not be null; set it here, or make it 'option<*T>'
From here on, every proc takes the world as *kayan_world in level. It lives there, so a pointer to its table is a level pointer too, good for as long as the level is.
Spawning is Part 1's spawn with the row coming from the table:
row: u64 = kayan_table_add_row(r, w, w*.table&, e);
vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.row = cast(row, u32);
w*.live += 1;
return e;
and despawning is Part 1's despawn calling kayan_table_remove_row.
Look at the order in that sample. The pointer into slots is taken after adding the row, and used at once. slots is a vec, and a vec's items move when it grows, so a pointer into one is only good until the next thing that might grow it. Adding a row doesn't touch slots here, but the habit costs nothing, and in 2.4 the checker will insist on it.
Step 5: writing a new entity's data
There's no typed get or set yet (that's 2.5), so the program finds the row and writes through the column slices:
row: u64 = kayan_row(w, drones[i]);
ps: []position in level = kayan_slice<position>(w*.table&);
vs: []velocity in level = kayan_slice<velocity>(w*.table&);
hs: []health in level = kayan_slice<health>(w*.table&);
ps[row] = { .at = { .x = 10.0 * cast(i, f32), .y = 0.0 } };
Run it
spawned 6 drones into one table; it grew from 4 rows to 8
6 rows, room for 8
row 0 0:1 position ( 0.00, 0.00) health 100
...
despawn 0:1 and 3:1
4 rows, room for 8
row 0 5:1 position (50.00, 0.00) health 100
row 1 1:1 position (10.00, 0.00) health 100
row 2 2:1 position (20.00, 0.00) health 100
row 3 4:1 position (40.00, 0.00) health 100
spawn 3:2: a reused number, the next row, zeroed except what we set
...
kayan_verify: 0 problems
Row 0 went to 5:1 (the last row moved into the hole), then row 3 went to 4:1 the same way. Every column moved together, and the slots followed.
Try
- Remove
velocityfrom the table's signature and run it. Which line fails, and why is failing there the right behaviour? - Make
kayan_table_growstart at 1 row. Count the grows for 6 drones. - Delete the slot fix in
kayan_table_remove_rowand runkayan_verify. - Leave
.freeout ofkayan_world_makeand compile. Then make the fieldoption<*vec<u32> in level>, as the message suggests. How many procs now have to test it?
1// 2.3 A table2//3// Two halves meet. Part 1 built the entity index: handles, slots, packed rows,4// swap-remove. 2.1 and 2.2 built the component side: ids, a registry, columns5// of bytes. A table is the packed rows of 1.4 with their typed arrays replaced6// by columns, one per component in the table's signature:7//8// table {position velocity health}9// entities [ 0:1 | 3:1 | 2:1 | ... ] who owns each row10// columns position [ ... ] velocity [ ... ] health [ ... ]11// count, capacity shared by every column12//13// Row r is entity entities[r], and its components are row r of every column.14// Adding a row grows every column together, and swap-remove copies the last15// row over the hole in every column together, so a row stays one entity.16//17// This world still has a single table, so every entity has the same18// components. 2.4 gives each combination its own.19//20// The arena is the `level` region: the world, its vecs and21// every column live in it, and the procs that allocate take it as `r: region22// level`. The two trusted procs are the column's, from 2.2. Everything else is23// checked, which costs two things worth noticing below: registry slots are24// `option<*const reflect>`, and a pointer into the slots is taken again after a25// call that may grow them. Every26// proc takes the world as `*kayan_world in level`: it lives there, so a27// pointer to its table is a level pointer too, good for as long as the level.28 29import "std/print.rin"30import "std/memops.rin"31import "std/vec.rin"32import "std/slice.rin"33 34level: region;35 36vec2: struct = {37 x: f32;38 y: f32;39}40 41position: struct = { at: vec2; }42velocity: struct = { per_second: vec2; }43health: struct = { points: f32; max: f32; }44 45component: enum = {46 position,47 velocity,48 health,49}50 51component_of: proc<position>() -> component = { return component.position; }52component_of: proc<velocity>() -> component = { return component.velocity; }53component_of: proc<health>() -> component = { return component.health; }54 55// ---------------------------------------------------------------------------56// Entities (Part 1)57// ---------------------------------------------------------------------------58 59kayan_entity: struct = {60 index: u32;61 generation: u32;62}63 64kayan_status: enum[must_use] = {65 ok,66 dead,67}68 69kayan_slot: struct = {70 generation: u32;71 row: u32;72}73 74// ---------------------------------------------------------------------------75// Columns (2.2)76// ---------------------------------------------------------------------------77 78kayan_column: struct = {79 data: []u8 in level;80 size: u64;81 align: u64;82}83 84kayan_column_init: proc(c: *kayan_column, info: *const reflect) -> void = {85 c*.size = info*.size;86 c*.align = info*.align;87}88 89kayan_column_grow: proc(r: region level, c: *kayan_column, count: u64, capacity: u64) -> void = {90 unsafe {91 raw: ^u8 = cast(memops_arena_push_zero(r, c*.size * capacity, c*.align), ^u8);92 bytes: []u8 in level = cast(raw[0..c*.size * capacity], []u8 in level);93 // Slicing checks both ends: a count past either block stops the program.94 kept: u64 = c*.size * count;95 slice<u8>copy(bytes[0..kept], c*.data[0..kept]);96 c*.data = bytes;97 }98}99 100kayan_column_copy_row: proc(c: *kayan_column, to: u64, from: u64) -> void = {101 for (i: u64 = 0; i < c*.size; i += 1) {102 c*.data[to * c*.size + i] = c*.data[from * c*.size + i];103 }104}105 106kayan_column_zero_row: proc(c: *kayan_column, row: u64) -> void = {107 for (i: u64 = 0; i < c*.size; i += 1) {108 c*.data[row * c*.size + i] = 0;109 }110}111 112// The typed view: the column's first `count` rows as `[]T`. This is the one113// place bytes become a type, so it takes nothing on trust. T must be the114// column's size, and plain data: any bytes are then a valid T, so reading the115// wrong type gives wrong numbers, never a bad pointer. The rows must fit, which116// the slice knows and no field can lie about, and the bytes must be aligned for117// T. The view is `[]T in level`: the bytes are in the region, so the view is118// good memory for as long as the region is. After the column grows it is the119// old block, still readable, no longer the column: take the view again.120kayan_column_slice: proc<T>(c: *const kayan_column, count: u64) -> []T in level = {121 if (c*.size != sizeof(T)) {122 printfmt("kayan: a column of {}-byte rows read as a {}-byte type\n", c*.size, sizeof(T));123 exit(1);124 }125 if (!reflect_is_plain(T<>&)) {126 printfmt("kayan: a column holds plain data, and this {}-byte type is not\n", sizeof(T));127 exit(1);128 }129 if (count > c*.data.length / sizeof(T)) {130 printfmt("kayan: a view of {} rows on a column with room for {}\n", count, c*.data.length / sizeof(T));131 exit(1);132 }133 unsafe {134 if (cast(c*.data.data, usize) % alignof(T) != 0) {135 printfmt("kayan: a column's bytes are not aligned to {}\n", alignof(T));136 exit(1);137 }138 raw: ^T = cast(c*.data.data, ^T);139 return cast(raw[0..count], []T in level);140 }141}142 143// ---------------------------------------------------------------------------144// The table145// ---------------------------------------------------------------------------146 147// `columns[id]` is in use when the signature has bit `id`; the others stay148// zeroed. Count and capacity are the table's, shared by its columns and by149// `entities`, so they can never disagree.150kayan_table: struct = {151 signature: u64;152 count: u64;153 capacity: u64;154 entities: []kayan_entity in level;155 columns: [64]kayan_column;156}157 158// The world: the registry (2.1), the slots and free list (Part 1), and for now159// a single table. Slots and the free list are vecs now, so the 8-entity limit160// of Part 1 is gone; the region holds everything.161kayan_world: struct = {162 infos: [64]option<*const reflect in permanent>;163 component_count: u32;164 slots: *vec<kayan_slot> in level;165 free: *vec<u32> in level;166 live: u64;167 table: kayan_table;168}169 170kayan_bit: proc(id: u32) -> u64 = {171 return 1'u64 shl cast(id, u64);172}173 174kayan_id_of: proc<T>() -> u32 = {175 return cast(component_of<T>(), u32);176}177 178kayan_mask_of: proc<T>() -> u64 = {179 return kayan_bit(kayan_id_of<T>());180}181 182kayan_register: proc(w: *kayan_world in level, id: u32, info: *const reflect in permanent) -> void = {183 if (id >= 64) {184 printfmt("kayan: component id {} is past the 64 a signature can hold\n", id);185 exit(1);186 }187 w*.infos[id] = info;188 if (id + 1 > w*.component_count) {189 w*.component_count = id + 1;190 }191}192 193kayan_table_has: proc(t: *kayan_table, id: u32) -> b32 = {194 return (t*.signature & kayan_bit(id)) != 0;195}196 197// A table for a signature: one empty column per component in it, sized from198// the registry. Every component in the signature must be registered.199kayan_table_init: proc(w: *kayan_world in level, t: *kayan_table, signature: u64) -> void = {200 t*.signature = signature;201 for (id: u32 = 0; id < 64; id += 1) {202 if ((signature & kayan_bit(id)) != 0) {203 info: option<*const reflect in permanent> = id < w*.component_count ? w*.infos[id] : null;204 if (!info) {205 printfmt("kayan: signature {} uses component {}, which was never registered\n", signature, id);206 exit(1);207 } else {208 kayan_column_init(t*.columns[id]&, info);209 }210 }211 }212}213 214// Double the room in `entities` and in every column at once.215kayan_table_grow: proc(r: region level, w: *kayan_world in level, t: *kayan_table) -> void = {216 capacity: u64 = t*.capacity == 0 ? 4 : t*.capacity * 2;217 entities: []kayan_entity in level = new_slice<kayan_entity>(r, capacity);218 for (row: u64 = 0; row < t*.count; row += 1) {219 entities[row] = t*.entities[row];220 }221 t*.entities = entities;222 for (id: u32 = 0; id < w*.component_count; id += 1) {223 if (kayan_table_has(t, id)) {224 kayan_column_grow(r, t*.columns[id]&, t*.count, capacity);225 }226 }227 t*.capacity = capacity;228}229 230// A new row at the end, owned by `e`, with every component zeroed.231kayan_table_add_row: proc(r: region level, w: *kayan_world in level, t: *kayan_table, e: kayan_entity) -> u64 = {232 if (t*.count == t*.capacity) {233 kayan_table_grow(r, w, t);234 }235 row: u64 = t*.count;236 t*.count += 1;237 t*.entities[row] = e;238 for (id: u32 = 0; id < w*.component_count; id += 1) {239 if (kayan_table_has(t, id)) {240 kayan_column_zero_row(t*.columns[id]&, row);241 }242 }243 return row;244}245 246// 1.4's swap-remove, for every column: the last row is copied over `row`, and247// the slot of the entity that owned the last row is told where it went.248kayan_table_remove_row: proc(w: *kayan_world in level, t: *kayan_table, row: u64) -> void = {249 last: u64 = t*.count - 1;250 if (row != last) {251 for (id: u32 = 0; id < w*.component_count; id += 1) {252 if (kayan_table_has(t, id)) {253 kayan_column_copy_row(t*.columns[id]&, row, last);254 }255 }256 moved: kayan_entity = t*.entities[last];257 t*.entities[row] = moved;258 vec<kayan_slot>at(w*.slots, cast(moved.index, u64))*.row = cast(row, u32);259 }260 t*.count = last;261}262 263// The typed view of one of the table's columns: T's rows, all of them in use.264// Asking for a component the table does not have is a bug, so it stops.265kayan_slice: proc<T>(t: *kayan_table in level) -> []T in level = {266 id: u32 = kayan_id_of<T>();267 if (!kayan_table_has(t, id)) {268 printfmt("kayan: the table has no column for component {}\n", id);269 exit(1);270 }271 return kayan_column_slice<T>(t*.columns[id]&, t*.count);272}273 274// ---------------------------------------------------------------------------275// The world276// ---------------------------------------------------------------------------277 278// The world is made whole in its region: the two vecs it holds may not be279// null, so they are made with it.280kayan_world_make: proc(r: region level) -> *kayan_world in level = {281 return new<kayan_world>(r, {282 .slots = vec<kayan_slot>make(r, 8),283 .free = vec<u32>make(r, 8),284 });285}286 287kayan_alive: proc(w: *kayan_world in level, e: kayan_entity) -> b32 = {288 if (cast(e.index, u64) >= w*.slots*.length) {289 return false;290 }291 return vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.generation == e.generation;292}293 294// Part 1's spawn, with the row coming from the table.295kayan_spawn: proc(r: region level, w: *kayan_world in level) -> kayan_entity = {296 e: kayan_entity = {};297 if (!vec<u32>is_empty(w*.free)) {298 e.index = vec<u32>pop(w*.free);299 } else {300 e.index = cast(w*.slots*.length, u32);301 _ = vec<kayan_slot>push(w*.slots, { .generation = 1 });302 }303 e.generation = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.generation;304 // Adding the row is a call that is handed the world, so the pointer into305 // the slots is taken after it, not before.306 row: u64 = kayan_table_add_row(r, w, w*.table&, e);307 vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.row = cast(row, u32);308 w*.live += 1;309 return e;310}311 312kayan_despawn: proc(w: *kayan_world in level, e: kayan_entity) -> kayan_status = {313 if (!kayan_alive(w, e)) {314 return kayan_status.dead;315 }316 row: u64 = cast(vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.row, u64);317 kayan_table_remove_row(w, w*.table&, row);318 vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.generation += 1;319 _ = vec<u32>push(w*.free, e.index);320 w*.live -= 1;321 return kayan_status.ok;322}323 324// The row a live entity is in. Typed get and set arrive in 2.5; until then,325// game code finds the row and indexes the table's columns.326kayan_row: proc(w: *kayan_world in level, e: kayan_entity) -> u64 = {327 return cast(vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.row, u64);328}329 330// 1.4's invariant, unchanged: every row belongs to a live entity whose slot331// points back at it, and the rows add up to the live count.332kayan_verify: proc(w: *kayan_world in level) -> u64 = {333 problems: u64 = 0;334 t: *kayan_table in level = w*.table&;335 for (row: u64 = 0; row < t*.count; row += 1) {336 e: kayan_entity = t*.entities[row];337 if (!kayan_alive(w, e)) {338 printfmt("kayan_verify: row {} holds dead entity {}:{}\n", row, e.index, e.generation);339 problems += 1;340 } else if (kayan_row(w, e) != row) {341 printfmt("kayan_verify: {}:{} is in row {}, its slot says row {}\n",342 e.index, e.generation, row, kayan_row(w, e));343 problems += 1;344 }345 }346 if (t*.count != w*.live) {347 printfmt("kayan_verify: {} rows, {} live entities\n", t*.count, w*.live);348 problems += 1;349 }350 return problems;351}352 353// ---------------------------------------------------------------------------354// The program355// ---------------------------------------------------------------------------356 357print_table: proc(w: *kayan_world in level) -> void = {358 t: *kayan_table in level = w*.table&;359 ps: []position in level = kayan_slice<position>(t);360 hs: []health in level = kayan_slice<health>(t);361 printfmt(" {} rows, room for {}\n", t*.count, t*.capacity);362 for (row: u64 = 0; row < t*.count; row += 1) {363 e: kayan_entity = t*.entities[row];364 p: position = ps[row];365 h: health = hs[row];366 printfmt(" row {} {}:{} position ({:5.2}, {:5.2}) health {:3.0}\n", cast(row, u32), e.index, e.generation,367 cast(p.at.x, f64), cast(p.at.y, f64), cast(h.points, f64));368 }369}370 371main: proc() -> i32 = {372 level: region = {373 w: *kayan_world in level = kayan_world_make(level);374 kayan_register(w, kayan_id_of<position>(), position<>&);375 kayan_register(w, kayan_id_of<velocity>(), velocity<>&);376 kayan_register(w, kayan_id_of<health>(), health<>&);377 378 // One table, so one kind of entity: every entity is a drone.379 kayan_table_init(w, w*.table&, kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<health>());380 381 drones: [6]kayan_entity = {};382 for (i: u32 = 0; i < 6; i += 1) {383 drones[i] = kayan_spawn(level, w);384 // Write the new row through the typed views of its columns.385 row: u64 = kayan_row(w, drones[i]);386 ps: []position in level = kayan_slice<position>(w*.table&);387 vs: []velocity in level = kayan_slice<velocity>(w*.table&);388 hs: []health in level = kayan_slice<health>(w*.table&);389 ps[row] = { .at = { .x = 10.0 * cast(i, f32), .y = 0.0 } };390 vs[row] = { .per_second = { .x = 0.0, .y = 1.0 } };391 hs[row] = { .points = 100.0, .max = 100.0 };392 }393 printfmt("spawned 6 drones into one table; it grew from 4 rows to 8\n");394 print_table(w);395 396 printfmt("despawn {}:{} and {}:{}\n", drones[0].index, drones[0].generation, drones[3].index, drones[3].generation);397 _ = kayan_despawn(w, drones[0]);398 _ = kayan_despawn(w, drones[3]);399 print_table(w);400 401 again: kayan_entity = kayan_spawn(level, w);402 hs: []health in level = kayan_slice<health>(w*.table&);403 hs[kayan_row(w, again)] = { .points = 40.0, .max = 100.0 };404 printfmt("spawn {}:{}: a reused number, the next row, zeroed except what we set\n",405 again.index, again.generation);406 print_table(w);407 408 printfmt("kayan_verify: {} problems\n", kayan_verify(w));409 }410 return 0;411}What tables prints. tools/check.py fails the chapter if a single byte differs.
spawned 6 drones into one table; it grew from 4 rows to 8
6 rows, room for 8
row 0 0:1 position ( 0.00, 0.00) health 100
row 1 1:1 position (10.00, 0.00) health 100
row 2 2:1 position (20.00, 0.00) health 100
row 3 3:1 position (30.00, 0.00) health 100
row 4 4:1 position (40.00, 0.00) health 100
row 5 5:1 position (50.00, 0.00) health 100
despawn 0:1 and 3:1
4 rows, room for 8
row 0 5:1 position (50.00, 0.00) health 100
row 1 1:1 position (10.00, 0.00) health 100
row 2 2:1 position (20.00, 0.00) health 100
row 3 4:1 position (40.00, 0.00) health 100
spawn 3:2: a reused number, the next row, zeroed except what we set
5 rows, room for 8
row 0 5:1 position (50.00, 0.00) health 100
row 1 1:1 position (10.00, 0.00) health 100
row 2 2:1 position (20.00, 0.00) health 100
row 3 4:1 position (40.00, 0.00) health 100
row 4 3:2 position ( 0.00, 0.00) health 40
kayan_verify: 0 problems
Build an ECS · part 2, components · 2.4
Many tables: archetypes
ecs/src/2.components/2.4.archetypes/archetypes.rin
In the dojo: 3. the slot map, 8. index, not pointer, 9. interning, each on its own, to watch and to practise.
2.3's world had one table, so every entity had the same components. Real games mix: a turret has a position and a turret component, a drone has a position, velocity and health, a bullet has a position, velocity, damage and lifetime. Each combination gets a table of its own. Every row of a table has every component in its signature, so there are no holes and no flags. A table for one combination is called an archetype.
table 0 {position turret} 0:1
table 1 {position velocity health} 1:1 2:1 3:1
table 2 {position velocity damage lifetime} 4:1 5:1
Nobody declares the archetypes. The first time something spawns with a new combination, the world makes its table and remembers it by signature, so the next spawn finds it.
Step 1: slots say which table
kayan_slot: struct = {
generation: u32;
table: u32;
row: u32;
}
Handle to slot to (table, row). That's the whole entity index from here on.
Step 2: the world keeps every table
kayan_world: struct = {
infos: [64]option<*const reflect in permanent>;
component_count: u32;
tables: *vec<*kayan_table in level> in level;
by_signature: *hashmap<u64, u32> in level;
slots: *vec<kayan_slot> in level;
free: *vec<u32> in level;
live: u64;
}
tables holds them all, and by_signature maps a signature to its table's index.
Look at the type of tables: a vec of pointers to tables. The obvious shape is a vec<kayan_table>, with the tables themselves side by side. It has a classic bug. Making a new table can grow the vec and move every table in memory, so a pointer to a table is only good until the next table is made. It works until the first new archetype appears mid-game. The checker would catch that one, calling any such pointer kept across a push "may be stale", but then every proc holding a table would be fighting it.
So instead of being careful, the design removes the problem. Each table is allocated once, on its own, in the region, and a table never moves. The vec of pointers may grow, but what it points at stays where it is, so *kayan_table in level means just what it says: good for as long as the level is, across anything. That costs one extra pointer hop per table, not per row, and the loops that matter never see it.
kayan_table_at: proc(w: *kayan_world in level, index: u32) -> *kayan_table in level = {
return vec<*kayan_table in level>at(w*.tables, cast(index, u64))*;
}
Slots still name their table by index. A slot is plain data: 4 bytes rather than 8, no region to name, and an index can be stored anywhere.
Step 3: where archetypes come from
kayan_table_for: proc(r: region level, w: *kayan_world in level, signature: u64) -> u32 = {
found: option<*u32> = hashmap<u64, u32>get(w*.by_signature, signature);
if (found) {
return found*;
}
index: u32 = cast(w*.tables*.length, u32);
t: *kayan_table in level = new<kayan_table>(r);
kayan_table_init(w, t, signature);
_ = vec<*kayan_table in level>push(w*.tables, t);
_ = hashmap<u64, u32>set(w*.by_signature, signature, index);
return index;
}
Found, or made. A game has tens of archetypes, maybe a few hundred, so a linear search would do too. The hashmap costs two lines with rin's std, and 2.6 makes the lookup rare anyway. A lookup that can miss returns an option, the same as get_position in 1.3.
Step 4: spawning takes a signature
kayan_spawn: proc(r: region level, w: *kayan_world in level, signature: u64) -> kayan_entity = {
and puts the new entity in that signature's table:
table_index: u32 = kayan_table_for(r, w, signature);
t: *kayan_table in level = kayan_table_at(w, table_index);
row: u64 = kayan_table_add_row(r, w, t, e);
// Both calls were handed the world, so the pointer into the slots is
// taken after them.
slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));
slot*.table = table_index;
slot*.row = cast(row, u32);
Note the order. t isn't the pointer to worry about any more, since tables don't move. slot is: slots are items in a vec, right there in the vec's storage. Take it before those two calls, and the checker stops you:
type error: 'slot' may be stale: 'w.slots', which it was taken from, has been changed since; take it again
kayan_table_for grows tables, not slots, but the checker doesn't track which of the world's vecs a call changes. It sees a call that may change the world, and it ends every borrow from that world. So the rule is simple: do the calls first, take the pointer into a vec last, and use it right away.
Despawning finds the table through the slot, and avoids the question another way:
slot: kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*;
kayan_table_remove_row(w, kayan_table_at(w, slot.table), cast(slot.row, u64));
The * at the end copies the slot out. A copy can't go stale, and it's two numbers and a generation, so that's nearly always the easy way.
Step 5: the invariant, over every table
kayan_verify now walks every table: each row belongs to a live entity whose slot points back at that table and row, and the rows of all tables add up to the live count. Same invariant, one more field.
Step 6: one column, many tables
The end of the program reads every position in the world:
for (ti: u32 = 0; ti < cast(w*.tables*.length, u32); ti += 1) {
t: *kayan_table in level = kayan_table_at(w, ti);
ps: []position in level = kayan_slice<position>(t);
The position column of each table, in turn. Positions are no longer one array; they're one array per archetype that has them. This is exactly what a query does in Part 3: visit the tables that have the right components, and stream their columns.
Run it
spawned the turret: the world made table 0 for it
spawned 3 drones: the first made a table, the others found it
spawned 2 bullets
6 entities in 3 tables:
table 0 {position turret}: 0:1
table 1 {position velocity health}: 1:1 2:1 3:1
table 2 {position velocity damage lifetime}: 4:1 5:1
despawn drone 1:1: a swap-remove inside its own table
table 0 {position turret}: 0:1
table 1 {position velocity health}: 3:1 2:1
table 2 {position velocity damage lifetime}: 4:1 5:1
every position, table by table:
...
kayan_verify: 0 problems
Try
- Spawn a "drone with damage" (position, velocity, health, damage). How many tables now?
- Spawn 100 entities with 100 different signatures (use the bits of a counter). What does that do to a system that visits tables? This is called archetype fragmentation.
- In
kayan_spawn, takeslotbeforekayan_table_for, and read the error. Then get a table pointer beforekayan_table_for, spawn something with a new signature, and use the pointer afterwards. Why does the checker have no complaint this time?
1// 2.4 Many tables: archetypes2//3// 2.3's world had one table, so every entity had the same components. Real4// games mix: a turret has a position and a turret component, a drone a5// position, velocity and health, a bullet a position, velocity, damage and a6// lifetime. Each combination gets a table of its own, and every row of a table7// has every component in its signature, so there are no holes and no flags.8// A table for one combination is called an archetype.9//10// table 0 {position turret} 0:111// table 1 {position velocity health} 1:1 2:1 3:112// table 2 {position velocity damage lifetime} 4:1 5:113//14// Nobody declares the archetypes. The first time something spawns with a new15// combination, the world makes its table, and remembers it by signature so the16// next spawn finds it. A slot now says which table an entity is in as well as17// which row.18//19// The world keeps a pointer to each table, `*kayan_table in20// level`, rather than the tables themselves in a vec: a table never moves, so21// a pointer to one is good for as long as the level is and may be kept across22// anything. The slot still names the table by index. The two trusted procs23// are the column's, from 2.2.24 25import "std/print.rin"26import "std/memops.rin"27import "std/vec.rin"28import "std/hashmap.rin"29import "std/slice.rin"30 31level: region;32 33vec2: struct = {34 x: f32;35 y: f32;36}37 38position: struct = { at: vec2; }39velocity: struct = { per_second: vec2; }40health: struct = { points: f32; max: f32; }41damage: struct = { amount: f32; }42lifetime: struct = { seconds: f32; }43turret: struct = { cooldown: f32; timer: f32; }44 45component: enum = {46 position,47 velocity,48 health,49 damage,50 lifetime,51 turret,52}53 54component_of: proc<position>() -> component = { return component.position; }55component_of: proc<velocity>() -> component = { return component.velocity; }56component_of: proc<health>() -> component = { return component.health; }57component_of: proc<damage>() -> component = { return component.damage; }58component_of: proc<lifetime>() -> component = { return component.lifetime; }59component_of: proc<turret>() -> component = { return component.turret; }60 61// ---------------------------------------------------------------------------62// Entities (Part 1)63// ---------------------------------------------------------------------------64 65kayan_entity: struct = {66 index: u32;67 generation: u32;68}69 70kayan_status: enum[must_use] = {71 ok,72 dead,73}74 75// Where an entity lives: which table, and which row of it.76kayan_slot: struct = {77 generation: u32;78 table: u32;79 row: u32;80}81 82// ---------------------------------------------------------------------------83// Columns (2.2)84// ---------------------------------------------------------------------------85 86kayan_column: struct = {87 data: []u8 in level;88 size: u64;89 align: u64;90}91 92kayan_column_init: proc(c: *kayan_column, info: *const reflect) -> void = {93 c*.size = info*.size;94 c*.align = info*.align;95}96 97kayan_column_grow: proc(r: region level, c: *kayan_column, count: u64, capacity: u64) -> void = {98 unsafe {99 raw: ^u8 = cast(memops_arena_push_zero(r, c*.size * capacity, c*.align), ^u8);100 bytes: []u8 in level = cast(raw[0..c*.size * capacity], []u8 in level);101 // Slicing checks both ends: a count past either block stops the program.102 kept: u64 = c*.size * count;103 slice<u8>copy(bytes[0..kept], c*.data[0..kept]);104 c*.data = bytes;105 }106}107 108kayan_column_copy_row: proc(c: *kayan_column, to: u64, from: u64) -> void = {109 for (i: u64 = 0; i < c*.size; i += 1) {110 c*.data[to * c*.size + i] = c*.data[from * c*.size + i];111 }112}113 114kayan_column_zero_row: proc(c: *kayan_column, row: u64) -> void = {115 for (i: u64 = 0; i < c*.size; i += 1) {116 c*.data[row * c*.size + i] = 0;117 }118}119 120// The typed view: the column's first `count` rows as `[]T`. This is the one121// place bytes become a type, so it takes nothing on trust. T must be the122// column's size, and plain data: any bytes are then a valid T, so reading the123// wrong type gives wrong numbers, never a bad pointer. The rows must fit, which124// the slice knows and no field can lie about, and the bytes must be aligned for125// T. The view is `[]T in level`: the bytes are in the region, so the view is126// good memory for as long as the region is. After the column grows it is the127// old block, still readable, no longer the column: take the view again.128kayan_column_slice: proc<T>(c: *const kayan_column, count: u64) -> []T in level = {129 if (c*.size != sizeof(T)) {130 printfmt("kayan: a column of {}-byte rows read as a {}-byte type\n", c*.size, sizeof(T));131 exit(1);132 }133 if (!reflect_is_plain(T<>&)) {134 printfmt("kayan: a column holds plain data, and this {}-byte type is not\n", sizeof(T));135 exit(1);136 }137 if (count > c*.data.length / sizeof(T)) {138 printfmt("kayan: a view of {} rows on a column with room for {}\n", count, c*.data.length / sizeof(T));139 exit(1);140 }141 unsafe {142 if (cast(c*.data.data, usize) % alignof(T) != 0) {143 printfmt("kayan: a column's bytes are not aligned to {}\n", alignof(T));144 exit(1);145 }146 raw: ^T = cast(c*.data.data, ^T);147 return cast(raw[0..count], []T in level);148 }149}150 151// ---------------------------------------------------------------------------152// The table153// ---------------------------------------------------------------------------154 155// `columns[id]` is in use when the signature has bit `id`; the others stay156// zeroed. Count and capacity are the table's, shared by its columns and by157// `entities`, so they can never disagree.158kayan_table: struct = {159 signature: u64;160 count: u64;161 capacity: u64;162 entities: []kayan_entity in level;163 columns: [64]kayan_column;164}165 166// The world: the registry, the slots and free list, and every table, with a167// map from signature to table index so finding one is a lookup, not a search.168// Tables are referred to by index in the slots, and by pointer everywhere169// else: a table is made once in the region and never moves.170kayan_world: struct = {171 infos: [64]option<*const reflect in permanent>;172 component_count: u32;173 tables: *vec<*kayan_table in level> in level;174 by_signature: *hashmap<u64, u32> in level;175 slots: *vec<kayan_slot> in level;176 free: *vec<u32> in level;177 live: u64;178}179 180kayan_bit: proc(id: u32) -> u64 = {181 return 1'u64 shl cast(id, u64);182}183 184kayan_id_of: proc<T>() -> u32 = {185 return cast(component_of<T>(), u32);186}187 188kayan_mask_of: proc<T>() -> u64 = {189 return kayan_bit(kayan_id_of<T>());190}191 192kayan_register: proc(w: *kayan_world in level, id: u32, info: *const reflect in permanent) -> void = {193 if (id >= 64) {194 printfmt("kayan: component id {} is past the 64 a signature can hold\n", id);195 exit(1);196 }197 w*.infos[id] = info;198 if (id + 1 > w*.component_count) {199 w*.component_count = id + 1;200 }201}202 203// A component's name, for printing: a registered id's, or "?" for a slot204// nobody filled.205kayan_name: proc(w: *kayan_world in level, id: u32) -> ^const char = {206 info: option<*const reflect in permanent> = w*.infos[id];207 return info ? info*.name : "?";208}209 210kayan_table_has: proc(t: *kayan_table in level, id: u32) -> b32 = {211 return (t*.signature & kayan_bit(id)) != 0;212}213 214// A table for a signature: one empty column per component in it, sized from215// the registry. Every component in the signature must be registered.216kayan_table_init: proc(w: *kayan_world in level, t: *kayan_table in level, signature: u64) -> void = {217 t*.signature = signature;218 for (id: u32 = 0; id < 64; id += 1) {219 if ((signature & kayan_bit(id)) != 0) {220 info: option<*const reflect in permanent> = id < w*.component_count ? w*.infos[id] : null;221 if (!info) {222 printfmt("kayan: signature {} uses component {}, which was never registered\n", signature, id);223 exit(1);224 } else {225 kayan_column_init(t*.columns[id]&, info);226 }227 }228 }229}230 231// Double the room in `entities` and in every column at once.232kayan_table_grow: proc(r: region level, w: *kayan_world in level, t: *kayan_table in level) -> void = {233 capacity: u64 = t*.capacity == 0 ? 4 : t*.capacity * 2;234 entities: []kayan_entity in level = new_slice<kayan_entity>(r, capacity);235 for (row: u64 = 0; row < t*.count; row += 1) {236 entities[row] = t*.entities[row];237 }238 t*.entities = entities;239 for (id: u32 = 0; id < w*.component_count; id += 1) {240 if (kayan_table_has(t, id)) {241 kayan_column_grow(r, t*.columns[id]&, t*.count, capacity);242 }243 }244 t*.capacity = capacity;245}246 247// A new row at the end, owned by `e`, with every component zeroed.248kayan_table_add_row: proc(r: region level, w: *kayan_world in level, t: *kayan_table in level, e: kayan_entity) -> u64 = {249 if (t*.count == t*.capacity) {250 kayan_table_grow(r, w, t);251 }252 row: u64 = t*.count;253 t*.count += 1;254 t*.entities[row] = e;255 for (id: u32 = 0; id < w*.component_count; id += 1) {256 if (kayan_table_has(t, id)) {257 kayan_column_zero_row(t*.columns[id]&, row);258 }259 }260 return row;261}262 263// 1.4's swap-remove, for every column: the last row is copied over `row`, and264// the slot of the entity that owned the last row is told where it went.265kayan_table_remove_row: proc(w: *kayan_world in level, t: *kayan_table in level, row: u64) -> void = {266 last: u64 = t*.count - 1;267 if (row != last) {268 for (id: u32 = 0; id < w*.component_count; id += 1) {269 if (kayan_table_has(t, id)) {270 kayan_column_copy_row(t*.columns[id]&, row, last);271 }272 }273 moved: kayan_entity = t*.entities[last];274 t*.entities[row] = moved;275 vec<kayan_slot>at(w*.slots, cast(moved.index, u64))*.row = cast(row, u32);276 }277 t*.count = last;278}279 280// The typed view of one of the table's columns: T's rows, all of them in use.281// Asking for a component the table does not have is a bug, so it stops.282kayan_slice: proc<T>(t: *kayan_table in level) -> []T in level = {283 id: u32 = kayan_id_of<T>();284 if (!kayan_table_has(t, id)) {285 printfmt("kayan: the table has no column for component {}\n", id);286 exit(1);287 }288 return kayan_column_slice<T>(t*.columns[id]&, t*.count);289}290 291// The table at an index: a pointer into the region, good for as long as the292// level is. The vec of pointers may grow, but what it points at does not move.293kayan_table_at: proc(w: *kayan_world in level, index: u32) -> *kayan_table in level = {294 return vec<*kayan_table in level>at(w*.tables, cast(index, u64))*;295}296 297// The table for a signature: found if it exists, made if it does not. This is298// where archetypes come from.299kayan_table_for: proc(r: region level, w: *kayan_world in level, signature: u64) -> u32 = {300 found: option<*u32> = hashmap<u64, u32>get(w*.by_signature, signature);301 if (found) {302 return found*;303 }304 index: u32 = cast(w*.tables*.length, u32);305 t: *kayan_table in level = new<kayan_table>(r);306 kayan_table_init(w, t, signature);307 _ = vec<*kayan_table in level>push(w*.tables, t);308 _ = hashmap<u64, u32>set(w*.by_signature, signature, index);309 return index;310}311 312// ---------------------------------------------------------------------------313// The world314// ---------------------------------------------------------------------------315 316kayan_world_make: proc(r: region level) -> *kayan_world in level = {317 return new<kayan_world>(r, {318 .tables = vec<*kayan_table in level>make(r, 8),319 .by_signature = hashmap<u64, u32>make(r),320 .slots = vec<kayan_slot>make(r, 8),321 .free = vec<u32>make(r, 8),322 });323}324 325kayan_alive: proc(w: *kayan_world in level, e: kayan_entity) -> b32 = {326 if (cast(e.index, u64) >= w*.slots*.length) {327 return false;328 }329 return vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.generation == e.generation;330}331 332// A new entity with the components in `signature`, all zeroed, in the table333// for that signature.334kayan_spawn: proc(r: region level, w: *kayan_world in level, signature: u64) -> kayan_entity = {335 e: kayan_entity = {};336 if (!vec<u32>is_empty(w*.free)) {337 e.index = vec<u32>pop(w*.free);338 } else {339 e.index = cast(w*.slots*.length, u32);340 _ = vec<kayan_slot>push(w*.slots, { .generation = 1 });341 }342 e.generation = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.generation;343 table_index: u32 = kayan_table_for(r, w, signature);344 t: *kayan_table in level = kayan_table_at(w, table_index);345 row: u64 = kayan_table_add_row(r, w, t, e);346 // Both calls were handed the world, so the pointer into the slots is347 // taken after them.348 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));349 slot*.table = table_index;350 slot*.row = cast(row, u32);351 w*.live += 1;352 return e;353}354 355kayan_despawn: proc(w: *kayan_world in level, e: kayan_entity) -> kayan_status = {356 if (!kayan_alive(w, e)) {357 return kayan_status.dead;358 }359 slot: kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*;360 kayan_table_remove_row(w, kayan_table_at(w, slot.table), cast(slot.row, u64));361 vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.generation += 1;362 _ = vec<u32>push(w*.free, e.index);363 w*.live -= 1;364 return kayan_status.ok;365}366 367// The table and row a live entity is in. Typed get and set arrive in 2.5;368// until then, game code finds the table and row and indexes the columns.369kayan_table_of: proc(w: *kayan_world in level, e: kayan_entity) -> *kayan_table in level = {370 return kayan_table_at(w, vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.table);371}372 373kayan_row: proc(w: *kayan_world in level, e: kayan_entity) -> u64 = {374 return cast(vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.row, u64);375}376 377// The invariant, over every table: each row belongs to a live entity whose378// slot points back at that table and row, and the rows of all tables add up to379// the live count.380kayan_verify: proc(w: *kayan_world in level) -> u64 = {381 problems: u64 = 0;382 rows: u64 = 0;383 for (ti: u64 = 0; ti < w*.tables*.length; ti += 1) {384 t: *kayan_table in level = kayan_table_at(w, cast(ti, u32));385 for (row: u64 = 0; row < t*.count; row += 1) {386 e: kayan_entity = t*.entities[row];387 if (!kayan_alive(w, e)) {388 printfmt("kayan_verify: table {} row {} holds dead entity {}:{}\n", ti, row, e.index, e.generation);389 problems += 1;390 continue;391 }392 slot: kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*;393 if (cast(slot.table, u64) != ti || cast(slot.row, u64) != row) {394 printfmt("kayan_verify: {}:{} is in table {} row {}, its slot says table {} row {}\n",395 e.index, e.generation, ti, row, slot.table, slot.row);396 problems += 1;397 }398 }399 rows += t*.count;400 }401 if (rows != w*.live) {402 printfmt("kayan_verify: {} rows in tables, {} live entities\n", rows, w*.live);403 problems += 1;404 }405 return problems;406}407 408kayan_print_signature: proc(w: *kayan_world in level, signature: u64) -> void = {409 printfmt("{");410 first: b32 = true;411 for (id: u32 = 0; id < w*.component_count; id += 1) {412 if ((signature & kayan_bit(id)) != 0) {413 if (!first) {414 printfmt(" ");415 }416 printfmt("{}", kayan_name(w, id));417 first = false;418 }419 }420 printfmt("}");421}422 423kayan_print_tables: proc(w: *kayan_world in level) -> void = {424 for (ti: u64 = 0; ti < w*.tables*.length; ti += 1) {425 t: *kayan_table in level = kayan_table_at(w, cast(ti, u32));426 printfmt(" table {} ", cast(ti, u32));427 kayan_print_signature(w, t*.signature);428 printfmt(":");429 for (row: u64 = 0; row < t*.count; row += 1) {430 e: kayan_entity = t*.entities[row];431 printfmt(" {}:{}", e.index, e.generation);432 }433 printfmt("\n");434 }435}436 437// ---------------------------------------------------------------------------438// The program439// ---------------------------------------------------------------------------440 441// Until 2.5's typed get, writing a component is: find the entity's table and442// row, take the typed view of the column, write the row.443place: proc(w: *kayan_world in level, e: kayan_entity, x: f32, y: f32) -> void = {444 ps: []position in level = kayan_slice<position>(kayan_table_of(w, e));445 ps[kayan_row(w, e)] = { .at = { .x = x, .y = y } };446}447 448main: proc() -> i32 = {449 level: region = {450 w: *kayan_world in level = kayan_world_make(level);451 kayan_register(w, kayan_id_of<position>(), position<>&);452 kayan_register(w, kayan_id_of<velocity>(), velocity<>&);453 kayan_register(w, kayan_id_of<health>(), health<>&);454 kayan_register(w, kayan_id_of<damage>(), damage<>&);455 kayan_register(w, kayan_id_of<lifetime>(), lifetime<>&);456 kayan_register(w, kayan_id_of<turret>(), turret<>&);457 458 // What each kind of thing is, as a set of components.459 gun: u64 = kayan_mask_of<position>() | kayan_mask_of<turret>();460 drone: u64 = kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<health>();461 bullet: u64 = kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<damage>()462 | kayan_mask_of<lifetime>();463 464 t0: kayan_entity = kayan_spawn(level, w, gun);465 place(w, t0, 0.0, 0.0);466 printfmt("spawned the turret: the world made table {} for it\n", w*.tables*.length - 1);467 468 drones: [3]kayan_entity = {};469 for (i: u32 = 0; i < 3; i += 1) {470 drones[i] = kayan_spawn(level, w, drone);471 place(w, drones[i], 10.0 * cast(i + 1, f32), 5.0);472 }473 printfmt("spawned 3 drones: the first made a table, the others found it\n");474 475 b0: kayan_entity = kayan_spawn(level, w, bullet);476 b1: kayan_entity = kayan_spawn(level, w, bullet);477 place(w, b0, 0.0, 1.0);478 place(w, b1, 1.0, 0.0);479 printfmt("spawned 2 bullets\n");480 printfmt("{} entities in {} tables:\n", w*.live, w*.tables*.length);481 kayan_print_tables(w);482 483 printfmt("despawn drone {}:{}: a swap-remove inside its own table\n", drones[0].index, drones[0].generation);484 _ = kayan_despawn(w, drones[0]);485 kayan_print_tables(w);486 487 // The positions of everything, table by table: the same column, in each488 // table that has it.489 printfmt("every position, table by table:\n");490 for (ti: u32 = 0; ti < cast(w*.tables*.length, u32); ti += 1) {491 t: *kayan_table in level = kayan_table_at(w, ti);492 ps: []position in level = kayan_slice<position>(t);493 for (row: u64 = 0; row < t*.count; row += 1) {494 e: kayan_entity = t*.entities[row];495 p: position = ps[row];496 printfmt(" table {} row {} {}:{} at ({:5.2}, {:5.2})\n", ti, cast(row, u32),497 e.index, e.generation, cast(p.at.x, f64), cast(p.at.y, f64));498 }499 }500 printfmt("kayan_verify: {} problems\n", kayan_verify(w));501 }502 return 0;503}What archetypes prints. tools/check.py fails the chapter if a single byte differs.
spawned the turret: the world made table 0 for it
spawned 3 drones: the first made a table, the others found it
spawned 2 bullets
6 entities in 3 tables:
table 0 {position turret}: 0:1
table 1 {position velocity health}: 1:1 2:1 3:1
table 2 {position velocity damage lifetime}: 4:1 5:1
despawn drone 1:1: a swap-remove inside its own table
table 0 {position turret}: 0:1
table 1 {position velocity health}: 3:1 2:1
table 2 {position velocity damage lifetime}: 4:1 5:1
every position, table by table:
table 0 row 0 0:1 at ( 0.00, 0.00)
table 1 row 0 3:1 at (30.00, 5.00)
table 1 row 1 2:1 at (20.00, 5.00)
table 2 row 0 4:1 at ( 0.00, 1.00)
table 2 row 1 5:1 at ( 1.00, 0.00)
kayan_verify: 0 problems
Build an ECS · part 2, components · 2.5
Typed access
ecs/src/2.components/2.5.typed_access/typed_access.rin
In the dojo: 4. a type-erased column, 8. index, not pointer, each on its own, to watch and to practise.
The world stores any component in any combination, as bytes. Game code should never see the bytes. This chapter adds the typed layer on top:
kayan_get<position>(w, e) -> option<*position in level>: null if missing
kayan_has<velocity>(w, e) -> b32
kayan_set<health>(w, e, value) -> ok, dead, or missing
Each is a few lines that turn the type into an id, find the entity's table and row through its slot, and hand the row back as a T. The byte-level core underneath doesn't change at all. This is the split: a small unsafe core of byte fiddling below, checked generic procs on top, each written once.
Step 1: get
kayan_get: proc<T>(w: *kayan_world in level, e: kayan_entity) -> option<*T in level> = {
if (!kayan_alive(w, e)) {
return null;
}
slot: kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*;
t: *kayan_table in level = kayan_table_at(w, slot.table);
if (!kayan_table_has(t, kayan_id_of<T>())) {
return null;
}
rows: []T in level = kayan_slice<T>(t);
return rows[cast(slot.row, u64)]&;
}
Handle, slot, table, row, typed view. It's the same path as 1.4's get_position, for any component. Two ways to get null: the handle is dead, or the entity has no T. The return type carries both facts the caller needs: option means test it first, and in level means that once tested, it's memory the program owns. The caller checks once and then has a real pointer.
Step 2: has and set
kayan_has: proc<T>(w: *kayan_world in level, e: kayan_entity) -> b32 = {
return (kayan_signature(w, e) & kayan_mask_of<T>()) != 0;
}
kayan_has doesn't touch the row at all. An entity's components are its table's signature, one AND away.
kayan_set: proc<T>(w: *kayan_world in level, e: kayan_entity, value: T) -> kayan_status = {
if (!kayan_alive(w, e)) {
return kayan_status.dead;
}
p: option<*T in level> = kayan_get<T>(w, e);
if (!p) {
return kayan_status.missing;
}
p* = value;
return kayan_status.ok;
}
kayan_set writes a component the entity already has. Giving it a component it doesn't have changes which table it belongs in, and that's 2.6. So the status gains a third value:
kayan_status: enum[must_use] = {
ok,
dead,
missing,
}
Step 3: building things
With kayan_set, game code builds entities with small procs. Engines call these prefabs or archetype constructors:
spawn_drone: proc(r: region level, w: *kayan_world in level, x: f32, y: f32) -> kayan_entity = {
e: kayan_entity = kayan_spawn(r, w, kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<health>());
_ = kayan_set<position>(w, e, { .at = { .x = x, .y = y } });
_ = kayan_set<velocity>(w, e, { .per_second = { .x = -1.0, .y = 0.0 } });
_ = kayan_set<health>(w, e, { .points = 100.0, .max = 100.0 });
return e;
}
The _ = says "I know this returns a status and I'm ignoring it on purpose". Here that's safe: the entity was just spawned with exactly these components.
Reading and changing is kayan_get, and a test:
h: option<*health in level> = kayan_get<health>(w, a);
if (h) {
h*.points -= 25.0;
}
Step 4: keep handles, borrow pointers
The pointer kayan_get returns points into a column. 2.2 showed what columns do when their table fills up: they move. So a pointer is good until the next structural change: a spawn, a despawn, or (from 2.6) an add or remove. The program proves it:
kept: option<*health in level> = kayan_get<health>(w, a);
for (i: u32 = 0; i < 8; i += 1) {
_ = spawn_drone(level, w, 40.0 + cast(i, f32), 0.0);
}
if (kept) {
kept*.points = 1.0;
}
kept a pointer to 0:1's health, spawned 8 drones (the table grew), wrote 1 through it:
0:1 at (20.00, 5.00), health 75
the write went to the column's old block; the handle still finds the real one
The health is still 75. The write landed in the column's old block, which the region still owns, so nothing crashes and no sanitizer complains. It's just lost. Hence the rule every ECS has in some form:
Keep handles. Borrow pointers: get, use, forget.
It's worth being exact about what the checker did here, because this is the line between what regions give you and what they don't. It lets the program through, because in level asks for one thing: this pointer is memory in level, for as long as level is open. That's true. The old block is still there. What the pointer no longer is, is the component, and no amount of checking level can tell you that. That's what a handle is for: the handle goes through the slot, so it finds the row wherever it is now.
You can also ask for more. Write the pointer with no region, option<*health>, and it's no longer a claim about memory. It's a borrow from w, and the checker ends the borrow at the first call that may change w:
type error: 'kept' may be stale: 'w', which it was taken from, has been changed since; take it again
So a pointer you mean to use right away is best left without a region. Part 3's systems write their column slices exactly that way.
Run it
two drones and a bullet, built with kayan_set:
0:1 at (20.00, 5.00), health 100
1:1 at (30.00, -5.00), health 100
drone 0:1 is hit through kayan_get<health>:
0:1 at (20.00, 5.00), health 75
the bullet has a velocity: yes; a health: no
kayan_get<health> on the bullet is null: yes
kayan_set<health> on the bullet: missing
despawned 1:1; kayan_set<health> on it now: dead
1:1 is not a live drone
...
kayan_verify: 0 problems
Try
- Write
spawn_turretandspawn_bullet-style procs for your own kinds of thing. - Move the
keptpointer'skayan_getto after the spawning loop. Now the write sticks. Why? - Make
kayan_setadd the component when it's missing, instead of returningmissing. What would it need from 2.6? - Change
keptto anoption<*health>and compile. Why is the program refused now, when the pointer goes to exactly the same place?
1// 2.5 Typed access2//3// The world now stores any component in any combination, as bytes. Game code4// should never see the bytes. This chapter adds the typed layer on top:5//6// kayan_get<position>(w, e) -> option<*position in level>: null if missing7// kayan_has<velocity>(w, e) -> b328// kayan_set<health>(w, e, value) -> ok, dead, or missing9//10// Each is a few lines that turn the type into an id (kayan_id_of<T>), find the11// entity's table and row through its slot, and hand back the row as a T. The12// byte-level core underneath does not change at all.13//14// One rule comes with it, and the program shows why: keep handles, borrow15// pointers. A pointer from kayan_get points into a column, and the column16// moves when its table grows. A handle never goes stale silently.17//18// The pointer is `option<*T in level>`: it may be null, so the19// checker makes every caller test it, and it points into the level, so it is20// always memory the program owns. What it is not, after the table grows, is21// the component: the old block stays in the region, readable, and no longer22// the entity's. That is the bug the last part of the program shows, and the23// handle is the answer to it. The two trusted procs are the column's.24 25import "std/print.rin"26import "std/memops.rin"27import "std/vec.rin"28import "std/hashmap.rin"29import "std/slice.rin"30 31level: region;32 33vec2: struct = {34 x: f32;35 y: f32;36}37 38position: struct = { at: vec2; }39velocity: struct = { per_second: vec2; }40health: struct = { points: f32; max: f32; }41damage: struct = { amount: f32; }42lifetime: struct = { seconds: f32; }43turret: struct = { cooldown: f32; timer: f32; }44 45component: enum = {46 position,47 velocity,48 health,49 damage,50 lifetime,51 turret,52}53 54component_of: proc<position>() -> component = { return component.position; }55component_of: proc<velocity>() -> component = { return component.velocity; }56component_of: proc<health>() -> component = { return component.health; }57component_of: proc<damage>() -> component = { return component.damage; }58component_of: proc<lifetime>() -> component = { return component.lifetime; }59component_of: proc<turret>() -> component = { return component.turret; }60 61// ---------------------------------------------------------------------------62// Entities (Part 1)63// ---------------------------------------------------------------------------64 65kayan_entity: struct = {66 index: u32;67 generation: u32;68}69 70// `missing`: the entity is alive but has no such component.71kayan_status: enum[must_use] = {72 ok,73 dead,74 missing,75}76 77// Where an entity lives: which table, and which row of it.78kayan_slot: struct = {79 generation: u32;80 table: u32;81 row: u32;82}83 84// ---------------------------------------------------------------------------85// Columns (2.2)86// ---------------------------------------------------------------------------87 88kayan_column: struct = {89 data: []u8 in level;90 size: u64;91 align: u64;92}93 94kayan_column_init: proc(c: *kayan_column, info: *const reflect) -> void = {95 c*.size = info*.size;96 c*.align = info*.align;97}98 99kayan_column_grow: proc(r: region level, c: *kayan_column, count: u64, capacity: u64) -> void = {100 unsafe {101 raw: ^u8 = cast(memops_arena_push_zero(r, c*.size * capacity, c*.align), ^u8);102 bytes: []u8 in level = cast(raw[0..c*.size * capacity], []u8 in level);103 // Slicing checks both ends: a count past either block stops the program.104 kept: u64 = c*.size * count;105 slice<u8>copy(bytes[0..kept], c*.data[0..kept]);106 c*.data = bytes;107 }108}109 110kayan_column_copy_row: proc(c: *kayan_column, to: u64, from: u64) -> void = {111 for (i: u64 = 0; i < c*.size; i += 1) {112 c*.data[to * c*.size + i] = c*.data[from * c*.size + i];113 }114}115 116kayan_column_zero_row: proc(c: *kayan_column, row: u64) -> void = {117 for (i: u64 = 0; i < c*.size; i += 1) {118 c*.data[row * c*.size + i] = 0;119 }120}121 122// The typed view: the column's first `count` rows as `[]T`. This is the one123// place bytes become a type, so it takes nothing on trust. T must be the124// column's size, and plain data: any bytes are then a valid T, so reading the125// wrong type gives wrong numbers, never a bad pointer. The rows must fit, which126// the slice knows and no field can lie about, and the bytes must be aligned for127// T. The view is `[]T in level`: the bytes are in the region, so the view is128// good memory for as long as the region is. After the column grows it is the129// old block, still readable, no longer the column: take the view again.130kayan_column_slice: proc<T>(c: *const kayan_column, count: u64) -> []T in level = {131 if (c*.size != sizeof(T)) {132 printfmt("kayan: a column of {}-byte rows read as a {}-byte type\n", c*.size, sizeof(T));133 exit(1);134 }135 if (!reflect_is_plain(T<>&)) {136 printfmt("kayan: a column holds plain data, and this {}-byte type is not\n", sizeof(T));137 exit(1);138 }139 if (count > c*.data.length / sizeof(T)) {140 printfmt("kayan: a view of {} rows on a column with room for {}\n", count, c*.data.length / sizeof(T));141 exit(1);142 }143 unsafe {144 if (cast(c*.data.data, usize) % alignof(T) != 0) {145 printfmt("kayan: a column's bytes are not aligned to {}\n", alignof(T));146 exit(1);147 }148 raw: ^T = cast(c*.data.data, ^T);149 return cast(raw[0..count], []T in level);150 }151}152 153// ---------------------------------------------------------------------------154// The table155// ---------------------------------------------------------------------------156 157// `columns[id]` is in use when the signature has bit `id`; the others stay158// zeroed. Count and capacity are the table's, shared by its columns and by159// `entities`, so they can never disagree.160kayan_table: struct = {161 signature: u64;162 count: u64;163 capacity: u64;164 entities: []kayan_entity in level;165 columns: [64]kayan_column;166}167 168// The world: the registry, the slots and free list, and every table, with a169// map from signature to table index so finding one is a lookup, not a search.170// Tables are referred to by index in the slots, and by pointer everywhere171// else: a table is made once in the region and never moves.172kayan_world: struct = {173 infos: [64]option<*const reflect in permanent>;174 component_count: u32;175 tables: *vec<*kayan_table in level> in level;176 by_signature: *hashmap<u64, u32> in level;177 slots: *vec<kayan_slot> in level;178 free: *vec<u32> in level;179 live: u64;180}181 182kayan_bit: proc(id: u32) -> u64 = {183 return 1'u64 shl cast(id, u64);184}185 186kayan_id_of: proc<T>() -> u32 = {187 return cast(component_of<T>(), u32);188}189 190kayan_mask_of: proc<T>() -> u64 = {191 return kayan_bit(kayan_id_of<T>());192}193 194kayan_register: proc(w: *kayan_world in level, id: u32, info: *const reflect in permanent) -> void = {195 if (id >= 64) {196 printfmt("kayan: component id {} is past the 64 a signature can hold\n", id);197 exit(1);198 }199 w*.infos[id] = info;200 if (id + 1 > w*.component_count) {201 w*.component_count = id + 1;202 }203}204 205// A component's name, for printing: a registered id's, or "?" for a slot206// nobody filled.207kayan_name: proc(w: *kayan_world in level, id: u32) -> ^const char = {208 info: option<*const reflect in permanent> = w*.infos[id];209 return info ? info*.name : "?";210}211 212kayan_table_has: proc(t: *kayan_table in level, id: u32) -> b32 = {213 return (t*.signature & kayan_bit(id)) != 0;214}215 216// A table for a signature: one empty column per component in it, sized from217// the registry. Every component in the signature must be registered.218kayan_table_init: proc(w: *kayan_world in level, t: *kayan_table in level, signature: u64) -> void = {219 t*.signature = signature;220 for (id: u32 = 0; id < 64; id += 1) {221 if ((signature & kayan_bit(id)) != 0) {222 info: option<*const reflect in permanent> = id < w*.component_count ? w*.infos[id] : null;223 if (!info) {224 printfmt("kayan: signature {} uses component {}, which was never registered\n", signature, id);225 exit(1);226 } else {227 kayan_column_init(t*.columns[id]&, info);228 }229 }230 }231}232 233// Double the room in `entities` and in every column at once.234kayan_table_grow: proc(r: region level, w: *kayan_world in level, t: *kayan_table in level) -> void = {235 capacity: u64 = t*.capacity == 0 ? 4 : t*.capacity * 2;236 entities: []kayan_entity in level = new_slice<kayan_entity>(r, capacity);237 for (row: u64 = 0; row < t*.count; row += 1) {238 entities[row] = t*.entities[row];239 }240 t*.entities = entities;241 for (id: u32 = 0; id < w*.component_count; id += 1) {242 if (kayan_table_has(t, id)) {243 kayan_column_grow(r, t*.columns[id]&, t*.count, capacity);244 }245 }246 t*.capacity = capacity;247}248 249// A new row at the end, owned by `e`, with every component zeroed.250kayan_table_add_row: proc(r: region level, w: *kayan_world in level, t: *kayan_table in level, e: kayan_entity) -> u64 = {251 if (t*.count == t*.capacity) {252 kayan_table_grow(r, w, t);253 }254 row: u64 = t*.count;255 t*.count += 1;256 t*.entities[row] = e;257 for (id: u32 = 0; id < w*.component_count; id += 1) {258 if (kayan_table_has(t, id)) {259 kayan_column_zero_row(t*.columns[id]&, row);260 }261 }262 return row;263}264 265// 1.4's swap-remove, for every column: the last row is copied over `row`, and266// the slot of the entity that owned the last row is told where it went.267kayan_table_remove_row: proc(w: *kayan_world in level, t: *kayan_table in level, row: u64) -> void = {268 last: u64 = t*.count - 1;269 if (row != last) {270 for (id: u32 = 0; id < w*.component_count; id += 1) {271 if (kayan_table_has(t, id)) {272 kayan_column_copy_row(t*.columns[id]&, row, last);273 }274 }275 moved: kayan_entity = t*.entities[last];276 t*.entities[row] = moved;277 vec<kayan_slot>at(w*.slots, cast(moved.index, u64))*.row = cast(row, u32);278 }279 t*.count = last;280}281 282// The typed view of one of the table's columns: T's rows, all of them in use.283// Asking for a component the table does not have is a bug, so it stops.284kayan_slice: proc<T>(t: *kayan_table in level) -> []T in level = {285 id: u32 = kayan_id_of<T>();286 if (!kayan_table_has(t, id)) {287 printfmt("kayan: the table has no column for component {}\n", id);288 exit(1);289 }290 return kayan_column_slice<T>(t*.columns[id]&, t*.count);291}292 293// The table at an index: a pointer into the region, good for as long as the294// level is. The vec of pointers may grow, but what it points at does not move.295kayan_table_at: proc(w: *kayan_world in level, index: u32) -> *kayan_table in level = {296 return vec<*kayan_table in level>at(w*.tables, cast(index, u64))*;297}298 299// The table for a signature: found if it exists, made if it does not. This is300// where archetypes come from.301kayan_table_for: proc(r: region level, w: *kayan_world in level, signature: u64) -> u32 = {302 found: option<*u32> = hashmap<u64, u32>get(w*.by_signature, signature);303 if (found) {304 return found*;305 }306 index: u32 = cast(w*.tables*.length, u32);307 t: *kayan_table in level = new<kayan_table>(r);308 kayan_table_init(w, t, signature);309 _ = vec<*kayan_table in level>push(w*.tables, t);310 _ = hashmap<u64, u32>set(w*.by_signature, signature, index);311 return index;312}313 314// ---------------------------------------------------------------------------315// The world316// ---------------------------------------------------------------------------317 318kayan_world_make: proc(r: region level) -> *kayan_world in level = {319 return new<kayan_world>(r, {320 .tables = vec<*kayan_table in level>make(r, 8),321 .by_signature = hashmap<u64, u32>make(r),322 .slots = vec<kayan_slot>make(r, 8),323 .free = vec<u32>make(r, 8),324 });325}326 327kayan_alive: proc(w: *kayan_world in level, e: kayan_entity) -> b32 = {328 if (cast(e.index, u64) >= w*.slots*.length) {329 return false;330 }331 return vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.generation == e.generation;332}333 334// A new entity with the components in `signature`, all zeroed, in the table335// for that signature.336kayan_spawn: proc(r: region level, w: *kayan_world in level, signature: u64) -> kayan_entity = {337 e: kayan_entity = {};338 if (!vec<u32>is_empty(w*.free)) {339 e.index = vec<u32>pop(w*.free);340 } else {341 e.index = cast(w*.slots*.length, u32);342 _ = vec<kayan_slot>push(w*.slots, { .generation = 1 });343 }344 e.generation = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.generation;345 table_index: u32 = kayan_table_for(r, w, signature);346 t: *kayan_table in level = kayan_table_at(w, table_index);347 row: u64 = kayan_table_add_row(r, w, t, e);348 // Both calls were handed the world, so the pointer into the slots is349 // taken after them.350 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));351 slot*.table = table_index;352 slot*.row = cast(row, u32);353 w*.live += 1;354 return e;355}356 357kayan_despawn: proc(w: *kayan_world in level, e: kayan_entity) -> kayan_status = {358 if (!kayan_alive(w, e)) {359 return kayan_status.dead;360 }361 slot: kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*;362 kayan_table_remove_row(w, kayan_table_at(w, slot.table), cast(slot.row, u64));363 vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.generation += 1;364 _ = vec<u32>push(w*.free, e.index);365 w*.live -= 1;366 return kayan_status.ok;367}368 369// The components a live entity has: its table's signature. 0 for a dead one.370kayan_signature: proc(w: *kayan_world in level, e: kayan_entity) -> u64 = {371 if (!kayan_alive(w, e)) {372 return 0;373 }374 return kayan_table_at(w, vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.table)*.signature;375}376 377// ---------------------------------------------------------------------------378// The typed layer379// ---------------------------------------------------------------------------380 381// e's T, or null if e is dead or has no T. Handle, slot, table, row, typed382// view: the same path as 1.4's get_position, for any component. The pointer is383// good until the next structural change, so use it now and keep the handle.384kayan_get: proc<T>(w: *kayan_world in level, e: kayan_entity) -> option<*T in level> = {385 if (!kayan_alive(w, e)) {386 return null;387 }388 slot: kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*;389 t: *kayan_table in level = kayan_table_at(w, slot.table);390 if (!kayan_table_has(t, kayan_id_of<T>())) {391 return null;392 }393 rows: []T in level = kayan_slice<T>(t);394 return rows[cast(slot.row, u64)]&;395}396 397kayan_has: proc<T>(w: *kayan_world in level, e: kayan_entity) -> b32 = {398 return (kayan_signature(w, e) & kayan_mask_of<T>()) != 0;399}400 401// Write e's T. It must already have one: giving an entity a component it does402// not have changes its table, which is 2.6.403kayan_set: proc<T>(w: *kayan_world in level, e: kayan_entity, value: T) -> kayan_status = {404 if (!kayan_alive(w, e)) {405 return kayan_status.dead;406 }407 p: option<*T in level> = kayan_get<T>(w, e);408 if (!p) {409 return kayan_status.missing;410 }411 p* = value;412 return kayan_status.ok;413}414 415// The invariant, over every table: each row belongs to a live entity whose416// slot points back at that table and row, and the rows of all tables add up to417// the live count.418kayan_verify: proc(w: *kayan_world in level) -> u64 = {419 problems: u64 = 0;420 rows: u64 = 0;421 for (ti: u64 = 0; ti < w*.tables*.length; ti += 1) {422 t: *kayan_table in level = kayan_table_at(w, cast(ti, u32));423 for (row: u64 = 0; row < t*.count; row += 1) {424 e: kayan_entity = t*.entities[row];425 if (!kayan_alive(w, e)) {426 printfmt("kayan_verify: table {} row {} holds dead entity {}:{}\n", ti, row, e.index, e.generation);427 problems += 1;428 continue;429 }430 slot: kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*;431 if (cast(slot.table, u64) != ti || cast(slot.row, u64) != row) {432 printfmt("kayan_verify: {}:{} is in table {} row {}, its slot says table {} row {}\n",433 e.index, e.generation, ti, row, slot.table, slot.row);434 problems += 1;435 }436 }437 rows += t*.count;438 }439 if (rows != w*.live) {440 printfmt("kayan_verify: {} rows in tables, {} live entities\n", rows, w*.live);441 problems += 1;442 }443 return problems;444}445 446kayan_print_signature: proc(w: *kayan_world in level, signature: u64) -> void = {447 printfmt("{");448 first: b32 = true;449 for (id: u32 = 0; id < w*.component_count; id += 1) {450 if ((signature & kayan_bit(id)) != 0) {451 if (!first) {452 printfmt(" ");453 }454 printfmt("{}", kayan_name(w, id));455 first = false;456 }457 }458 printfmt("}");459}460 461kayan_print_tables: proc(w: *kayan_world in level) -> void = {462 for (ti: u64 = 0; ti < w*.tables*.length; ti += 1) {463 t: *kayan_table in level = kayan_table_at(w, cast(ti, u32));464 printfmt(" table {} ", cast(ti, u32));465 kayan_print_signature(w, t*.signature);466 printfmt(":");467 for (row: u64 = 0; row < t*.count; row += 1) {468 e: kayan_entity = t*.entities[row];469 printfmt(" {}:{}", e.index, e.generation);470 }471 printfmt("\n");472 }473}474 475// ---------------------------------------------------------------------------476// The program477// ---------------------------------------------------------------------------478 479// Spawning something is: spawn with its signature, then kayan_set a value for480// each component. The statuses are ignored on purpose, because the entity was481// just spawned with exactly these components.482spawn_drone: proc(r: region level, w: *kayan_world in level, x: f32, y: f32) -> kayan_entity = {483 e: kayan_entity = kayan_spawn(r, w, kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<health>());484 _ = kayan_set<position>(w, e, { .at = { .x = x, .y = y } });485 _ = kayan_set<velocity>(w, e, { .per_second = { .x = -1.0, .y = 0.0 } });486 _ = kayan_set<health>(w, e, { .points = 100.0, .max = 100.0 });487 return e;488}489 490spawn_bullet: proc(r: region level, w: *kayan_world in level, x: f32, y: f32) -> kayan_entity = {491 e: kayan_entity = kayan_spawn(r, w, kayan_mask_of<position>() | kayan_mask_of<velocity>()492 | kayan_mask_of<damage>() | kayan_mask_of<lifetime>());493 _ = kayan_set<position>(w, e, { .at = { .x = x, .y = y } });494 _ = kayan_set<velocity>(w, e, { .per_second = { .x = 8.0, .y = 0.0 } });495 _ = kayan_set<damage>(w, e, { .amount = 25.0 });496 _ = kayan_set<lifetime>(w, e, { .seconds = 2.0 });497 return e;498}499 500print_drone: proc(w: *kayan_world in level, e: kayan_entity) -> void = {501 p: option<*position in level> = kayan_get<position>(w, e);502 h: option<*health in level> = kayan_get<health>(w, e);503 if (!p || !h) {504 printfmt(" {}:{} is not a live drone\n", e.index, e.generation);505 return;506 }507 printfmt(" {}:{} at ({:5.2}, {:5.2}), health {:.0}\n", e.index, e.generation,508 cast(p*.at.x, f64), cast(p*.at.y, f64), cast(h*.points, f64));509}510 511main: proc() -> i32 = {512 level: region = {513 w: *kayan_world in level = kayan_world_make(level);514 kayan_register(w, kayan_id_of<position>(), position<>&);515 kayan_register(w, kayan_id_of<velocity>(), velocity<>&);516 kayan_register(w, kayan_id_of<health>(), health<>&);517 kayan_register(w, kayan_id_of<damage>(), damage<>&);518 kayan_register(w, kayan_id_of<lifetime>(), lifetime<>&);519 kayan_register(w, kayan_id_of<turret>(), turret<>&);520 521 a: kayan_entity = spawn_drone(level, w, 20.0, 5.0);522 b: kayan_entity = spawn_drone(level, w, 30.0, -5.0);523 shot: kayan_entity = spawn_bullet(level, w, 0.0, 0.0);524 printfmt("two drones and a bullet, built with kayan_set:\n");525 print_drone(w, a);526 print_drone(w, b);527 528 // Read, change, done: the pointer is used right away.529 h: option<*health in level> = kayan_get<health>(w, a);530 if (h) {531 h*.points -= 25.0;532 }533 printfmt("drone {}:{} is hit through kayan_get<health>:\n", a.index, a.generation);534 print_drone(w, a);535 536 printfmt("the bullet has a velocity: {}; a health: {}\n",537 kayan_has<velocity>(w, shot) ? "yes" : "no", kayan_has<health>(w, shot) ? "yes" : "no");538 printfmt("kayan_get<health> on the bullet is null: {}\n", kayan_get<health>(w, shot) == null ? "yes" : "no");539 printfmt("kayan_set<health> on the bullet: {}\n", kayan_set<health>(w, shot, { .points = 1.0, .max = 1.0 }));540 541 _ = kayan_despawn(w, b);542 printfmt("despawned {}:{}; kayan_set<health> on it now: {}\n", b.index, b.generation,543 kayan_set<health>(w, b, { .points = 1.0, .max = 1.0 }));544 print_drone(w, b);545 546 // Keep handles, borrow pointers. A pointer kept across spawns points into547 // a column that moves when the table grows. The checker lets this through:548 // the old block is still the region's memory, so nothing is unsafe. It is549 // only wrong, which no checker catches and a handle never is.550 kept: option<*health in level> = kayan_get<health>(w, a);551 for (i: u32 = 0; i < 8; i += 1) {552 _ = spawn_drone(level, w, 40.0 + cast(i, f32), 0.0);553 }554 if (kept) {555 kept*.points = 1.0;556 }557 printfmt("kept a pointer to {}:{}'s health, spawned 8 drones (the table grew), wrote 1 through it:\n",558 a.index, a.generation);559 print_drone(w, a);560 printfmt(" the write went to the column's old block; the handle still finds the real one\n");561 printfmt("kayan_verify: {} problems\n", kayan_verify(w));562 }563 return 0;564}What typed_access prints. tools/check.py fails the chapter if a single byte differs.
two drones and a bullet, built with kayan_set:
0:1 at (20.00, 5.00), health 100
1:1 at (30.00, -5.00), health 100
drone 0:1 is hit through kayan_get<health>:
0:1 at (20.00, 5.00), health 75
the bullet has a velocity: yes; a health: no
kayan_get<health> on the bullet is null: yes
kayan_set<health> on the bullet: missing
despawned 1:1; kayan_set<health> on it now: dead
1:1 is not a live drone
kept a pointer to 0:1's health, spawned 8 drones (the table grew), wrote 1 through it:
0:1 at (20.00, 5.00), health 75
the write went to the column's old block; the handle still finds the real one
kayan_verify: 0 problems
Build an ECS · part 2, components · 2.6
Adding and removing components
ecs/src/2.components/2.6.add_remove/add_remove.rin
In the dojo: 2. swap-remove and the back-reference, 10. memoized edges, each on its own, to watch and to practise.
A drone catches fire: it gains a burning component. The fire goes out: it loses it. Changing what an entity is, while the game runs, is the thing the C kayan couldn't do, because its archetypes were structs fixed at compile time. In an archetype ECS a table holds exactly one combination, so a drone that gains a component no longer belongs in its table. It moves:
- find the table for its signature plus (or minus) the component;
- add a row there, and copy every component the two tables share;
- swap-remove its old row, fixing whoever moves into the hole;
- point its slot at the new table and row.
Its handle doesn't change. Everything else about it is where it was, one table over.
Step 1: the move
kayan_move: proc(r: region level, w: *kayan_world in level, e: kayan_entity, to_index: u32) -> void = {
slot: kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*;
from_index: u32 = slot.table;
if (from_index == to_index) {
return;
}
from: *kayan_table in level = kayan_table_at(w, from_index);
to: *kayan_table in level = kayan_table_at(w, to_index);
from_row: u64 = cast(slot.row, u64);
to_row: u64 = kayan_table_add_row(r, w, to, e);
for (id: u32 = 0; id < w*.component_count; id += 1) {
if (kayan_table_has(from, id) && kayan_table_has(to, id)) {
kayan_column_copy_row_from(to*.columns[id]&, to_row, from*.columns[id]&, from_row);
}
}
kayan_table_remove_row(w, from, from_row);
// The slot is written after the calls that were handed the world.
moved: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));
moved*.table = to_index;
moved*.row = cast(to_row, u32);
}
Every piece is something you've already built: kayan_table_add_row (2.3), a byte copy per shared column (2.2), kayan_table_remove_row with its slot fix (1.4). Components only the new table has start zeroed (add_row zeroes them). Components only the old table has disappear with the old row.
The pointers follow 2.4's rules, and they make this proc easy to read. from and to are held across everything, because tables don't move. The slot is copied out at the top, and the one pointer into it that writes is taken at the bottom, after the calls.
The new copy goes between two columns, so it's a new small proc, the same bounds-checked byte loop as kayan_column_copy_row:
kayan_column_copy_row_from: proc(dst: *kayan_column, to: u64, src: *const kayan_column, from: u64) -> void = {
for (i: u64 = 0; i < dst*.size; i += 1) {
dst*.data[to * dst*.size + i] = src*.data[from * src*.size + i];
}
}
A move changes where an entity's data lives, and that's the thing type erasure makes risky, yet nothing here is unsafe. The two trusted procs are still the column's own, from 2.2.
A move costs one row copy in and one row copy out. That's cheap, but not free, and it's why components that flip on and off every frame are sometimes better as a field (a bool on_fire inside health) than as a component.
Step 2: add and remove, typed
kayan_add: proc<T>(r: region level, w: *kayan_world in level, e: kayan_entity, value: T) -> kayan_status = {
if (!kayan_alive(w, e)) {
return kayan_status.dead;
}
if (!kayan_has<T>(w, e)) {
from: u32 = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.table;
kayan_move(r, w, e, kayan_table_with(r, w, from, kayan_id_of<T>()));
}
return kayan_set<T>(w, e, value);
}
If the entity already has a T, adding is just setting. Otherwise it moves to the table with T, then sets it. kayan_remove<T> is the mirror image: missing if it has no T, otherwise a move to the table without T.
Step 3: the archetype graph
Which table is "the table with T"? Its signature is the old one plus T's bit, so it could be looked up in by_signature every time. But a game toggling components does that constantly, always between the same few tables. So each table remembers its neighbours:
kayan_table: struct = {
signature: u64;
count: u64;
capacity: u64;
entities: []kayan_entity in level;
columns: [64]kayan_column;
add_edges: [64]u32;
remove_edges: [64]u32;
}
add_edges[id] is where this table's entities go when they gain component id. remove_edges[id] is where they go when they lose it. Each is stored as index + 1, so the zero a new table starts with means "not known yet", with no initialization pass.
kayan_table_with: proc(r: region level, w: *kayan_world in level, from: u32, id: u32) -> u32 = {
edge: u32 = kayan_table_at(w, from)*.add_edges[id];
if (edge != 0) {
w*.edge_hits += 1;
return edge - 1;
}
w*.lookups += 1;
to: u32 = kayan_table_for(r, w, kayan_table_at(w, from)*.signature | kayan_bit(id));
kayan_table_at(w, from)*.add_edges[id] = to + 1;
kayan_table_at(w, to)*.remove_edges[id] = from + 1;
return to;
}
The first move from {position velocity health} with burning looks the table up (and makes it). It then writes the answer on both tables: an add edge one way and a remove edge back. After that, catching fire and putting it out both follow edges. The tables and their edges form a graph, the archetype graph, and flecs is built around it.
Run it
4 drones:
table 0 {position velocity health}: 0:1 1:1 2:1 3:1
frame 1
0:1 catches fire: ok
2:1 catches fire: ok
table 0 {position velocity health}: 3:1 1:1
table 1 {position velocity health burning}: 0:1 2:1
frame 2
0:1 is put out: ok
1:1 catches fire: ok
2:1 catches fire: ok
...
frame 3
1:1 is put out: ok
2:1 is put out: ok
3:1 is put out: missing
3:1 catches fire: ok
...
2:1 moved tables twice and kept its data: at (30.00, 0.00), health 100
moves that looked up a table by signature: 1; that followed an edge: 6
kayan_verify: 0 problems
Watch row order shuffle as entities move: 3:1 jumps to row 0 of table 0 when 0:1 leaves. Order is never promised, and swap-remove is why. And one lookup for seven moves: that's the graph working.
Try
- Add a
stunnedcomponent and a second edge: stun a burning drone. How many tables exist now, and which edges were written? - Print every table's known edges after frame 3.
- Make
kayan_add<burning>on a drone that's already burning add to the rate instead of replacing it. Where does that logic belong: in kayan or in the game?
1// 2.6 Adding and removing components2//3// A drone catches fire: it gains a `burning` component. The fire goes out: it4// loses it. In an archetype ECS a table holds exactly one combination, so a5// drone that gains a component no longer belongs in its table. It moves:6//7// 1. find the table for its signature plus (or minus) the component;8// 2. add a row there, and copy every component the two tables share;9// 3. swap-remove its old row, fixing whoever moved into the hole;10// 4. point its slot at the new table and row.11//12// Its handle does not change. Everything else about it is where it was, one13// table over.14//15// Step 1 is a hashmap lookup by signature, and a game toggling components16// does it constantly, always between the same few tables. So each table17// remembers its neighbours: "add burning: table 2", "remove burning: table 1".18// These edges make the tables a graph, the archetype graph, and after the19// first move between two tables every later one follows an edge instead of20// looking anything up.21//22// As in 2.4 and 2.5: the world and its tables live in the23// `level` region, the two trusted procs are the column's, and the row copy24// between tables is a bounds-checked byte loop.25 26import "std/print.rin"27import "std/memops.rin"28import "std/vec.rin"29import "std/hashmap.rin"30import "std/slice.rin"31 32level: region;33 34vec2: struct = {35 x: f32;36 y: f32;37}38 39position: struct = { at: vec2; }40velocity: struct = { per_second: vec2; }41health: struct = { points: f32; max: f32; }42damage: struct = { amount: f32; }43lifetime: struct = { seconds: f32; }44turret: struct = { cooldown: f32; timer: f32; }45burning: struct = { per_second: f32; }46 47component: enum = {48 position,49 velocity,50 health,51 damage,52 lifetime,53 turret,54 burning,55}56 57component_of: proc<position>() -> component = { return component.position; }58component_of: proc<velocity>() -> component = { return component.velocity; }59component_of: proc<health>() -> component = { return component.health; }60component_of: proc<damage>() -> component = { return component.damage; }61component_of: proc<lifetime>() -> component = { return component.lifetime; }62component_of: proc<turret>() -> component = { return component.turret; }63component_of: proc<burning>() -> component = { return component.burning; }64 65// ---------------------------------------------------------------------------66// Entities (Part 1)67// ---------------------------------------------------------------------------68 69kayan_entity: struct = {70 index: u32;71 generation: u32;72}73 74// `missing`: the entity is alive but has no such component.75kayan_status: enum[must_use] = {76 ok,77 dead,78 missing,79}80 81// Where an entity lives: which table, and which row of it.82kayan_slot: struct = {83 generation: u32;84 table: u32;85 row: u32;86}87 88// ---------------------------------------------------------------------------89// Columns (2.2)90// ---------------------------------------------------------------------------91 92kayan_column: struct = {93 data: []u8 in level;94 size: u64;95 align: u64;96}97 98kayan_column_init: proc(c: *kayan_column, info: *const reflect) -> void = {99 c*.size = info*.size;100 c*.align = info*.align;101}102 103kayan_column_grow: proc(r: region level, c: *kayan_column, count: u64, capacity: u64) -> void = {104 unsafe {105 raw: ^u8 = cast(memops_arena_push_zero(r, c*.size * capacity, c*.align), ^u8);106 bytes: []u8 in level = cast(raw[0..c*.size * capacity], []u8 in level);107 // Slicing checks both ends: a count past either block stops the program.108 kept: u64 = c*.size * count;109 slice<u8>copy(bytes[0..kept], c*.data[0..kept]);110 c*.data = bytes;111 }112}113 114kayan_column_copy_row: proc(c: *kayan_column, to: u64, from: u64) -> void = {115 for (i: u64 = 0; i < c*.size; i += 1) {116 c*.data[to * c*.size + i] = c*.data[from * c*.size + i];117 }118}119 120// Row `from` of one column over row `to` of another: the same component in121// two tables, as a move between tables copies it.122kayan_column_copy_row_from: proc(dst: *kayan_column, to: u64, src: *const kayan_column, from: u64) -> void = {123 for (i: u64 = 0; i < dst*.size; i += 1) {124 dst*.data[to * dst*.size + i] = src*.data[from * src*.size + i];125 }126}127 128kayan_column_zero_row: proc(c: *kayan_column, row: u64) -> void = {129 for (i: u64 = 0; i < c*.size; i += 1) {130 c*.data[row * c*.size + i] = 0;131 }132}133 134// The typed view: the column's first `count` rows as `[]T`. This is the one135// place bytes become a type, so it takes nothing on trust. T must be the136// column's size, and plain data: any bytes are then a valid T, so reading the137// wrong type gives wrong numbers, never a bad pointer. The rows must fit, which138// the slice knows and no field can lie about, and the bytes must be aligned for139// T. The view is `[]T in level`: the bytes are in the region, so the view is140// good memory for as long as the region is. After the column grows it is the141// old block, still readable, no longer the column: take the view again.142kayan_column_slice: proc<T>(c: *const kayan_column, count: u64) -> []T in level = {143 if (c*.size != sizeof(T)) {144 printfmt("kayan: a column of {}-byte rows read as a {}-byte type\n", c*.size, sizeof(T));145 exit(1);146 }147 if (!reflect_is_plain(T<>&)) {148 printfmt("kayan: a column holds plain data, and this {}-byte type is not\n", sizeof(T));149 exit(1);150 }151 if (count > c*.data.length / sizeof(T)) {152 printfmt("kayan: a view of {} rows on a column with room for {}\n", count, c*.data.length / sizeof(T));153 exit(1);154 }155 unsafe {156 if (cast(c*.data.data, usize) % alignof(T) != 0) {157 printfmt("kayan: a column's bytes are not aligned to {}\n", alignof(T));158 exit(1);159 }160 raw: ^T = cast(c*.data.data, ^T);161 return cast(raw[0..count], []T in level);162 }163}164 165// ---------------------------------------------------------------------------166// The table167// ---------------------------------------------------------------------------168 169// `columns[id]` is in use when the signature has bit `id`; the others stay170// zeroed. Count and capacity are the table's, shared by its columns and by171// `entities`, so they can never disagree.172//173// The edges: `add_edges[id]` is the table this one's entities move to when174// they gain component `id`, and `remove_edges[id]` when they lose it. Each is175// the table's index plus one, so 0, which a new table starts with, means "not176// known yet".177kayan_table: struct = {178 signature: u64;179 count: u64;180 capacity: u64;181 entities: []kayan_entity in level;182 columns: [64]kayan_column;183 add_edges: [64]u32;184 remove_edges: [64]u32;185}186 187// The world: the registry, the slots and free list, and every table, with a188// map from signature to table index so finding one is a lookup, not a search.189// Tables are referred to by index in the slots, and by pointer everywhere190// else: a table is made once in the region and never moves.191kayan_world: struct = {192 infos: [64]option<*const reflect in permanent>;193 component_count: u32;194 tables: *vec<*kayan_table in level> in level;195 by_signature: *hashmap<u64, u32> in level;196 slots: *vec<kayan_slot> in level;197 free: *vec<u32> in level;198 live: u64;199 // How table moves found their destination, for this chapter's output.200 lookups: u64;201 edge_hits: u64;202}203 204kayan_bit: proc(id: u32) -> u64 = {205 return 1'u64 shl cast(id, u64);206}207 208kayan_id_of: proc<T>() -> u32 = {209 return cast(component_of<T>(), u32);210}211 212kayan_mask_of: proc<T>() -> u64 = {213 return kayan_bit(kayan_id_of<T>());214}215 216kayan_register: proc(w: *kayan_world in level, id: u32, info: *const reflect in permanent) -> void = {217 if (id >= 64) {218 printfmt("kayan: component id {} is past the 64 a signature can hold\n", id);219 exit(1);220 }221 w*.infos[id] = info;222 if (id + 1 > w*.component_count) {223 w*.component_count = id + 1;224 }225}226 227// A component's name, for printing: a registered id's, or "?" for a slot228// nobody filled.229kayan_name: proc(w: *kayan_world in level, id: u32) -> ^const char = {230 info: option<*const reflect in permanent> = w*.infos[id];231 return info ? info*.name : "?";232}233 234kayan_table_has: proc(t: *kayan_table in level, id: u32) -> b32 = {235 return (t*.signature & kayan_bit(id)) != 0;236}237 238// A table for a signature: one empty column per component in it, sized from239// the registry. Every component in the signature must be registered.240kayan_table_init: proc(w: *kayan_world in level, t: *kayan_table in level, signature: u64) -> void = {241 t*.signature = signature;242 for (id: u32 = 0; id < 64; id += 1) {243 if ((signature & kayan_bit(id)) != 0) {244 info: option<*const reflect in permanent> = id < w*.component_count ? w*.infos[id] : null;245 if (!info) {246 printfmt("kayan: signature {} uses component {}, which was never registered\n", signature, id);247 exit(1);248 } else {249 kayan_column_init(t*.columns[id]&, info);250 }251 }252 }253}254 255// Double the room in `entities` and in every column at once.256kayan_table_grow: proc(r: region level, w: *kayan_world in level, t: *kayan_table in level) -> void = {257 capacity: u64 = t*.capacity == 0 ? 4 : t*.capacity * 2;258 entities: []kayan_entity in level = new_slice<kayan_entity>(r, capacity);259 for (row: u64 = 0; row < t*.count; row += 1) {260 entities[row] = t*.entities[row];261 }262 t*.entities = entities;263 for (id: u32 = 0; id < w*.component_count; id += 1) {264 if (kayan_table_has(t, id)) {265 kayan_column_grow(r, t*.columns[id]&, t*.count, capacity);266 }267 }268 t*.capacity = capacity;269}270 271// A new row at the end, owned by `e`, with every component zeroed.272kayan_table_add_row: proc(r: region level, w: *kayan_world in level, t: *kayan_table in level, e: kayan_entity) -> u64 = {273 if (t*.count == t*.capacity) {274 kayan_table_grow(r, w, t);275 }276 row: u64 = t*.count;277 t*.count += 1;278 t*.entities[row] = e;279 for (id: u32 = 0; id < w*.component_count; id += 1) {280 if (kayan_table_has(t, id)) {281 kayan_column_zero_row(t*.columns[id]&, row);282 }283 }284 return row;285}286 287// 1.4's swap-remove, for every column: the last row is copied over `row`, and288// the slot of the entity that owned the last row is told where it went.289kayan_table_remove_row: proc(w: *kayan_world in level, t: *kayan_table in level, row: u64) -> void = {290 last: u64 = t*.count - 1;291 if (row != last) {292 for (id: u32 = 0; id < w*.component_count; id += 1) {293 if (kayan_table_has(t, id)) {294 kayan_column_copy_row(t*.columns[id]&, row, last);295 }296 }297 moved: kayan_entity = t*.entities[last];298 t*.entities[row] = moved;299 vec<kayan_slot>at(w*.slots, cast(moved.index, u64))*.row = cast(row, u32);300 }301 t*.count = last;302}303 304// The typed view of one of the table's columns: T's rows, all of them in use.305// Asking for a component the table does not have is a bug, so it stops.306kayan_slice: proc<T>(t: *kayan_table in level) -> []T in level = {307 id: u32 = kayan_id_of<T>();308 if (!kayan_table_has(t, id)) {309 printfmt("kayan: the table has no column for component {}\n", id);310 exit(1);311 }312 return kayan_column_slice<T>(t*.columns[id]&, t*.count);313}314 315// The table at an index: a pointer into the region, good for as long as the316// level is. The vec of pointers may grow, but what it points at does not move.317kayan_table_at: proc(w: *kayan_world in level, index: u32) -> *kayan_table in level = {318 return vec<*kayan_table in level>at(w*.tables, cast(index, u64))*;319}320 321// The table for a signature: found if it exists, made if it does not. This is322// where archetypes come from.323kayan_table_for: proc(r: region level, w: *kayan_world in level, signature: u64) -> u32 = {324 found: option<*u32> = hashmap<u64, u32>get(w*.by_signature, signature);325 if (found) {326 return found*;327 }328 index: u32 = cast(w*.tables*.length, u32);329 t: *kayan_table in level = new<kayan_table>(r);330 kayan_table_init(w, t, signature);331 _ = vec<*kayan_table in level>push(w*.tables, t);332 _ = hashmap<u64, u32>set(w*.by_signature, signature, index);333 return index;334}335 336// ---------------------------------------------------------------------------337// The world338// ---------------------------------------------------------------------------339 340kayan_world_make: proc(r: region level) -> *kayan_world in level = {341 return new<kayan_world>(r, {342 .tables = vec<*kayan_table in level>make(r, 8),343 .by_signature = hashmap<u64, u32>make(r),344 .slots = vec<kayan_slot>make(r, 8),345 .free = vec<u32>make(r, 8),346 });347}348 349kayan_alive: proc(w: *kayan_world in level, e: kayan_entity) -> b32 = {350 if (cast(e.index, u64) >= w*.slots*.length) {351 return false;352 }353 return vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.generation == e.generation;354}355 356// A new entity with the components in `signature`, all zeroed, in the table357// for that signature.358kayan_spawn: proc(r: region level, w: *kayan_world in level, signature: u64) -> kayan_entity = {359 e: kayan_entity = {};360 if (!vec<u32>is_empty(w*.free)) {361 e.index = vec<u32>pop(w*.free);362 } else {363 e.index = cast(w*.slots*.length, u32);364 _ = vec<kayan_slot>push(w*.slots, { .generation = 1 });365 }366 e.generation = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.generation;367 table_index: u32 = kayan_table_for(r, w, signature);368 t: *kayan_table in level = kayan_table_at(w, table_index);369 row: u64 = kayan_table_add_row(r, w, t, e);370 // Both calls were handed the world, so the pointer into the slots is371 // taken after them.372 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));373 slot*.table = table_index;374 slot*.row = cast(row, u32);375 w*.live += 1;376 return e;377}378 379kayan_despawn: proc(w: *kayan_world in level, e: kayan_entity) -> kayan_status = {380 if (!kayan_alive(w, e)) {381 return kayan_status.dead;382 }383 slot: kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*;384 kayan_table_remove_row(w, kayan_table_at(w, slot.table), cast(slot.row, u64));385 vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.generation += 1;386 _ = vec<u32>push(w*.free, e.index);387 w*.live -= 1;388 return kayan_status.ok;389}390 391// The components a live entity has: its table's signature. 0 for a dead one.392kayan_signature: proc(w: *kayan_world in level, e: kayan_entity) -> u64 = {393 if (!kayan_alive(w, e)) {394 return 0;395 }396 return kayan_table_at(w, vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.table)*.signature;397}398 399// ---------------------------------------------------------------------------400// Moving between tables401// ---------------------------------------------------------------------------402 403// The table an entity of table `from` moves to when it gains component `id`.404// The first time, it is found by signature, which can make the table; the405// answer is written on both tables, as from's add edge and the destination's406// remove edge, so neither direction is looked up again.407kayan_table_with: proc(r: region level, w: *kayan_world in level, from: u32, id: u32) -> u32 = {408 edge: u32 = kayan_table_at(w, from)*.add_edges[id];409 if (edge != 0) {410 w*.edge_hits += 1;411 return edge - 1;412 }413 w*.lookups += 1;414 to: u32 = kayan_table_for(r, w, kayan_table_at(w, from)*.signature | kayan_bit(id));415 kayan_table_at(w, from)*.add_edges[id] = to + 1;416 kayan_table_at(w, to)*.remove_edges[id] = from + 1;417 return to;418}419 420// The same for losing component `id`.421kayan_table_without: proc(r: region level, w: *kayan_world in level, from: u32, id: u32) -> u32 = {422 edge: u32 = kayan_table_at(w, from)*.remove_edges[id];423 if (edge != 0) {424 w*.edge_hits += 1;425 return edge - 1;426 }427 w*.lookups += 1;428 to: u32 = kayan_table_for(r, w, kayan_table_at(w, from)*.signature & ~kayan_bit(id));429 kayan_table_at(w, from)*.remove_edges[id] = to + 1;430 kayan_table_at(w, to)*.add_edges[id] = from + 1;431 return to;432}433 434// Move a live entity to table `to_index`. Components both tables have are435// copied, byte for byte; components only the new table has start zeroed;436// components only the old table has are dropped with its old row.437kayan_move: proc(r: region level, w: *kayan_world in level, e: kayan_entity, to_index: u32) -> void = {438 slot: kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*;439 from_index: u32 = slot.table;440 if (from_index == to_index) {441 return;442 }443 from: *kayan_table in level = kayan_table_at(w, from_index);444 to: *kayan_table in level = kayan_table_at(w, to_index);445 from_row: u64 = cast(slot.row, u64);446 to_row: u64 = kayan_table_add_row(r, w, to, e);447 for (id: u32 = 0; id < w*.component_count; id += 1) {448 if (kayan_table_has(from, id) && kayan_table_has(to, id)) {449 kayan_column_copy_row_from(to*.columns[id]&, to_row, from*.columns[id]&, from_row);450 }451 }452 kayan_table_remove_row(w, from, from_row);453 // The slot is written after the calls that were handed the world.454 moved: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));455 moved*.table = to_index;456 moved*.row = cast(to_row, u32);457}458 459// ---------------------------------------------------------------------------460// The typed layer461// ---------------------------------------------------------------------------462 463// e's T, or null if e is dead or has no T. Handle, slot, table, row, typed464// view: the same path as 1.4's get_position, for any component. The pointer is465// good until the next structural change, so use it now and keep the handle.466kayan_get: proc<T>(w: *kayan_world in level, e: kayan_entity) -> option<*T in level> = {467 if (!kayan_alive(w, e)) {468 return null;469 }470 slot: kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*;471 t: *kayan_table in level = kayan_table_at(w, slot.table);472 if (!kayan_table_has(t, kayan_id_of<T>())) {473 return null;474 }475 rows: []T in level = kayan_slice<T>(t);476 return rows[cast(slot.row, u64)]&;477}478 479kayan_has: proc<T>(w: *kayan_world in level, e: kayan_entity) -> b32 = {480 return (kayan_signature(w, e) & kayan_mask_of<T>()) != 0;481}482 483// Write e's T. It must already have one: giving an entity a component it does484// not have changes its table, which is 2.6.485kayan_set: proc<T>(w: *kayan_world in level, e: kayan_entity, value: T) -> kayan_status = {486 if (!kayan_alive(w, e)) {487 return kayan_status.dead;488 }489 p: option<*T in level> = kayan_get<T>(w, e);490 if (!p) {491 return kayan_status.missing;492 }493 p* = value;494 return kayan_status.ok;495}496 497// Give e a T. If it already has one, this is kayan_set; if not, it moves to498// the table with T and then sets it.499kayan_add: proc<T>(r: region level, w: *kayan_world in level, e: kayan_entity, value: T) -> kayan_status = {500 if (!kayan_alive(w, e)) {501 return kayan_status.dead;502 }503 if (!kayan_has<T>(w, e)) {504 from: u32 = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.table;505 kayan_move(r, w, e, kayan_table_with(r, w, from, kayan_id_of<T>()));506 }507 return kayan_set<T>(w, e, value);508}509 510// Take e's T away, moving it to the table without T.511kayan_remove: proc<T>(r: region level, w: *kayan_world in level, e: kayan_entity) -> kayan_status = {512 if (!kayan_alive(w, e)) {513 return kayan_status.dead;514 }515 if (!kayan_has<T>(w, e)) {516 return kayan_status.missing;517 }518 from: u32 = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.table;519 kayan_move(r, w, e, kayan_table_without(r, w, from, kayan_id_of<T>()));520 return kayan_status.ok;521}522 523// The invariant, over every table: each row belongs to a live entity whose524// slot points back at that table and row, and the rows of all tables add up to525// the live count.526kayan_verify: proc(w: *kayan_world in level) -> u64 = {527 problems: u64 = 0;528 rows: u64 = 0;529 for (ti: u64 = 0; ti < w*.tables*.length; ti += 1) {530 t: *kayan_table in level = kayan_table_at(w, cast(ti, u32));531 for (row: u64 = 0; row < t*.count; row += 1) {532 e: kayan_entity = t*.entities[row];533 if (!kayan_alive(w, e)) {534 printfmt("kayan_verify: table {} row {} holds dead entity {}:{}\n", ti, row, e.index, e.generation);535 problems += 1;536 continue;537 }538 slot: kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*;539 if (cast(slot.table, u64) != ti || cast(slot.row, u64) != row) {540 printfmt("kayan_verify: {}:{} is in table {} row {}, its slot says table {} row {}\n",541 e.index, e.generation, ti, row, slot.table, slot.row);542 problems += 1;543 }544 }545 rows += t*.count;546 }547 if (rows != w*.live) {548 printfmt("kayan_verify: {} rows in tables, {} live entities\n", rows, w*.live);549 problems += 1;550 }551 return problems;552}553 554kayan_print_signature: proc(w: *kayan_world in level, signature: u64) -> void = {555 printfmt("{");556 first: b32 = true;557 for (id: u32 = 0; id < w*.component_count; id += 1) {558 if ((signature & kayan_bit(id)) != 0) {559 if (!first) {560 printfmt(" ");561 }562 printfmt("{}", kayan_name(w, id));563 first = false;564 }565 }566 printfmt("}");567}568 569kayan_print_tables: proc(w: *kayan_world in level) -> void = {570 for (ti: u64 = 0; ti < w*.tables*.length; ti += 1) {571 t: *kayan_table in level = kayan_table_at(w, cast(ti, u32));572 printfmt(" table {} ", cast(ti, u32));573 kayan_print_signature(w, t*.signature);574 printfmt(":");575 for (row: u64 = 0; row < t*.count; row += 1) {576 e: kayan_entity = t*.entities[row];577 printfmt(" {}:{}", e.index, e.generation);578 }579 printfmt("\n");580 }581}582 583// ---------------------------------------------------------------------------584// The program585// ---------------------------------------------------------------------------586 587spawn_drone: proc(r: region level, w: *kayan_world in level, x: f32, y: f32) -> kayan_entity = {588 e: kayan_entity = kayan_spawn(r, w, kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<health>());589 _ = kayan_set<position>(w, e, { .at = { .x = x, .y = y } });590 _ = kayan_set<velocity>(w, e, { .per_second = { .x = -1.0, .y = 0.0 } });591 _ = kayan_set<health>(w, e, { .points = 100.0, .max = 100.0 });592 return e;593}594 595ignite: proc(r: region level, w: *kayan_world in level, e: kayan_entity) -> void = {596 printfmt(" {}:{} catches fire: {}\n", e.index, e.generation,597 kayan_add<burning>(r, w, e, { .per_second = 10.0 }));598}599 600put_out: proc(r: region level, w: *kayan_world in level, e: kayan_entity) -> void = {601 printfmt(" {}:{} is put out: {}\n", e.index, e.generation, kayan_remove<burning>(r, w, e));602}603 604main: proc() -> i32 = {605 level: region = {606 w: *kayan_world in level = kayan_world_make(level);607 kayan_register(w, kayan_id_of<position>(), position<>&);608 kayan_register(w, kayan_id_of<velocity>(), velocity<>&);609 kayan_register(w, kayan_id_of<health>(), health<>&);610 kayan_register(w, kayan_id_of<damage>(), damage<>&);611 kayan_register(w, kayan_id_of<lifetime>(), lifetime<>&);612 kayan_register(w, kayan_id_of<turret>(), turret<>&);613 kayan_register(w, kayan_id_of<burning>(), burning<>&);614 615 d: [4]kayan_entity = {};616 for (i: u32 = 0; i < 4; i += 1) {617 d[i] = spawn_drone(level, w, 10.0 * cast(i + 1, f32), 0.0);618 }619 printfmt("4 drones:\n");620 kayan_print_tables(w);621 622 printfmt("frame 1\n");623 ignite(level, w, d[0]);624 ignite(level, w, d[2]);625 kayan_print_tables(w);626 627 printfmt("frame 2\n");628 put_out(level, w, d[0]);629 ignite(level, w, d[1]);630 ignite(level, w, d[2]);631 kayan_print_tables(w);632 633 printfmt("frame 3\n");634 put_out(level, w, d[1]);635 put_out(level, w, d[2]);636 put_out(level, w, d[3]);637 ignite(level, w, d[3]);638 kayan_print_tables(w);639 640 p: option<*position in level> = kayan_get<position>(w, d[2]);641 h: option<*health in level> = kayan_get<health>(w, d[2]);642 if (p && h) {643 printfmt("{}:{} moved tables twice and kept its data: at ({:.2}, {:.2}), health {:.0}\n", d[2].index, d[2].generation,644 cast(p*.at.x, f64), cast(p*.at.y, f64), cast(h*.points, f64));645 }646 printfmt("moves that looked up a table by signature: {}; that followed an edge: {}\n", w*.lookups, w*.edge_hits);647 printfmt("kayan_verify: {} problems\n", kayan_verify(w));648 }649 return 0;650}What add_remove prints. tools/check.py fails the chapter if a single byte differs.
4 drones:
table 0 {position velocity health}: 0:1 1:1 2:1 3:1
frame 1
0:1 catches fire: ok
2:1 catches fire: ok
table 0 {position velocity health}: 3:1 1:1
table 1 {position velocity health burning}: 0:1 2:1
frame 2
0:1 is put out: ok
1:1 catches fire: ok
2:1 catches fire: ok
table 0 {position velocity health}: 3:1 0:1
table 1 {position velocity health burning}: 2:1 1:1
frame 3
1:1 is put out: ok
2:1 is put out: ok
3:1 is put out: missing
3:1 catches fire: ok
table 0 {position velocity health}: 2:1 0:1 1:1
table 1 {position velocity health burning}: 3:1
2:1 moved tables twice and kept its data: at (30.00, 0.00), health 100
moves that looked up a table by signature: 1; that followed an edge: 6
kayan_verify: 0 problems
Build an ECS · part 2, components · 2.7
Inspecting the world
ecs/src/2.components/2.7.inspecting/inspecting.rin
In the dojo: 11. a reflection walk, on its own, to watch and to practise.
The world can store, copy and move any component, but it can't show you one. When something goes wrong at frame 3,000, you want to see an entity: every component it has, every field, with names. In C that means writing a printer per component and keeping each in step with its struct. In rin, the compiler already wrote down everything a printer needs. Each component's reflect record lists its fields with their names, types and byte offsets. The world registered those records back in 2.1 to learn component sizes. Here they pay off a second time.
kayan_inspect(w, e) every component of e, field by field
kayan_print_layout(w) every table, its columns, and their memory
Neither names a single component type. Add a component tomorrow and both print it.
Step 1: a component as bytes
Tools work from reflect records, not types, so they need the byte view:
kayan_get_bytes: proc(w: *kayan_world in level, e: kayan_entity, id: u32) -> []u8 in level = {
none: []u8 in level = {};
if (!kayan_alive(w, e) || id >= w*.component_count) {
return none;
}
slot: kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*;
t: *kayan_table in level = kayan_table_at(w, slot.table);
if (!kayan_table_has(t, id)) {
return none;
}
c: *kayan_column in level = t*.columns[id]&;
row: u64 = cast(slot.row, u64);
return c*.data[row * c*.size..(row + 1) * c*.size];
}
It's kayan_get<T>'s path, stopping one step early: the row's bytes as a slice, sized by the column. Game code uses the typed layer; tools use this.
Step 2: walking fields
kayan_print_fields: proc(type: *const reflect, base: []const u8, depth: u32) -> void = {
for (i: u64 = 0; i < type*.count; i += 1) {
f: option<*const reflect_field> = reflect_field_at(type, i);
if (!f) {
continue;
}
if (reflect_field_has_attr(f, "hidden")) {
continue;
}
at: []const u8 = base[f*.offset..f*.offset + f*.size];
kayan_indent(depth);
printfmt("{}:", f*.name);
info: option<*const reflect> = f*.info;
if (info != null && info*.kind == reflect_kind_struct) {
printfmt("\n");
kayan_print_fields(info, at, depth + 1);
} else {
printfmt(" ");
kayan_print_builtin(f*.type, at);
printfmt("\n");
}
}
}
For each field the record says, cut its bytes out of the component by offset and size. A struct field (at: vec2 inside position) has a record of its own, so the printer recurses into it. A builtin field (f32, u32) is printed by its type's name.
reflect_field_at is how checked code walks a record: it says when i is past the end, as a null, so there's no raw array to index. And because each field's bytes are cut with a bounds-checked slice, a record that disagreed with the data would stop the program instead of printing garbage.
That leaves the last step, turning bytes into a value, and it's the inspector's one trusted proc:
kayan_print_builtin: proc(type_name: ^const char, at: []const u8) -> void = {
if (at.length < 4) {
printfmt("<{} bytes>", at.length);
return;
}
unsafe {
if (strcmp(type_name, "f32") == 0) {
f: f32 = 0.0;
memcpy(f&, at.data, 4);
printfmt("{:.2}", cast(f, f64));
} else if (strcmp(type_name, "u32") == 0) {
u: u32 = 0;
memcpy(u&, at.data, 4);
printfmt("{}", u);
} else if (strcmp(type_name, "i32") == 0) {
i: i32 = 0;
memcpy(i&, at.data, 4);
printfmt("{}", i);
} else {
printfmt("<{}>", type_name);
}
}
}
This is unsafe for the same reason as the column's typed view in 2.2: bytes read as a type the checker can't know. And it believes as little. at carries its own length, so that's tested before the block. It copies into a local rather than casting the address, so a field at any offset is fine, aligned or not. And f32, u32 and i32 are plain numbers, so any four bytes are a valid one: a record that disagreed with the data would print a wrong number and nothing worse. Everything else in the inspector is checked.
Step 3: hiding bookkeeping
Fields can carry tags with @:
turret: struct = { cooldown: f32; timer: f32 @ "hidden"; }
The inspector skips anything tagged "hidden". The component decides what a tool shows; the tool doesn't need to know the component.
Step 4: an entity, and the layout
kayan_inspect walks the entity's signature and prints each component with kayan_print_fields. kayan_print_layout walks every table and prints each column's row size and bytes used out of bytes allocated, which shows where memory goes.
Run it
0:1 {position turret}
position
at:
x: 0.00
y: 0.00
turret
cooldown: 0.50
...
2:1 {position velocity health burning}
...
health
points: 60.00
max: 100.00
burning
per_second: 10.00
...
1:1 (dead)
table 1 {position velocity health}: 0 rows, room for 4
position 8 bytes a row, 0 of 32 bytes in use
velocity 8 bytes a row, 0 of 32 bytes in use
health 8 bytes a row, 0 of 32 bytes in use
The turret's timer isn't printed. Table 1 is empty but keeps its capacity: drone 1:1 was its only remaining row after 2:1 caught fire and moved out, and then 1:1 was despawned. Tables never shrink in this course. A real engine might trim empty ones between levels.
Part 2 in one picture
handle 2:1 ──> slots[2] { generation 1, table 2, row 0 }
│
table 2 {position velocity health burning}
entities [ 2:1 ]
position [ (30, -5) ] velocity [ (-1, 0) ] health [ 60/100 ] burning [ 10 ]
add_edges / remove_edges ──> neighbouring tables in the archetype graph
The storage half of the ECS is done: identity (Part 1), and any component, in any combination, changing at run time (Part 2). Part 3 is the behaviour: queries to find tables, and systems to run over them.
And here's how it stayed safe. Everything lives in level and ends with it, so nothing can be freed early. Things that may be missing are options, so nothing reads a null. Tables never move, so a table pointer is always good. A column can move, but its old block stays in the region, so a stale pointer is a wrong answer and never a bad address, and handles are the cure for wrong answers. In kayan, the only code the checker takes on trust is the three unsafe { } blocks where bytes meet types, and those check what they're handed instead of believing it.
Try
- Add a field to
healthand run it. Nothing in the inspector changes. - Tag
health.maxas hidden. - Write
kayan_set_f32_by_name(w, e, "health", "points", 10.0)with the reflect records: find the component by name, the field by name, check it's anf32, write it. That's the core of an editor or a console. The search is all checked code. The last step, putting anf32into bytes, iskayan_print_builtinin reverse, so give it a smallunsafe { }of its own: test the slice's length first, thenmemcpyfrom a local.
1// 2.7 Inspecting the world2//3// The world can store, copy and move any component, but it cannot show you4// one. When something goes wrong at frame 3,000 you want to see an entity:5// every component it has, every field, with names. In C that means writing a6// printer per component and keeping it in step with the struct. In rin the7// compiler already wrote down everything a printer needs: each component's8// reflect record lists its fields with their names, types and offsets. The9// world registered those records in 2.1 to know component sizes; here they10// pay off a second time.11//12// kayan_inspect(w, e) every component of e, field by field13// kayan_print_layout(w) every table, its columns, and their memory14//15// Neither names a single component type. Add a component tomorrow and both16// print it.17//18// The inspector walks the reflect record with19// `reflect_field_at`, which says when it is past the end, and cuts each20// field's bytes out of the component's bytes as a slice, bounds-checked. The21// one place that reads bytes as a value, `kayan_print_builtin`, is the22// chapter's third trusted proc, beside the column's two.23 24import "std/print.rin"25import "std/memops.rin"26import "std/vec.rin"27import "std/hashmap.rin"28import "std/slice.rin"29 30level: region;31 32vec2: struct = {33 x: f32;34 y: f32;35}36 37position: struct = { at: vec2; }38velocity: struct = { per_second: vec2; }39health: struct = { points: f32; max: f32; }40damage: struct = { amount: f32; }41lifetime: struct = { seconds: f32; }42// A field tagged "hidden" is bookkeeping the inspector does not show.43turret: struct = { cooldown: f32; timer: f32 @ "hidden"; }44burning: struct = { per_second: f32; }45 46component: enum = {47 position,48 velocity,49 health,50 damage,51 lifetime,52 turret,53 burning,54}55 56component_of: proc<position>() -> component = { return component.position; }57component_of: proc<velocity>() -> component = { return component.velocity; }58component_of: proc<health>() -> component = { return component.health; }59component_of: proc<damage>() -> component = { return component.damage; }60component_of: proc<lifetime>() -> component = { return component.lifetime; }61component_of: proc<turret>() -> component = { return component.turret; }62component_of: proc<burning>() -> component = { return component.burning; }63 64// ---------------------------------------------------------------------------65// Entities (Part 1)66// ---------------------------------------------------------------------------67 68kayan_entity: struct = {69 index: u32;70 generation: u32;71}72 73// `missing`: the entity is alive but has no such component.74kayan_status: enum[must_use] = {75 ok,76 dead,77 missing,78}79 80// Where an entity lives: which table, and which row of it.81kayan_slot: struct = {82 generation: u32;83 table: u32;84 row: u32;85}86 87// ---------------------------------------------------------------------------88// Columns (2.2)89// ---------------------------------------------------------------------------90 91kayan_column: struct = {92 data: []u8 in level;93 size: u64;94 align: u64;95}96 97kayan_column_init: proc(c: *kayan_column, info: *const reflect) -> void = {98 c*.size = info*.size;99 c*.align = info*.align;100}101 102kayan_column_grow: proc(r: region level, c: *kayan_column, count: u64, capacity: u64) -> void = {103 unsafe {104 raw: ^u8 = cast(memops_arena_push_zero(r, c*.size * capacity, c*.align), ^u8);105 bytes: []u8 in level = cast(raw[0..c*.size * capacity], []u8 in level);106 // Slicing checks both ends: a count past either block stops the program.107 kept: u64 = c*.size * count;108 slice<u8>copy(bytes[0..kept], c*.data[0..kept]);109 c*.data = bytes;110 }111}112 113kayan_column_copy_row: proc(c: *kayan_column, to: u64, from: u64) -> void = {114 for (i: u64 = 0; i < c*.size; i += 1) {115 c*.data[to * c*.size + i] = c*.data[from * c*.size + i];116 }117}118 119// Row `from` of one column over row `to` of another: the same component in120// two tables, as a move between tables copies it.121kayan_column_copy_row_from: proc(dst: *kayan_column, to: u64, src: *const kayan_column, from: u64) -> void = {122 for (i: u64 = 0; i < dst*.size; i += 1) {123 dst*.data[to * dst*.size + i] = src*.data[from * src*.size + i];124 }125}126 127kayan_column_zero_row: proc(c: *kayan_column, row: u64) -> void = {128 for (i: u64 = 0; i < c*.size; i += 1) {129 c*.data[row * c*.size + i] = 0;130 }131}132 133// The typed view: the column's first `count` rows as `[]T`. This is the one134// place bytes become a type, so it takes nothing on trust. T must be the135// column's size, and plain data: any bytes are then a valid T, so reading the136// wrong type gives wrong numbers, never a bad pointer. The rows must fit, which137// the slice knows and no field can lie about, and the bytes must be aligned for138// T. The view is `[]T in level`: the bytes are in the region, so the view is139// good memory for as long as the region is. After the column grows it is the140// old block, still readable, no longer the column: take the view again.141kayan_column_slice: proc<T>(c: *const kayan_column, count: u64) -> []T in level = {142 if (c*.size != sizeof(T)) {143 printfmt("kayan: a column of {}-byte rows read as a {}-byte type\n", c*.size, sizeof(T));144 exit(1);145 }146 if (!reflect_is_plain(T<>&)) {147 printfmt("kayan: a column holds plain data, and this {}-byte type is not\n", sizeof(T));148 exit(1);149 }150 if (count > c*.data.length / sizeof(T)) {151 printfmt("kayan: a view of {} rows on a column with room for {}\n", count, c*.data.length / sizeof(T));152 exit(1);153 }154 unsafe {155 if (cast(c*.data.data, usize) % alignof(T) != 0) {156 printfmt("kayan: a column's bytes are not aligned to {}\n", alignof(T));157 exit(1);158 }159 raw: ^T = cast(c*.data.data, ^T);160 return cast(raw[0..count], []T in level);161 }162}163 164// ---------------------------------------------------------------------------165// The table166// ---------------------------------------------------------------------------167 168// `columns[id]` is in use when the signature has bit `id`; the others stay169// zeroed. Count and capacity are the table's, shared by its columns and by170// `entities`, so they can never disagree.171//172// The edges: `add_edges[id]` is the table this one's entities move to when173// they gain component `id`, and `remove_edges[id]` when they lose it. Each is174// the table's index plus one, so 0, which a new table starts with, means "not175// known yet".176kayan_table: struct = {177 signature: u64;178 count: u64;179 capacity: u64;180 entities: []kayan_entity in level;181 columns: [64]kayan_column;182 add_edges: [64]u32;183 remove_edges: [64]u32;184}185 186// The world: the registry, the slots and free list, and every table, with a187// map from signature to table index so finding one is a lookup, not a search.188// Tables are referred to by index in the slots, and by pointer everywhere189// else: a table is made once in the region and never moves.190kayan_world: struct = {191 infos: [64]option<*const reflect in permanent>;192 component_count: u32;193 tables: *vec<*kayan_table in level> in level;194 by_signature: *hashmap<u64, u32> in level;195 slots: *vec<kayan_slot> in level;196 free: *vec<u32> in level;197 live: u64;198 // How table moves found their destination, for this chapter's output.199 lookups: u64;200 edge_hits: u64;201}202 203kayan_bit: proc(id: u32) -> u64 = {204 return 1'u64 shl cast(id, u64);205}206 207kayan_id_of: proc<T>() -> u32 = {208 return cast(component_of<T>(), u32);209}210 211kayan_mask_of: proc<T>() -> u64 = {212 return kayan_bit(kayan_id_of<T>());213}214 215kayan_register: proc(w: *kayan_world in level, id: u32, info: *const reflect in permanent) -> void = {216 if (id >= 64) {217 printfmt("kayan: component id {} is past the 64 a signature can hold\n", id);218 exit(1);219 }220 w*.infos[id] = info;221 if (id + 1 > w*.component_count) {222 w*.component_count = id + 1;223 }224}225 226// A component's name, for printing: a registered id's, or "?" for a slot227// nobody filled.228kayan_name: proc(w: *kayan_world in level, id: u32) -> ^const char = {229 info: option<*const reflect in permanent> = w*.infos[id];230 return info ? info*.name : "?";231}232 233kayan_table_has: proc(t: *kayan_table in level, id: u32) -> b32 = {234 return (t*.signature & kayan_bit(id)) != 0;235}236 237// A table for a signature: one empty column per component in it, sized from238// the registry. Every component in the signature must be registered.239kayan_table_init: proc(w: *kayan_world in level, t: *kayan_table in level, signature: u64) -> void = {240 t*.signature = signature;241 for (id: u32 = 0; id < 64; id += 1) {242 if ((signature & kayan_bit(id)) != 0) {243 info: option<*const reflect in permanent> = id < w*.component_count ? w*.infos[id] : null;244 if (!info) {245 printfmt("kayan: signature {} uses component {}, which was never registered\n", signature, id);246 exit(1);247 } else {248 kayan_column_init(t*.columns[id]&, info);249 }250 }251 }252}253 254// Double the room in `entities` and in every column at once.255kayan_table_grow: proc(r: region level, w: *kayan_world in level, t: *kayan_table in level) -> void = {256 capacity: u64 = t*.capacity == 0 ? 4 : t*.capacity * 2;257 entities: []kayan_entity in level = new_slice<kayan_entity>(r, capacity);258 for (row: u64 = 0; row < t*.count; row += 1) {259 entities[row] = t*.entities[row];260 }261 t*.entities = entities;262 for (id: u32 = 0; id < w*.component_count; id += 1) {263 if (kayan_table_has(t, id)) {264 kayan_column_grow(r, t*.columns[id]&, t*.count, capacity);265 }266 }267 t*.capacity = capacity;268}269 270// A new row at the end, owned by `e`, with every component zeroed.271kayan_table_add_row: proc(r: region level, w: *kayan_world in level, t: *kayan_table in level, e: kayan_entity) -> u64 = {272 if (t*.count == t*.capacity) {273 kayan_table_grow(r, w, t);274 }275 row: u64 = t*.count;276 t*.count += 1;277 t*.entities[row] = e;278 for (id: u32 = 0; id < w*.component_count; id += 1) {279 if (kayan_table_has(t, id)) {280 kayan_column_zero_row(t*.columns[id]&, row);281 }282 }283 return row;284}285 286// 1.4's swap-remove, for every column: the last row is copied over `row`, and287// the slot of the entity that owned the last row is told where it went.288kayan_table_remove_row: proc(w: *kayan_world in level, t: *kayan_table in level, row: u64) -> void = {289 last: u64 = t*.count - 1;290 if (row != last) {291 for (id: u32 = 0; id < w*.component_count; id += 1) {292 if (kayan_table_has(t, id)) {293 kayan_column_copy_row(t*.columns[id]&, row, last);294 }295 }296 moved: kayan_entity = t*.entities[last];297 t*.entities[row] = moved;298 vec<kayan_slot>at(w*.slots, cast(moved.index, u64))*.row = cast(row, u32);299 }300 t*.count = last;301}302 303// The typed view of one of the table's columns: T's rows, all of them in use.304// Asking for a component the table does not have is a bug, so it stops.305kayan_slice: proc<T>(t: *kayan_table in level) -> []T in level = {306 id: u32 = kayan_id_of<T>();307 if (!kayan_table_has(t, id)) {308 printfmt("kayan: the table has no column for component {}\n", id);309 exit(1);310 }311 return kayan_column_slice<T>(t*.columns[id]&, t*.count);312}313 314// The table at an index: a pointer into the region, good for as long as the315// level is. The vec of pointers may grow, but what it points at does not move.316kayan_table_at: proc(w: *kayan_world in level, index: u32) -> *kayan_table in level = {317 return vec<*kayan_table in level>at(w*.tables, cast(index, u64))*;318}319 320// The table for a signature: found if it exists, made if it does not. This is321// where archetypes come from.322kayan_table_for: proc(r: region level, w: *kayan_world in level, signature: u64) -> u32 = {323 found: option<*u32> = hashmap<u64, u32>get(w*.by_signature, signature);324 if (found) {325 return found*;326 }327 index: u32 = cast(w*.tables*.length, u32);328 t: *kayan_table in level = new<kayan_table>(r);329 kayan_table_init(w, t, signature);330 _ = vec<*kayan_table in level>push(w*.tables, t);331 _ = hashmap<u64, u32>set(w*.by_signature, signature, index);332 return index;333}334 335// ---------------------------------------------------------------------------336// The world337// ---------------------------------------------------------------------------338 339kayan_world_make: proc(r: region level) -> *kayan_world in level = {340 return new<kayan_world>(r, {341 .tables = vec<*kayan_table in level>make(r, 8),342 .by_signature = hashmap<u64, u32>make(r),343 .slots = vec<kayan_slot>make(r, 8),344 .free = vec<u32>make(r, 8),345 });346}347 348kayan_alive: proc(w: *kayan_world in level, e: kayan_entity) -> b32 = {349 if (cast(e.index, u64) >= w*.slots*.length) {350 return false;351 }352 return vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.generation == e.generation;353}354 355// A new entity with the components in `signature`, all zeroed, in the table356// for that signature.357kayan_spawn: proc(r: region level, w: *kayan_world in level, signature: u64) -> kayan_entity = {358 e: kayan_entity = {};359 if (!vec<u32>is_empty(w*.free)) {360 e.index = vec<u32>pop(w*.free);361 } else {362 e.index = cast(w*.slots*.length, u32);363 _ = vec<kayan_slot>push(w*.slots, { .generation = 1 });364 }365 e.generation = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.generation;366 table_index: u32 = kayan_table_for(r, w, signature);367 t: *kayan_table in level = kayan_table_at(w, table_index);368 row: u64 = kayan_table_add_row(r, w, t, e);369 // Both calls were handed the world, so the pointer into the slots is370 // taken after them.371 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));372 slot*.table = table_index;373 slot*.row = cast(row, u32);374 w*.live += 1;375 return e;376}377 378kayan_despawn: proc(w: *kayan_world in level, e: kayan_entity) -> kayan_status = {379 if (!kayan_alive(w, e)) {380 return kayan_status.dead;381 }382 slot: kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*;383 kayan_table_remove_row(w, kayan_table_at(w, slot.table), cast(slot.row, u64));384 vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.generation += 1;385 _ = vec<u32>push(w*.free, e.index);386 w*.live -= 1;387 return kayan_status.ok;388}389 390// The components a live entity has: its table's signature. 0 for a dead one.391kayan_signature: proc(w: *kayan_world in level, e: kayan_entity) -> u64 = {392 if (!kayan_alive(w, e)) {393 return 0;394 }395 return kayan_table_at(w, vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.table)*.signature;396}397 398// ---------------------------------------------------------------------------399// Moving between tables400// ---------------------------------------------------------------------------401 402// The table an entity of table `from` moves to when it gains component `id`.403// The first time, it is found by signature, which can make the table; the404// answer is written on both tables, as from's add edge and the destination's405// remove edge, so neither direction is looked up again.406kayan_table_with: proc(r: region level, w: *kayan_world in level, from: u32, id: u32) -> u32 = {407 edge: u32 = kayan_table_at(w, from)*.add_edges[id];408 if (edge != 0) {409 w*.edge_hits += 1;410 return edge - 1;411 }412 w*.lookups += 1;413 to: u32 = kayan_table_for(r, w, kayan_table_at(w, from)*.signature | kayan_bit(id));414 kayan_table_at(w, from)*.add_edges[id] = to + 1;415 kayan_table_at(w, to)*.remove_edges[id] = from + 1;416 return to;417}418 419// The same for losing component `id`.420kayan_table_without: proc(r: region level, w: *kayan_world in level, from: u32, id: u32) -> u32 = {421 edge: u32 = kayan_table_at(w, from)*.remove_edges[id];422 if (edge != 0) {423 w*.edge_hits += 1;424 return edge - 1;425 }426 w*.lookups += 1;427 to: u32 = kayan_table_for(r, w, kayan_table_at(w, from)*.signature & ~kayan_bit(id));428 kayan_table_at(w, from)*.remove_edges[id] = to + 1;429 kayan_table_at(w, to)*.add_edges[id] = from + 1;430 return to;431}432 433// Move a live entity to table `to_index`. Components both tables have are434// copied, byte for byte; components only the new table has start zeroed;435// components only the old table has are dropped with its old row.436kayan_move: proc(r: region level, w: *kayan_world in level, e: kayan_entity, to_index: u32) -> void = {437 slot: kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*;438 from_index: u32 = slot.table;439 if (from_index == to_index) {440 return;441 }442 from: *kayan_table in level = kayan_table_at(w, from_index);443 to: *kayan_table in level = kayan_table_at(w, to_index);444 from_row: u64 = cast(slot.row, u64);445 to_row: u64 = kayan_table_add_row(r, w, to, e);446 for (id: u32 = 0; id < w*.component_count; id += 1) {447 if (kayan_table_has(from, id) && kayan_table_has(to, id)) {448 kayan_column_copy_row_from(to*.columns[id]&, to_row, from*.columns[id]&, from_row);449 }450 }451 kayan_table_remove_row(w, from, from_row);452 // The slot is written after the calls that were handed the world.453 moved: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));454 moved*.table = to_index;455 moved*.row = cast(to_row, u32);456}457 458// ---------------------------------------------------------------------------459// The byte layer, for tools460// ---------------------------------------------------------------------------461 462// e's component `id` as its bytes: an empty slice if e is dead or has no such463// component. The typed layer is for game code, which knows its types; this is464// for tools that work from reflect records instead.465kayan_get_bytes: proc(w: *kayan_world in level, e: kayan_entity, id: u32) -> []u8 in level = {466 none: []u8 in level = {};467 if (!kayan_alive(w, e) || id >= w*.component_count) {468 return none;469 }470 slot: kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*;471 t: *kayan_table in level = kayan_table_at(w, slot.table);472 if (!kayan_table_has(t, id)) {473 return none;474 }475 c: *kayan_column in level = t*.columns[id]&;476 row: u64 = cast(slot.row, u64);477 return c*.data[row * c*.size..(row + 1) * c*.size];478}479 480// ---------------------------------------------------------------------------481// The typed layer482// ---------------------------------------------------------------------------483 484// e's T, or null if e is dead or has no T. Handle, slot, table, row, typed485// view: the same path as 1.4's get_position, for any component. The pointer is486// good until the next structural change, so use it now and keep the handle.487kayan_get: proc<T>(w: *kayan_world in level, e: kayan_entity) -> option<*T in level> = {488 if (!kayan_alive(w, e)) {489 return null;490 }491 slot: kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*;492 t: *kayan_table in level = kayan_table_at(w, slot.table);493 if (!kayan_table_has(t, kayan_id_of<T>())) {494 return null;495 }496 rows: []T in level = kayan_slice<T>(t);497 return rows[cast(slot.row, u64)]&;498}499 500kayan_has: proc<T>(w: *kayan_world in level, e: kayan_entity) -> b32 = {501 return (kayan_signature(w, e) & kayan_mask_of<T>()) != 0;502}503 504// Write e's T. It must already have one: giving an entity a component it does505// not have changes its table, which is 2.6.506kayan_set: proc<T>(w: *kayan_world in level, e: kayan_entity, value: T) -> kayan_status = {507 if (!kayan_alive(w, e)) {508 return kayan_status.dead;509 }510 p: option<*T in level> = kayan_get<T>(w, e);511 if (!p) {512 return kayan_status.missing;513 }514 p* = value;515 return kayan_status.ok;516}517 518// Give e a T. If it already has one, this is kayan_set; if not, it moves to519// the table with T and then sets it.520kayan_add: proc<T>(r: region level, w: *kayan_world in level, e: kayan_entity, value: T) -> kayan_status = {521 if (!kayan_alive(w, e)) {522 return kayan_status.dead;523 }524 if (!kayan_has<T>(w, e)) {525 from: u32 = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.table;526 kayan_move(r, w, e, kayan_table_with(r, w, from, kayan_id_of<T>()));527 }528 return kayan_set<T>(w, e, value);529}530 531// Take e's T away, moving it to the table without T.532kayan_remove: proc<T>(r: region level, w: *kayan_world in level, e: kayan_entity) -> kayan_status = {533 if (!kayan_alive(w, e)) {534 return kayan_status.dead;535 }536 if (!kayan_has<T>(w, e)) {537 return kayan_status.missing;538 }539 from: u32 = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.table;540 kayan_move(r, w, e, kayan_table_without(r, w, from, kayan_id_of<T>()));541 return kayan_status.ok;542}543 544// The invariant, over every table: each row belongs to a live entity whose545// slot points back at that table and row, and the rows of all tables add up to546// the live count.547kayan_verify: proc(w: *kayan_world in level) -> u64 = {548 problems: u64 = 0;549 rows: u64 = 0;550 for (ti: u64 = 0; ti < w*.tables*.length; ti += 1) {551 t: *kayan_table in level = kayan_table_at(w, cast(ti, u32));552 for (row: u64 = 0; row < t*.count; row += 1) {553 e: kayan_entity = t*.entities[row];554 if (!kayan_alive(w, e)) {555 printfmt("kayan_verify: table {} row {} holds dead entity {}:{}\n", ti, row, e.index, e.generation);556 problems += 1;557 continue;558 }559 slot: kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*;560 if (cast(slot.table, u64) != ti || cast(slot.row, u64) != row) {561 printfmt("kayan_verify: {}:{} is in table {} row {}, its slot says table {} row {}\n",562 e.index, e.generation, ti, row, slot.table, slot.row);563 problems += 1;564 }565 }566 rows += t*.count;567 }568 if (rows != w*.live) {569 printfmt("kayan_verify: {} rows in tables, {} live entities\n", rows, w*.live);570 problems += 1;571 }572 return problems;573}574 575kayan_print_signature: proc(w: *kayan_world in level, signature: u64) -> void = {576 printfmt("{");577 first: b32 = true;578 for (id: u32 = 0; id < w*.component_count; id += 1) {579 if ((signature & kayan_bit(id)) != 0) {580 if (!first) {581 printfmt(" ");582 }583 printfmt("{}", kayan_name(w, id));584 first = false;585 }586 }587 printfmt("}");588}589 590kayan_print_tables: proc(w: *kayan_world in level) -> void = {591 for (ti: u64 = 0; ti < w*.tables*.length; ti += 1) {592 t: *kayan_table in level = kayan_table_at(w, cast(ti, u32));593 printfmt(" table {} ", cast(ti, u32));594 kayan_print_signature(w, t*.signature);595 printfmt(":");596 for (row: u64 = 0; row < t*.count; row += 1) {597 e: kayan_entity = t*.entities[row];598 printfmt(" {}:{}", e.index, e.generation);599 }600 printfmt("\n");601 }602}603 604// ---------------------------------------------------------------------------605// The inspector: reflect records, no component types606// ---------------------------------------------------------------------------607 608kayan_indent: proc(depth: u32) -> void = {609 for (i: u32 = 0; i < depth; i += 1) {610 printfmt(" ");611 }612}613 614// One value of a builtin type, read from its bytes by its type's name. The615// bytes are reinterpreted, which is why this is trusted; it checks there are616// enough of them first.617kayan_print_builtin: proc(type_name: ^const char, at: []const u8) -> void = {618 if (at.length < 4) {619 printfmt("<{} bytes>", at.length);620 return;621 }622 unsafe {623 if (strcmp(type_name, "f32") == 0) {624 f: f32 = 0.0;625 memcpy(f&, at.data, 4);626 printfmt("{:.2}", cast(f, f64));627 } else if (strcmp(type_name, "u32") == 0) {628 u: u32 = 0;629 memcpy(u&, at.data, 4);630 printfmt("{}", u);631 } else if (strcmp(type_name, "i32") == 0) {632 i: i32 = 0;633 memcpy(i&, at.data, 4);634 printfmt("{}", i);635 } else {636 printfmt("<{}>", type_name);637 }638 }639}640 641// Every field of a struct whose bytes are `base`, nested structs indented642// under their field. A field's bytes are cut from `base` by its offset and643// size, both from the reflect record.644kayan_print_fields: proc(type: *const reflect, base: []const u8, depth: u32) -> void = {645 for (i: u64 = 0; i < type*.count; i += 1) {646 f: option<*const reflect_field> = reflect_field_at(type, i);647 if (!f) {648 continue;649 }650 if (reflect_field_has_attr(f, "hidden")) {651 continue;652 }653 at: []const u8 = base[f*.offset..f*.offset + f*.size];654 kayan_indent(depth);655 printfmt("{}:", f*.name);656 info: option<*const reflect> = f*.info;657 if (info != null && info*.kind == reflect_kind_struct) {658 printfmt("\n");659 kayan_print_fields(info, at, depth + 1);660 } else {661 printfmt(" ");662 kayan_print_builtin(f*.type, at);663 printfmt("\n");664 }665 }666}667 668kayan_inspect: proc(w: *kayan_world in level, e: kayan_entity) -> void = {669 if (!kayan_alive(w, e)) {670 printfmt("{}:{} (dead)\n", e.index, e.generation);671 return;672 }673 printfmt("{}:{} ", e.index, e.generation);674 signature: u64 = kayan_signature(w, e);675 kayan_print_signature(w, signature);676 printfmt("\n");677 for (id: u32 = 0; id < w*.component_count; id += 1) {678 if ((signature & kayan_bit(id)) != 0) {679 info: option<*const reflect in permanent> = w*.infos[id];680 if (info) {681 printfmt(" {}\n", info*.name);682 bytes: []u8 in level = kayan_get_bytes(w, e, id);683 kayan_print_fields(info, bytes, 2);684 }685 }686 }687}688 689// Every table: rows in use out of room, and per column its row size and its690// bytes in use out of bytes allocated. The memory a table holds and does not691// use is the cost of doubling on growth.692kayan_print_layout: proc(w: *kayan_world in level) -> void = {693 for (ti: u32 = 0; ti < cast(w*.tables*.length, u32); ti += 1) {694 t: *kayan_table in level = kayan_table_at(w, ti);695 printfmt("table {} ", ti);696 kayan_print_signature(w, t*.signature);697 printfmt(": {} rows, room for {}\n", cast(t*.count, u32), cast(t*.capacity, u32));698 for (id: u32 = 0; id < w*.component_count; id += 1) {699 if (kayan_table_has(t, id)) {700 size: u64 = t*.columns[id].size;701 printfmt(" {:-9} {:2} bytes a row, {:4} of {:4} bytes in use\n", kayan_name(w, id), cast(size, u32),702 cast(size * t*.count, u32), cast(size * t*.capacity, u32));703 }704 }705 }706}707 708// ---------------------------------------------------------------------------709// The program710// ---------------------------------------------------------------------------711 712spawn_drone: proc(r: region level, w: *kayan_world in level, x: f32, y: f32) -> kayan_entity = {713 e: kayan_entity = kayan_spawn(r, w, kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<health>());714 _ = kayan_set<position>(w, e, { .at = { .x = x, .y = y } });715 _ = kayan_set<velocity>(w, e, { .per_second = { .x = -1.0, .y = 0.0 } });716 _ = kayan_set<health>(w, e, { .points = 100.0, .max = 100.0 });717 return e;718}719 720main: proc() -> i32 = {721 level: region = {722 w: *kayan_world in level = kayan_world_make(level);723 kayan_register(w, kayan_id_of<position>(), position<>&);724 kayan_register(w, kayan_id_of<velocity>(), velocity<>&);725 kayan_register(w, kayan_id_of<health>(), health<>&);726 kayan_register(w, kayan_id_of<damage>(), damage<>&);727 kayan_register(w, kayan_id_of<lifetime>(), lifetime<>&);728 kayan_register(w, kayan_id_of<turret>(), turret<>&);729 kayan_register(w, kayan_id_of<burning>(), burning<>&);730 731 gun: kayan_entity = kayan_spawn(level, w, kayan_mask_of<position>() | kayan_mask_of<turret>());732 _ = kayan_set<turret>(w, gun, { .cooldown = 0.5, .timer = 0.25 });733 a: kayan_entity = spawn_drone(level, w, 20.0, 5.0);734 b: kayan_entity = spawn_drone(level, w, 30.0, -5.0);735 _ = kayan_add<burning>(level, w, b, { .per_second = 10.0 });736 h: option<*health in level> = kayan_get<health>(w, b);737 if (h) {738 h*.points = 60.0;739 }740 shot: kayan_entity = kayan_spawn(level, w, kayan_mask_of<position>() | kayan_mask_of<velocity>()741 | kayan_mask_of<damage>() | kayan_mask_of<lifetime>());742 _ = kayan_set<velocity>(w, shot, { .per_second = { .x = 8.0, .y = 0.0 } });743 _ = kayan_set<damage>(w, shot, { .amount = 25.0 });744 _ = kayan_set<lifetime>(w, shot, { .seconds = 2.0 });745 746 kayan_inspect(w, gun);747 kayan_inspect(w, a);748 kayan_inspect(w, b);749 kayan_inspect(w, shot);750 _ = kayan_despawn(w, a);751 kayan_inspect(w, a);752 printfmt("\n");753 kayan_print_layout(w);754 printfmt("kayan_verify: {} problems\n", kayan_verify(w));755 }756 return 0;757}What inspecting prints. tools/check.py fails the chapter if a single byte differs.
0:1 {position turret}
position
at:
x: 0.00
y: 0.00
turret
cooldown: 0.50
1:1 {position velocity health}
position
at:
x: 20.00
y: 5.00
velocity
per_second:
x: -1.00
y: 0.00
health
points: 100.00
max: 100.00
2:1 {position velocity health burning}
position
at:
x: 30.00
y: -5.00
velocity
per_second:
x: -1.00
y: 0.00
health
points: 60.00
max: 100.00
burning
per_second: 10.00
3:1 {position velocity damage lifetime}
position
at:
x: 0.00
y: 0.00
velocity
per_second:
x: 8.00
y: 0.00
damage
amount: 25.00
lifetime
seconds: 2.00
1:1 (dead)
table 0 {position turret}: 1 rows, room for 4
position 8 bytes a row, 8 of 32 bytes in use
turret 8 bytes a row, 8 of 32 bytes in use
table 1 {position velocity health}: 0 rows, room for 4
position 8 bytes a row, 0 of 32 bytes in use
velocity 8 bytes a row, 0 of 32 bytes in use
health 8 bytes a row, 0 of 32 bytes in use
table 2 {position velocity health burning}: 1 rows, room for 4
position 8 bytes a row, 8 of 32 bytes in use
velocity 8 bytes a row, 8 of 32 bytes in use
health 8 bytes a row, 8 of 32 bytes in use
burning 4 bytes a row, 4 of 16 bytes in use
table 3 {position velocity damage lifetime}: 1 rows, room for 4
position 8 bytes a row, 8 of 32 bytes in use
velocity 8 bytes a row, 8 of 32 bytes in use
damage 4 bytes a row, 4 of 16 bytes in use
lifetime 4 bytes a row, 4 of 16 bytes in use
kayan_verify: 0 problems
Build an ECS · part 3, systems · 3.1
Queries
ecs/src/3.systems/3.1.queries/queries.rin
In the dojo: 6. bitsets, 12. a watermark, each on its own, to watch and to practise.
Part 2 finished the storage. Everything from here on is behaviour, and in an ECS behaviour lives in systems: code that runs over every entity with some set of components. "Every entity with a position and a velocity" isn't in one place. It's spread over every table whose signature has both bits. So a system's first job is finding those tables, and the thing that finds them is a query:
all {position velocity}, none {} -> tables 1, 2, 3
A query matches tables, not entities. A game has a handful of tables and rarely makes new ones, so the matching is done once and kept. Each frame, a system walks the kept list, and inside each table streams the columns.
First: the storage, ready for systems
Part 3 is about to call kayan from deep inside a frame, and that changes a few things about the storage. Open this chapter's file beside 2.7's and you'll see them, so here they are. The design is the same. What moved is who holds what.
The world remembers its region. In Part 2, every proc that could allocate took r: region level, and that was fine with main at the top of every call. A system won't have level to hand, so the world keeps it, the way a vec keeps its own:
kayan_heap: struct = {
arena: ^memops_arena;
}
kayan_spawn(w, signature) no longer takes a region. kayan_heap_make sets this up once, inside kayan_world_make, and nothing above the byte core hands an arena around.
Procs that only read say so. kayan_alive, kayan_get, kayan_verify and the printers take *const kayan_world.
Column slices are checked, and the caller says how long it wants one. This is the proc every system lives on, and there's no unsafe in it:
kayan_slice: proc<T>(t: *const kayan_table) -> []T in level = {
id: u32 = kayan_id_of<T>();
c: option<*kayan_column in level> = t*.columns[id];
if (!c) {
printfmt("kayan: the table has no column for component {}\n", id);
exit(1);
}
return kayan_column_slice<T>(c, t*.count);
}
The result is in level, because that's the truth about the memory. What a caller does with it is up to the local it goes in. Write ps: []position, with no region, and it's the kind of borrow 2.5 ended on: keep it across a call that may change the world it came from, and the checker refuses.
type error: 'keep' may be stale: 'w', which it was taken from, has been changed since; take it again
Nothing in the core believes the table. 2.2's rule matters more here, because now there's a lot of code above the core. rin has no private fields, so any proc holding a table can write t*.count, and kayan_slice passes that field straight down. It's safe anyway. The view measures count against the column's slice, and stops the program if it doesn't fit. A row's bytes are the same story:
kayan_column_bytes: proc(c: *kayan_column, row: u64) -> []u8 in level = {
return c*.data[row * c*.size..(row + 1) * c*.size];
}
That needs no unsafe, and it doesn't ask its caller to keep the row in range. data is a slice, so a wrong row, or a wrong size, is a bounds failure wherever it came from. Every proc above it, however it got its numbers, inherits that. This is the design: checked code can have every kind of bug, and the worst it can do to a column is stop the program or get a wrong number.
**The unsafe { } blocks are six**, each a few lines with an honest signature, and every other proc in the file is checked:
kayan_heap_make: keeping the region's arena where the core can reach it;kayan_column_make,kayan_table_new: a column and a table allocated from that arena;kayan_column_grow,kayan_column_slice: 2.2's core;kayan_print_builtin: 2.7's bytes read as a value.
One thing in these blocks is still a field believed on its word: the heap's arena pointer has no bound for anyone to check. rin seals a struct holding a raw pointer to the file that declares it, so nothing importing this file can write that field. Inside the file it's on the author: only kayan_heap_make writes it. That's the same line Rust draws at a module, and it's what unsafe code owes.
Tables are still allocated one by one and never move. A column still does, and 3.3 is about that.
Step 1: the query
kayan_query: struct = {
all: u64;
none: u64;
tables: *vec<u32> in level;
checked: u64;
}
all: a table must have every one of these components.none: and none of these. "Drones that aren't burning" isall {health}, none {burning}.tables: the matching tables, by index. An index is what a slot holds too: plain data, a few bytes, with no region to name.kayan_query_tableturns one into a pointer when a walk needs it.checked: how many of the world's tables the query has looked at.
The list of tables lives in the world's region, so a query is told which region it's made in. Making one costs nothing: it matches when first updated.
kayan_query_init: proc(r: region level, all: u64, none: u64) -> kayan_query = {
return { .all = all, .none = none, .tables = vec<u32>make(r, 4) };
}
Matching is two ANDs on the signature:
kayan_matches: proc(signature: u64, all: u64, none: u64) -> b32 = {
return (signature & all) == all && (signature & none) == 0;
}
Step 2: keeping it up to date
Tables are only ever added, always at the end of w.tables, and never removed. So a query that has checked the first checked tables only needs to look at the ones after that:
kayan_query_update: proc(w: *const kayan_world, q: *kayan_query) -> void = {
for (ti: u64 = q*.checked; ti < w*.tables*.length; ti += 1) {
if (kayan_matches(kayan_table_at(w, cast(ti, u32))*.signature, q*.all, q*.none)) {
_ = vec<u32>push(q*.tables, cast(ti, u32));
}
}
q*.checked = w*.tables*.length;
}
The first update checks every table. After that, a frame with no new tables costs one comparison. A new table is checked once, by each query, the next time each is updated. flecs does the same thing in reverse: when a table is made, it tells the queries.
Step 3: walking a query
This is the shape of every system:
kayan_query_update(w, movers&);
for (i: u64 = 0; i < movers.tables*.length; i += 1) {
t: *kayan_table in level = kayan_query_table(w, movers&, i);
ps: []position = kayan_slice<position>(t);
vs: []velocity = kayan_slice<velocity>(t);
for (row: u64 = 0; row < t*.count; row += 1) {
ps[row].at.x += vs[row].per_second.x * 0.5;
ps[row].at.y += vs[row].per_second.y * 0.5;
}
}
Two loops. The outer one is over tables, a few iterations. The inner one is over rows: two dense arrays, read and written in order. That's lesson 1's move_soa. The query's job is to hand you the arrays; what you do with them is the same data-oriented loop as always.
The drones and the bullet are in different tables, and the loop doesn't care. The turret isn't matched (no velocity), and the loop never sees it, not even to skip it.
The pointers fit the loop. t names level and can be kept. ps and vs are borrows, taken fresh for each table and gone at the end of its turn, which is exactly as long as a slice into a column should live.
Run it
movers: all {position velocity}, none {}; tables: 2, entities: 4
table 1 {position velocity health}: 1:1 2:1 3:1
table 2 {position velocity damage lifetime}: 4:1
targets: all {position health}, none {burning}; tables: 1, entities: 3
table 1 {position velocity health}: 1:1 2:1 3:1
on_fire: all {health burning}, none {}; tables: 0, entities: 0
Then drone 2:1 catches fire, which makes table 3, after the queries were made:
2:1 catches fire, and the world makes table 3 for it
the queries have checked 3 tables; updating them checks the new one only
movers: all {position velocity}, none {}; tables: 3, entities: 4
table 1 {position velocity health}: 1:1 3:1
table 2 {position velocity damage lifetime}: 4:1
table 3 {position velocity health burning}: 2:1
targets: all {position health}, none {burning}; tables: 1, entities: 2
table 1 {position velocity health}: 1:1 3:1
on_fire: all {health burning}, none {}; tables: 1, entities: 1
table 3 {position velocity health burning}: 2:1
movers picked up the new table, targets refused it (none {burning}), and on_fire got its first. Nobody re-scanned the old tables.
Last, the bullet is despawned. Its table is now empty, and stays in movers. Walking an empty table costs one look at count.
Why this is fast
A query's cost doesn't depend on how many entities there are. It depends on how many tables match, and that number is small and stable. 10,000 drones in one table is one table to visit. That's the payoff of grouping entities by signature back in 2.4: the "which entities have X and Y?" question gets answered per table, once, instead of per entity, every frame.
The flip side is archetype fragmentation (2.4's Try): if every entity had a unique signature, every query would match thousands of one-row tables.
Try
- Write a query for "everything with a position that isn't a turret" and print it.
- Make 100 drones and walk
movers. How many times does the outer loop run? - Add
any: u64to the query: at least one of these components. Where inkayan_matchesdoes it go? - After the walk, keep a
[]positionfromkayan_slice, callspawn_drone, and write through the slice. Read the error. Then write the local's type as[]position in leveland compile again. What did you just promise, and what did you give up? - Write
kayan_table_at(w, 1)*.count = 100000;and then take akayan_sliceof that table.rin checkhas no objection, sincecountis only a field. Run it. What stops the program, and what did it measurecountagainst?
1// 3.1 Queries2//3// Part 2 finished the storage. Everything from here is behaviour, and in an4// ECS behaviour is a system: code that runs over every entity with some set of5// components. "Every entity with a position and a velocity" is spread over6// every table whose signature has both bits, so a system's first job is7// finding those tables. That is a query:8//9// all {position velocity}, none {} -> tables 1, 2, 310//11// A query matches tables, not entities. A game has a handful of tables and12// makes new ones rarely, so the matching is done once and kept; each frame a13// system walks the kept list, and inside each table streams the columns.14//15// The world lives in a region,16// `level`, and every pointer kept in memory says so. The one thing the checker17// cannot see through is the type-erased column: a run of bytes addressed by a18// size and an alignment, read back as whatever type the signature says. That19// core is six small `unsafe { }` blocks, each with an honest signature; every20// other proc in the file is checked. A table pointer no longer goes stale when21// a table is made, because tables are allocated one by one and the world holds22// pointers to them; a pointer into a column still does, and 3.3 is about that.23 24import "std/print.rin"25import "std/vec.rin"26import "std/hashmap.rin"27import "std/slice.rin"28 29level: region;30 31vec2: struct = {32 x: f32;33 y: f32;34}35 36position: struct = { at: vec2; }37velocity: struct = { per_second: vec2; }38health: struct = { points: f32; max: f32; }39damage: struct = { amount: f32; }40lifetime: struct = { seconds: f32; }41// A field tagged "hidden" is bookkeeping the inspector does not show.42turret: struct = { cooldown: f32; timer: f32 @ "hidden"; }43burning: struct = { per_second: f32; }44 45component: enum = {46 position,47 velocity,48 health,49 damage,50 lifetime,51 turret,52 burning,53}54 55component_of: proc<position>() -> component = { return component.position; }56component_of: proc<velocity>() -> component = { return component.velocity; }57component_of: proc<health>() -> component = { return component.health; }58component_of: proc<damage>() -> component = { return component.damage; }59component_of: proc<lifetime>() -> component = { return component.lifetime; }60component_of: proc<turret>() -> component = { return component.turret; }61component_of: proc<burning>() -> component = { return component.burning; }62 63// ---------------------------------------------------------------------------64// Entities (Part 1)65// ---------------------------------------------------------------------------66 67kayan_entity: struct = {68 index: u32;69 generation: u32;70}71 72// `missing`: the entity is alive but has no such component.73kayan_status: enum[must_use] = {74 ok,75 dead,76 missing,77}78 79// Where an entity lives: which table, and which row of it.80kayan_slot: struct = {81 generation: u32;82 table: u32;83 row: u32;84}85 86// ---------------------------------------------------------------------------87// The byte core: columns (2.2), and the region they grow into88// ---------------------------------------------------------------------------89 90// The region's arena, kept where the trusted procs can reach it: the world91// remembers where it was made, as a vec does, so nothing above this layer92// hands an arena around.93kayan_heap: struct = {94 arena: ^memops_arena;95}96 97kayan_heap_make: proc(r: region R) -> *kayan_heap in R = {98 unsafe {99 h: *kayan_heap in R = cast(memops_arena_push_zero(r, sizeof(kayan_heap), alignof(kayan_heap)), *kayan_heap in R);100 h*.arena = r;101 return h;102 }103}104 105// One component's values for every row of a table, as bytes: `size` of them106// a row, `capacity` rows allocated (the table knows how many are in use).107kayan_column: struct = {108 data: []u8 in level;109 size: u64;110 align: u64;111}112 113kayan_column_make: proc(h: *kayan_heap in level, info: *const reflect in permanent) -> *kayan_column in level = {114 unsafe {115 c: *kayan_column in level = cast(memops_arena_push_zero(h*.arena[0]&, sizeof(kayan_column), alignof(kayan_column)), *kayan_column in level);116 c*.size = info*.size;117 c*.align = info*.align;118 return c;119 }120}121 122// Room for `capacity` rows, the first `count` copied over. The old storage is123// left in the region: a slice taken before the grow still points at valid124// memory, and stale values.125kayan_column_grow: proc(h: *kayan_heap, c: *kayan_column, count: u64, capacity: u64) -> void = {126 unsafe {127 raw: ^u8 = cast(memops_arena_push_zero(h*.arena[0]&, c*.size * capacity, c*.align), ^u8);128 bytes: []u8 in level = cast(raw[0..c*.size * capacity], []u8 in level);129 // Slicing checks both ends: a count past either block stops the program.130 kept: u64 = c*.size * count;131 slice<u8>copy(bytes[0..kept], c*.data[0..kept]);132 c*.data = bytes;133 }134}135 136// One row's bytes, bounds-checked like any slice.137kayan_column_bytes: proc(c: *kayan_column, row: u64) -> []u8 in level = {138 return c*.data[row * c*.size..(row + 1) * c*.size];139}140 141// The typed view: the column's first `count` rows as `[]T`. This is the one142// place bytes become a type, so it takes nothing on trust. T must be the143// column's size, and plain data: any bytes are then a valid T, so reading the144// wrong type gives wrong numbers, never a bad pointer. The rows must fit, which145// the slice knows and no field can lie about, and the bytes must be aligned for146// T. The view is `[]T in level`: the bytes are in the region, so the view is147// good memory for as long as the region is. After the column grows it is the148// old block, still readable, no longer the column: take the view again.149kayan_column_slice: proc<T>(c: *const kayan_column, count: u64) -> []T in level = {150 if (c*.size != sizeof(T)) {151 printfmt("kayan: a column of {}-byte rows read as a {}-byte type\n", c*.size, sizeof(T));152 exit(1);153 }154 if (!reflect_is_plain(T<>&)) {155 printfmt("kayan: a column holds plain data, and this {}-byte type is not\n", sizeof(T));156 exit(1);157 }158 if (count > c*.data.length / sizeof(T)) {159 printfmt("kayan: a view of {} rows on a column with room for {}\n", count, c*.data.length / sizeof(T));160 exit(1);161 }162 unsafe {163 if (cast(c*.data.data, usize) % alignof(T) != 0) {164 printfmt("kayan: a column's bytes are not aligned to {}\n", alignof(T));165 exit(1);166 }167 raw: ^T = cast(c*.data.data, ^T);168 return cast(raw[0..count], []T in level);169 }170}171 172// Copying and zeroing a row are loops over the row's bytes: checked code.173kayan_column_copy_row: proc(c: *kayan_column, to: u64, from: u64) -> void = {174 dst: []u8 = kayan_column_bytes(c, to);175 src: []u8 = kayan_column_bytes(c, from);176 for (i: u64 = 0; i < dst.length; i += 1) {177 dst[i] = src[i];178 }179}180 181kayan_column_zero_row: proc(c: *kayan_column, row: u64) -> void = {182 bytes: []u8 = kayan_column_bytes(c, row);183 for (i: u64 = 0; i < bytes.length; i += 1) {184 bytes[i] = 0;185 }186}187 188// ---------------------------------------------------------------------------189// The table190// ---------------------------------------------------------------------------191 192// `columns[id]` is a column when the signature has bit `id`, and null193// otherwise. Count and capacity are the table's, shared by its columns, so194// they can never disagree; `entities` grows on its own and always has `count`195// rows.196//197// The edges: `add_edges[id]` is the table this one's entities move to when198// they gain component `id`, and `remove_edges[id]` when they lose it. Each is199// the table's index plus one, so 0, which a new table starts with, means "not200// known yet".201kayan_table: struct = {202 signature: u64;203 count: u64;204 capacity: u64;205 entities: *vec<kayan_entity> in level;206 columns: [64]option<*kayan_column in level>;207 add_edges: [64]u32;208 remove_edges: [64]u32;209}210 211// A zeroed table in the region, with an empty entities vec: the world fills212// in the signature and the columns.213kayan_table_new: proc(h: *kayan_heap in level) -> *kayan_table in level = {214 unsafe {215 t: *kayan_table in level = cast(memops_arena_push_zero(h*.arena[0]&, sizeof(kayan_table), alignof(kayan_table)), *kayan_table in level);216 v: *vec<kayan_entity> in level = cast(memops_arena_push_zero(h*.arena[0]&, sizeof(vec<kayan_entity>), alignof(vec<kayan_entity>)), *vec<kayan_entity> in level);217 v*.arena = h*.arena;218 t*.entities = v;219 return t;220 }221}222 223// The world: the registry, the slots and free list, and every table, with a224// map from signature to table index so finding one is a lookup, not a search.225// Tables are referred to by index, so a table can be named in a slot; each226// table is its own allocation, so a pointer to one stays good when the next227// is made.228kayan_world: struct = {229 heap: *kayan_heap in level;230 infos: [64]option<*const reflect in permanent>;231 component_count: u32;232 tables: *vec<*kayan_table in level> in level;233 by_signature: *hashmap<u64, u32> in level;234 slots: *vec<kayan_slot> in level;235 free: *vec<u32> in level;236 live: u64;237}238 239kayan_bit: proc(id: u32) -> u64 = {240 return 1'u64 shl cast(id, u64);241}242 243kayan_id_of: proc<T>() -> u32 = {244 return cast(component_of<T>(), u32);245}246 247kayan_mask_of: proc<T>() -> u64 = {248 return kayan_bit(kayan_id_of<T>());249}250 251// The reflect records live for the whole program, so `permanent` is their252// region and the world may keep them.253kayan_register: proc(w: *kayan_world, id: u32, info: *const reflect in permanent) -> void = {254 if (id >= 64) {255 printfmt("kayan: component id {} is past the 64 a signature can hold\n", id);256 exit(1);257 }258 w*.infos[id] = info;259 if (id + 1 > w*.component_count) {260 w*.component_count = id + 1;261 }262}263 264// A registered component's record. The checker needs the null test; the265// return after exit is never reached.266kayan_info: proc(w: *const kayan_world, id: u32) -> *const reflect in permanent = {267 info: option<*const reflect in permanent> = w*.infos[id];268 if (info) {269 return info;270 }271 printfmt("kayan: component {} was never registered\n", id);272 exit(1);273 return kayan_entity<>&;274}275 276kayan_table_has: proc(t: *const kayan_table, id: u32) -> b32 = {277 return (t*.signature & kayan_bit(id)) != 0;278}279 280// A table for a signature: one empty column per component in it, sized from281// the registry. Every component in the signature must be registered.282kayan_table_make: proc(w: *kayan_world, signature: u64) -> *kayan_table in level = {283 t: *kayan_table in level = kayan_table_new(w*.heap);284 t*.signature = signature;285 for (id: u32 = 0; id < 64; id += 1) {286 if ((signature & kayan_bit(id)) != 0) {287 if (id >= w*.component_count) {288 printfmt("kayan: signature {} uses component {}, which was never registered\n", signature, id);289 exit(1);290 }291 t*.columns[id] = kayan_column_make(w*.heap, kayan_info(w, id));292 }293 }294 return t;295}296 297// Double the room in every column at once.298kayan_table_grow: proc(w: *kayan_world, t: *kayan_table) -> void = {299 capacity: u64 = t*.capacity == 0 ? 4 : t*.capacity * 2;300 for (id: u32 = 0; id < w*.component_count; id += 1) {301 c: option<*kayan_column in level> = t*.columns[id];302 if (c) {303 kayan_column_grow(w*.heap, c, t*.count, capacity);304 }305 }306 t*.capacity = capacity;307}308 309// A new row at the end, owned by `e`, with every component zeroed.310kayan_table_add_row: proc(w: *kayan_world, t: *kayan_table, e: kayan_entity) -> u64 = {311 if (t*.count == t*.capacity) {312 kayan_table_grow(w, t);313 }314 row: u64 = t*.count;315 t*.count += 1;316 _ = vec<kayan_entity>push(t*.entities, e);317 for (id: u32 = 0; id < w*.component_count; id += 1) {318 c: option<*kayan_column in level> = t*.columns[id];319 if (c) {320 kayan_column_zero_row(c, row);321 }322 }323 return row;324}325 326// Who owns each of the table's rows. It only reads, and the checker sees that,327// so a system holds this and two or three columns of one table at once.328kayan_entities: proc(t: *const kayan_table) -> []kayan_entity in level = {329 return vec<kayan_entity>items(t*.entities);330}331 332// 1.4's swap-remove, for every column: the last row is copied over `row`, and333// the slot of the entity that owned the last row is told where it went.334kayan_table_remove_row: proc(w: *kayan_world, t: *kayan_table, row: u64) -> void = {335 last: u64 = t*.count - 1;336 if (row != last) {337 for (id: u32 = 0; id < w*.component_count; id += 1) {338 c: option<*kayan_column in level> = t*.columns[id];339 if (c) {340 kayan_column_copy_row(c, row, last);341 }342 }343 moved: kayan_entity = vec<kayan_entity>at(t*.entities, last)*;344 vec<kayan_entity>at(t*.entities, row)* = moved;345 vec<kayan_slot>at(w*.slots, cast(moved.index, u64))*.row = cast(row, u32);346 }347 _ = vec<kayan_entity>pop(t*.entities);348 t*.count = last;349}350 351// The typed view of one of the table's columns: T's rows, all of them in use.352// Asking for a component the table does not have is a bug, so it stops. The353// slice is good memory for as long as the region, but when the table grows it is354// the old block, no longer the column: take it again, and 3.3 makes that a rule.355kayan_slice: proc<T>(t: *const kayan_table) -> []T in level = {356 id: u32 = kayan_id_of<T>();357 c: option<*kayan_column in level> = t*.columns[id];358 if (!c) {359 printfmt("kayan: the table has no column for component {}\n", id);360 exit(1);361 }362 return kayan_column_slice<T>(c, t*.count);363}364 365kayan_table_at: proc(w: *const kayan_world, index: u32) -> *kayan_table in level = {366 return vec<*kayan_table in level>at(w*.tables, cast(index, u64))*;367}368 369// The table for a signature: found if it exists, made if it does not. This is370// where archetypes come from.371kayan_table_for: proc(w: *kayan_world, signature: u64) -> u32 = {372 found: option<*u32> = hashmap<u64, u32>get(w*.by_signature, signature);373 if (found) {374 return found*;375 }376 index: u32 = cast(w*.tables*.length, u32);377 _ = vec<*kayan_table in level>push(w*.tables, kayan_table_make(w, signature));378 _ = hashmap<u64, u32>set(w*.by_signature, signature, index);379 return index;380}381 382// ---------------------------------------------------------------------------383// The world384// ---------------------------------------------------------------------------385 386kayan_world_make: proc(r: region level) -> *kayan_world in level = {387 return new<kayan_world>(r, {388 .heap = kayan_heap_make(r),389 .tables = vec<*kayan_table in level>make(r, 8),390 .by_signature = hashmap<u64, u32>make(r),391 .slots = vec<kayan_slot>make(r, 64),392 .free = vec<u32>make(r, 16),393 });394}395 396kayan_alive: proc(w: *const kayan_world, e: kayan_entity) -> b32 = {397 if (cast(e.index, u64) >= w*.slots*.length) {398 return false;399 }400 return vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.generation == e.generation;401}402 403// A new entity with the components in `signature`, all zeroed, in the table404// for that signature.405kayan_spawn: proc(w: *kayan_world, signature: u64) -> kayan_entity = {406 e: kayan_entity = {};407 if (w*.free*.length > 0) {408 e.index = vec<u32>pop(w*.free);409 } else {410 e.index = cast(w*.slots*.length, u32);411 _ = vec<kayan_slot>push(w*.slots, { .generation = 1 });412 }413 e.generation = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.generation;414 table_index: u32 = kayan_table_for(w, signature);415 t: *kayan_table in level = kayan_table_at(w, table_index);416 row: u64 = kayan_table_add_row(w, t, e);417 // The slot is taken after the calls that change the world: a pointer into418 // `slots` from before them may be stale, and the checker says so.419 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));420 slot*.table = table_index;421 slot*.row = cast(row, u32);422 w*.live += 1;423 return e;424}425 426kayan_despawn: proc(w: *kayan_world, e: kayan_entity) -> kayan_status = {427 if (!kayan_alive(w, e)) {428 return kayan_status.dead;429 }430 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));431 table: u32 = slot*.table;432 row: u32 = slot*.row;433 kayan_table_remove_row(w, kayan_table_at(w, table), cast(row, u64));434 slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));435 slot*.generation += 1;436 _ = vec<u32>push(w*.free, e.index);437 w*.live -= 1;438 return kayan_status.ok;439}440 441// The components a live entity has: its table's signature. 0 for a dead one.442kayan_signature: proc(w: *const kayan_world, e: kayan_entity) -> u64 = {443 if (!kayan_alive(w, e)) {444 return 0;445 }446 return kayan_table_at(w, vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.table)*.signature;447}448 449// ---------------------------------------------------------------------------450// Moving between tables451// ---------------------------------------------------------------------------452 453// The table an entity of table `from` moves to when it gains component `id`.454// The first time, it is found by signature, which can make the table; the455// answer is written on both tables, as from's add edge and the destination's456// remove edge, so neither direction is looked up again.457kayan_table_with: proc(w: *kayan_world, from: u32, id: u32) -> u32 = {458 edge: u32 = kayan_table_at(w, from)*.add_edges[id];459 if (edge != 0) {460 return edge - 1;461 }462 to: u32 = kayan_table_for(w, kayan_table_at(w, from)*.signature | kayan_bit(id));463 kayan_table_at(w, from)*.add_edges[id] = to + 1;464 kayan_table_at(w, to)*.remove_edges[id] = from + 1;465 return to;466}467 468// The same for losing component `id`.469kayan_table_without: proc(w: *kayan_world, from: u32, id: u32) -> u32 = {470 edge: u32 = kayan_table_at(w, from)*.remove_edges[id];471 if (edge != 0) {472 return edge - 1;473 }474 to: u32 = kayan_table_for(w, kayan_table_at(w, from)*.signature & ~kayan_bit(id));475 kayan_table_at(w, from)*.remove_edges[id] = to + 1;476 kayan_table_at(w, to)*.add_edges[id] = from + 1;477 return to;478}479 480// Move a live entity to table `to_index`. Components both tables have are481// copied, byte for byte; components only the new table has start zeroed;482// components only the old table has are dropped with its old row.483kayan_move: proc(w: *kayan_world, e: kayan_entity, to_index: u32) -> void = {484 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));485 from_index: u32 = slot*.table;486 if (from_index == to_index) {487 return;488 }489 from_row: u64 = cast(slot*.row, u64);490 from: *kayan_table in level = kayan_table_at(w, from_index);491 to: *kayan_table in level = kayan_table_at(w, to_index);492 to_row: u64 = kayan_table_add_row(w, to, e);493 for (id: u32 = 0; id < w*.component_count; id += 1) {494 fc: option<*kayan_column in level> = from*.columns[id];495 tc: option<*kayan_column in level> = to*.columns[id];496 if (fc && tc) {497 dst: []u8 = kayan_column_bytes(tc, to_row);498 src: []u8 = kayan_column_bytes(fc, from_row);499 for (i: u64 = 0; i < dst.length; i += 1) {500 dst[i] = src[i];501 }502 }503 }504 kayan_table_remove_row(w, from, from_row);505 slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));506 slot*.table = to_index;507 slot*.row = cast(to_row, u32);508}509 510// ---------------------------------------------------------------------------511// The byte layer, for tools512// ---------------------------------------------------------------------------513 514// e's component `id` as its bytes: an empty slice if e is dead or has no such515// component. The typed layer is for game code, which knows its types; this is516// for tools that work from reflect records instead.517kayan_get_bytes: proc(w: *const kayan_world, e: kayan_entity, id: u32) -> []u8 in level = {518 none: []u8 in level = {};519 if (!kayan_alive(w, e) || id >= w*.component_count) {520 return none;521 }522 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));523 table: u32 = slot*.table;524 row: u32 = slot*.row;525 t: *kayan_table in level = kayan_table_at(w, table);526 c: option<*kayan_column in level> = t*.columns[id];527 if (!c) {528 return none;529 }530 return kayan_column_bytes(c, cast(row, u64));531}532 533// ---------------------------------------------------------------------------534// The typed layer535// ---------------------------------------------------------------------------536 537// e's T, or null if e is dead or has no T. Handle, slot, table, row, typed538// view: the same path as 1.4's get_position, for any component. `option<*T>` says the539// null is possible, so the caller tests before it reads. The pointer is good540// until the next structural change, so use it now and keep the handle.541kayan_get: proc<T>(w: *const kayan_world, e: kayan_entity) -> option<*T in level> = {542 if (!kayan_alive(w, e)) {543 return null;544 }545 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));546 table: u32 = slot*.table;547 row: u32 = slot*.row;548 t: *kayan_table in level = kayan_table_at(w, table);549 if (!kayan_table_has(t, kayan_id_of<T>())) {550 return null;551 }552 rows: []T in level = kayan_slice<T>(t);553 return rows[cast(row, u64)]&;554}555 556kayan_has: proc<T>(w: *const kayan_world, e: kayan_entity) -> b32 = {557 return (kayan_signature(w, e) & kayan_mask_of<T>()) != 0;558}559 560// Write e's T. It must already have one: giving an entity a component it does561// not have changes its table, which is 2.6.562kayan_set: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> kayan_status = {563 if (!kayan_alive(w, e)) {564 return kayan_status.dead;565 }566 p: option<*T> = kayan_get<T>(w, e);567 if (!p) {568 return kayan_status.missing;569 }570 p* = value;571 return kayan_status.ok;572}573 574// Give e a T. If it already has one, this is kayan_set; if not, it moves to575// the table with T and then sets it.576kayan_add: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> kayan_status = {577 if (!kayan_alive(w, e)) {578 return kayan_status.dead;579 }580 if (!kayan_has<T>(w, e)) {581 from: u32 = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.table;582 kayan_move(w, e, kayan_table_with(w, from, kayan_id_of<T>()));583 }584 return kayan_set<T>(w, e, value);585}586 587// Take e's T away, moving it to the table without T.588kayan_remove: proc<T>(w: *kayan_world, e: kayan_entity) -> kayan_status = {589 if (!kayan_alive(w, e)) {590 return kayan_status.dead;591 }592 if (!kayan_has<T>(w, e)) {593 return kayan_status.missing;594 }595 from: u32 = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.table;596 kayan_move(w, e, kayan_table_without(w, from, kayan_id_of<T>()));597 return kayan_status.ok;598}599 600// The invariant, over every table: each row belongs to a live entity whose601// slot points back at that table and row, and the rows of all tables add up to602// the live count.603kayan_verify: proc(w: *const kayan_world) -> u64 = {604 problems: u64 = 0;605 rows: u64 = 0;606 for (ti: u64 = 0; ti < w*.tables*.length; ti += 1) {607 t: *kayan_table in level = kayan_table_at(w, cast(ti, u32));608 for (row: u64 = 0; row < t*.count; row += 1) {609 // Taken row by row: kayan_alive takes the world, and the checker610 // does not know that it leaves the rows where they are.611 es: []kayan_entity = kayan_entities(t);612 e: kayan_entity = es[row];613 if (!kayan_alive(w, e)) {614 printfmt("kayan_verify: table {} row {} holds dead entity {}:{}\n", ti, row, e.index, e.generation);615 problems += 1;616 continue;617 }618 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));619 if (cast(slot*.table, u64) != ti || cast(slot*.row, u64) != row) {620 printfmt("kayan_verify: {}:{} is in table {} row {}, its slot says table {} row {}\n",621 e.index, e.generation, ti, row, slot*.table, slot*.row);622 problems += 1;623 }624 }625 rows += t*.count;626 }627 if (rows != w*.live) {628 printfmt("kayan_verify: {} rows in tables, {} live entities\n", rows, w*.live);629 problems += 1;630 }631 return problems;632}633 634kayan_print_signature: proc(w: *const kayan_world, signature: u64) -> void = {635 printfmt("{");636 first: b32 = true;637 for (id: u32 = 0; id < w*.component_count; id += 1) {638 if ((signature & kayan_bit(id)) != 0) {639 if (!first) {640 printfmt(" ");641 }642 printfmt("{}", kayan_info(w, id)*.name);643 first = false;644 }645 }646 printfmt("}");647}648 649kayan_print_table: proc(w: *const kayan_world, ti: u32) -> void = {650 t: *kayan_table in level = kayan_table_at(w, ti);651 printfmt(" table {} ", ti);652 kayan_print_signature(w, t*.signature);653 printfmt(":");654 es: []kayan_entity = kayan_entities(t);655 for (row: u64 = 0; row < es.length; row += 1) {656 e: kayan_entity = es[row];657 printfmt(" {}:{}", e.index, e.generation);658 }659 printfmt("\n");660}661 662kayan_print_tables: proc(w: *const kayan_world) -> void = {663 for (ti: u32 = 0; ti < cast(w*.tables*.length, u32); ti += 1) {664 kayan_print_table(w, ti);665 }666}667 668// ---------------------------------------------------------------------------669// The inspector: reflect records, no component types670// ---------------------------------------------------------------------------671 672kayan_indent: proc(depth: u32) -> void = {673 for (i: u32 = 0; i < depth; i += 1) {674 printfmt(" ");675 }676}677 678// One value of a builtin type, read from its bytes by its type's name. The679// bytes may sit at any address, so the read is a copy into a local: this is the680// inspector's one unsafe block.681kayan_print_builtin: proc(type_name: ^const char, at: []const u8) -> void = {682 if (at.length < 4) {683 printfmt("<{} bytes>", at.length);684 return;685 }686 unsafe {687 if (reflect_cstr_equal(type_name, "f32")) {688 f: f32 = 0.0;689 memcpy(f&, at.data, 4);690 printfmt("{:.2}", cast(f, f64));691 } else if (reflect_cstr_equal(type_name, "u32")) {692 u: u32 = 0;693 memcpy(u&, at.data, 4);694 printfmt("{}", u);695 } else if (reflect_cstr_equal(type_name, "i32")) {696 i: i32 = 0;697 memcpy(i&, at.data, 4);698 printfmt("{}", i);699 } else {700 printfmt("<{}>", type_name);701 }702 }703}704 705// Every field of a struct whose bytes are `base`, nested structs indented706// under their field. A field's bytes are cut from `base` by its offset and707// size, both from the reflect record.708kayan_print_fields: proc(type: *const reflect, base: []const u8, depth: u32) -> void = {709 for (i: u64 = 0; i < type*.count; i += 1) {710 f: option<*const reflect_field> = reflect_field_at(type, i);711 if (!f) {712 continue;713 }714 if (reflect_field_has_attr(f, "hidden")) {715 continue;716 }717 at: []const u8 = base[f*.offset..f*.offset + f*.size];718 kayan_indent(depth);719 printfmt("{}:", f*.name);720 info: option<*const reflect> = reflect_field_info(f);721 if (info && info*.kind == reflect_kind_struct) {722 printfmt("\n");723 kayan_print_fields(info, at, depth + 1);724 } else {725 printfmt(" ");726 kayan_print_builtin(f*.type, at);727 printfmt("\n");728 }729 }730}731 732kayan_inspect: proc(w: *const kayan_world, e: kayan_entity) -> void = {733 if (!kayan_alive(w, e)) {734 printfmt("{}:{} (dead)\n", e.index, e.generation);735 return;736 }737 printfmt("{}:{} ", e.index, e.generation);738 signature: u64 = kayan_signature(w, e);739 kayan_print_signature(w, signature);740 printfmt("\n");741 for (id: u32 = 0; id < w*.component_count; id += 1) {742 if ((signature & kayan_bit(id)) != 0) {743 info: *const reflect in permanent = kayan_info(w, id);744 printfmt(" {}\n", info*.name);745 bytes: []u8 = kayan_get_bytes(w, e, id);746 kayan_print_fields(info, bytes, 2);747 }748 }749}750 751// Every table: rows in use out of room, and per column its row size and its752// bytes in use out of bytes allocated. The memory a table holds and does not753// use is the cost of doubling on growth.754kayan_print_layout: proc(w: *const kayan_world) -> void = {755 for (ti: u32 = 0; ti < cast(w*.tables*.length, u32); ti += 1) {756 t: *kayan_table in level = kayan_table_at(w, ti);757 printfmt("table {} ", ti);758 kayan_print_signature(w, t*.signature);759 printfmt(": {} rows, room for {}\n", cast(t*.count, u32), cast(t*.capacity, u32));760 for (id: u32 = 0; id < w*.component_count; id += 1) {761 c: option<*kayan_column in level> = t*.columns[id];762 if (c) {763 size: u64 = c*.size;764 printfmt(" {:-9} {:2} bytes a row, {:4} of {:4} bytes in use\n", kayan_info(w, id)*.name, cast(size, u32),765 cast(size * t*.count, u32), cast(size * t*.capacity, u32));766 }767 }768 }769}770 771// ---------------------------------------------------------------------------772// Queries773// ---------------------------------------------------------------------------774 775// The tables a system runs over: every table whose signature has all of `all`776// and none of `none`, kept by index. Tables are only ever added, at the end of777// w.tables, so the query remembers how many it has checked and, brought up to778// date, looks at the new ones only.779kayan_query: struct = {780 all: u64;781 none: u64;782 tables: *vec<u32> in level;783 checked: u64;784}785 786// The list of matched tables lives in the world's region, so a query says787// which region it is made in.788kayan_query_init: proc(r: region level, all: u64, none: u64) -> kayan_query = {789 return { .all = all, .none = none, .tables = vec<u32>make(r, 4) };790}791 792kayan_matches: proc(signature: u64, all: u64, none: u64) -> b32 = {793 return (signature & all) == all && (signature & none) == 0;794}795 796// Check the tables made since the query last looked. With none new, this is797// one comparison, so call it before every walk.798kayan_query_update: proc(w: *const kayan_world, q: *kayan_query) -> void = {799 for (ti: u64 = q*.checked; ti < w*.tables*.length; ti += 1) {800 if (kayan_matches(kayan_table_at(w, cast(ti, u32))*.signature, q*.all, q*.none)) {801 _ = vec<u32>push(q*.tables, cast(ti, u32));802 }803 }804 q*.checked = w*.tables*.length;805}806 807// The query's i-th table.808kayan_query_table: proc(w: *const kayan_world, q: *const kayan_query, i: u64) -> *kayan_table in level = {809 return kayan_table_at(w, vec<u32>at(q*.tables, i)*);810}811 812// How many entities the query matches: the rows of its tables.813kayan_query_count: proc(w: *const kayan_world, q: *kayan_query) -> u64 = {814 kayan_query_update(w, q);815 n: u64 = 0;816 for (i: u64 = 0; i < q*.tables*.length; i += 1) {817 n += kayan_query_table(w, q, i)*.count;818 }819 return n;820}821 822kayan_print_query: proc(w: *const kayan_world, name: ^const char, q: *kayan_query) -> void = {823 kayan_query_update(w, q);824 printfmt("{}: all ", name);825 kayan_print_signature(w, q*.all);826 printfmt(", none ");827 kayan_print_signature(w, q*.none);828 printfmt("; tables: {}, entities: {}\n", q*.tables*.length, kayan_query_count(w, q));829 for (i: u64 = 0; i < q*.tables*.length; i += 1) {830 kayan_print_table(w, vec<u32>at(q*.tables, i)*);831 }832}833 834// ---------------------------------------------------------------------------835// The program836// ---------------------------------------------------------------------------837 838spawn_turret: proc(w: *kayan_world, x: f32, y: f32) -> kayan_entity = {839 e: kayan_entity = kayan_spawn(w, kayan_mask_of<position>() | kayan_mask_of<turret>());840 _ = kayan_set<position>(w, e, { .at = { .x = x, .y = y } });841 _ = kayan_set<turret>(w, e, { .cooldown = 0.5, .timer = 0.0 });842 return e;843}844 845spawn_drone: proc(w: *kayan_world, x: f32, y: f32) -> kayan_entity = {846 e: kayan_entity = kayan_spawn(w, kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<health>());847 _ = kayan_set<position>(w, e, { .at = { .x = x, .y = y } });848 _ = kayan_set<velocity>(w, e, { .per_second = { .x = -1.0, .y = 0.0 } });849 _ = kayan_set<health>(w, e, { .points = 100.0, .max = 100.0 });850 return e;851}852 853spawn_bullet: proc(w: *kayan_world, x: f32, y: f32) -> kayan_entity = {854 e: kayan_entity = kayan_spawn(w, kayan_mask_of<position>() | kayan_mask_of<velocity>()855 | kayan_mask_of<damage>() | kayan_mask_of<lifetime>());856 _ = kayan_set<position>(w, e, { .at = { .x = x, .y = y } });857 _ = kayan_set<velocity>(w, e, { .per_second = { .x = 8.0, .y = 0.0 } });858 _ = kayan_set<damage>(w, e, { .amount = 25.0 });859 _ = kayan_set<lifetime>(w, e, { .seconds = 2.0 });860 return e;861}862 863// Every entity a query matches, table by table, with its position.864print_positions: proc(w: *const kayan_world, q: *kayan_query) -> void = {865 kayan_query_update(w, q);866 for (i: u64 = 0; i < q*.tables*.length; i += 1) {867 t: *kayan_table in level = kayan_query_table(w, q, i);868 ps: []position = kayan_slice<position>(t);869 es: []kayan_entity = kayan_entities(t);870 for (row: u64 = 0; row < t*.count; row += 1) {871 e: kayan_entity = es[row];872 p: position = ps[row];873 printfmt(" {}:{} at ({:5.2}, {:5.2})\n", e.index, e.generation, cast(p.at.x, f64), cast(p.at.y, f64));874 }875 }876}877 878main: proc() -> i32 = {879 level: region = {880 w: *kayan_world in level = kayan_world_make(level);881 kayan_register(w, kayan_id_of<position>(), position<>&);882 kayan_register(w, kayan_id_of<velocity>(), velocity<>&);883 kayan_register(w, kayan_id_of<health>(), health<>&);884 kayan_register(w, kayan_id_of<damage>(), damage<>&);885 kayan_register(w, kayan_id_of<lifetime>(), lifetime<>&);886 kayan_register(w, kayan_id_of<turret>(), turret<>&);887 kayan_register(w, kayan_id_of<burning>(), burning<>&);888 889 _ = spawn_turret(w, 0.0, 0.0);890 _ = spawn_drone(w, 20.0, 5.0);891 b: kayan_entity = spawn_drone(w, 30.0, -5.0);892 _ = spawn_drone(w, 40.0, 0.0);893 shot: kayan_entity = spawn_bullet(w, 1.0, 0.0);894 printfmt("{} entities in {} tables:\n", w*.live, w*.tables*.length);895 kayan_print_tables(w);896 897 // Three queries. Making one costs nothing: it matches when first updated.898 movers: kayan_query = kayan_query_init(level, kayan_mask_of<position>() | kayan_mask_of<velocity>(), 0);899 targets: kayan_query = kayan_query_init(level, kayan_mask_of<position>() | kayan_mask_of<health>(), kayan_mask_of<burning>());900 on_fire: kayan_query = kayan_query_init(level, kayan_mask_of<health>() | kayan_mask_of<burning>(), 0);901 printfmt("\n");902 kayan_print_query(w, "movers", movers&);903 kayan_print_query(w, "targets", targets&);904 kayan_print_query(w, "on_fire", on_fire&);905 906 // A system's shape: bring the query up to date, then for each of its907 // tables take the columns as slices and loop over the rows.908 kayan_query_update(w, movers&);909 for (i: u64 = 0; i < movers.tables*.length; i += 1) {910 t: *kayan_table in level = kayan_query_table(w, movers&, i);911 ps: []position = kayan_slice<position>(t);912 vs: []velocity = kayan_slice<velocity>(t);913 for (row: u64 = 0; row < t*.count; row += 1) {914 ps[row].at.x += vs[row].per_second.x * 0.5;915 ps[row].at.y += vs[row].per_second.y * 0.5;916 }917 }918 printfmt("\nhalf a second of movement, over the movers' tables:\n");919 print_positions(w, movers&);920 921 // A table made after the queries were.922 _ = kayan_add<burning>(w, b, { .per_second = 10.0 });923 printfmt("\n{}:{} catches fire, and the world makes table {} for it\n", b.index, b.generation, w*.tables*.length - 1);924 printfmt("the queries have checked {} tables; updating them checks the new one only\n", movers.checked);925 kayan_print_query(w, "movers", movers&);926 kayan_print_query(w, "targets", targets&);927 kayan_print_query(w, "on_fire", on_fire&);928 929 // A table emptied after the queries matched it.930 _ = kayan_despawn(w, shot);931 printfmt("\nthe bullet is gone; its table is empty and stays matched:\n");932 kayan_print_query(w, "movers", movers&);933 printfmt("kayan_verify: {} problems\n", kayan_verify(w));934 }935 return 0;936}What queries prints. tools/check.py fails the chapter if a single byte differs.
5 entities in 3 tables:
table 0 {position turret}: 0:1
table 1 {position velocity health}: 1:1 2:1 3:1
table 2 {position velocity damage lifetime}: 4:1
movers: all {position velocity}, none {}; tables: 2, entities: 4
table 1 {position velocity health}: 1:1 2:1 3:1
table 2 {position velocity damage lifetime}: 4:1
targets: all {position health}, none {burning}; tables: 1, entities: 3
table 1 {position velocity health}: 1:1 2:1 3:1
on_fire: all {health burning}, none {}; tables: 0, entities: 0
half a second of movement, over the movers' tables:
1:1 at (19.50, 5.00)
2:1 at (29.50, -5.00)
3:1 at (39.50, 0.00)
4:1 at ( 5.00, 0.00)
2:1 catches fire, and the world makes table 3 for it
the queries have checked 3 tables; updating them checks the new one only
movers: all {position velocity}, none {}; tables: 3, entities: 4
table 1 {position velocity health}: 1:1 3:1
table 2 {position velocity damage lifetime}: 4:1
table 3 {position velocity health burning}: 2:1
targets: all {position health}, none {burning}; tables: 1, entities: 2
table 1 {position velocity health}: 1:1 3:1
on_fire: all {health burning}, none {}; tables: 1, entities: 1
table 3 {position velocity health burning}: 2:1
the bullet is gone; its table is empty and stays matched:
movers: all {position velocity}, none {}; tables: 3, entities: 3
table 1 {position velocity health}: 1:1 3:1
table 2 {position velocity damage lifetime}:
table 3 {position velocity health burning}: 2:1
kayan_verify: 0 problems
Build an ECS · part 3, systems · 3.2
Systems
ecs/src/3.systems/3.2.systems/systems.rin
In the dojo: 13. a dispatch table, on its own, to watch and to practise.
3.1's movement loop had the shape of every system: bring a query up to date, then for each of its tables take the columns as slices and loop over the rows. This chapter names that shape and splits it in two:
- the kernel: a plain loop over plain slices. It knows nothing about tables, entities or worlds. This is lessons 1 to 7:
noalias, streaming, vectorization. - the system: a query, plus a small proc kayan calls on each table the query matches. It takes the table's columns and calls the kernel.
A frame runs the systems in order, each over all of its tables. The order is the schedule, and it's written down once, as data.
Step 1: the kernels
integrate: proc[unsafe](ps: noalias []position, vs: []const velocity, dt: f32) -> void = {
for (i: u64 = 0; i < ps.length; i += 1) {
ps[i].at.x += vs[i].per_second.x * dt;
ps[i].at.y += vs[i].per_second.y * dt;
}
}
That's the whole thing, and it's the code you'd want to write if there were no ECS at all. Kernels are where you profile, read assembly and reach for SIMD (the lessons). Everything else in this course exists to hand them the right slices. burn and age look the same.
Why proc[unsafe], on a loop this innocent? Because of one word. noalias tells the C optimiser these slices don't overlap, and that's a promise the checker has no way to keep. It wants you to mark that plainly:
type error: noalias is a promise to the C optimiser that the checker cannot keep: it belongs on a 'proc[unsafe]'
The body is as ordinary as it looks, bounds checks included, and overlap is still tested on the way in unless you turn the checks off. All that [unsafe] means here is that the performance promise is yours to keep.
Step 2: what a system gets
kayan_iter: struct = {
world: *kayan_world in level;
table: *kayan_table in level;
dt: f32;
}
kayan_system_proc: alias = *proc(it: *kayan_iter) -> void;
One struct instead of three parameters. It's the same idea as flecs' ecs_iter_t, and it has two practical reasons. Adding to it later (3.3 adds two fields) changes no system's signature. And a system that uses only the table still uses its one parameter. rin compiles to C with -Wextra -Werror, and an unused parameter is an error there. Both pointers name their region, so the struct can be made anywhere level is open.
The system itself is the adapter:
move_system: proc(it: *kayan_iter) -> void = {
unsafe { integrate(kayan_slice<position>(it*.table), kayan_slice<velocity>(it*.table), it*.dt); }
}
Columns out, kernel in. No loop here: the kernel has it.
This is where you keep the kernel's promise, so it's where the unsafe { } goes. It's easy to keep: a position column and a velocity column are different allocations, so they can't overlap. It's the only line in the system, and it says what you're vouching for.
Step 3: a system is data
A system is a name, a proc, a query, and the counts of the work it's done. Only the first two never change, and that turns out to be the seam the checker needs, so a system comes in two halves.
The code:
kayan_system: struct = {
name: ^const char;
run: kayan_system_proc;
}
kayan_system_make: proc(r: region level, name: ^const char, run: kayan_system_proc) -> *kayan_system in level = {
unsafe {
s: *kayan_system in level = cast(memops_arena_push_zero(r, sizeof(kayan_system), alignof(kayan_system)), *kayan_system in level);
s*.name = name;
s*.run = run;
return s;
}
}
A C string and a proc pointer are both pointers without a region, so the checker can't place them. It won't let new make a struct holding them, and it says what it wants instead:
type error: a value made by new holds 'kayan_system' by value, and it carries a pointer whose region the checker cannot see; safe code keeps such a value through a pointer into a region, '*kayan_system in R', made by a proc of its own; if a field of it is stored from two regions, write its region on the field
So a small trusted proc makes it, once, and everyone else holds a pointer. Nothing is lost, because nothing here changes either field after that.
The data:
kayan_scheduled: struct = {
system: *kayan_system in level;
query: kayan_query;
tables: u64;
rows: u64;
}
Everything that changes as the frames go is here, as plain checked data. tables and rows count the work, for the output. The schedule is a list of these:
systems: *vec<*kayan_scheduled in level> in level = vec<*kayan_scheduled in level>make(level, 3);
kayan_schedule_add(level, systems, "burn", kayan_mask_of<health>() | kayan_mask_of<burning>(), 0, burn_system);
kayan_schedule_add(level, systems, "age", kayan_mask_of<lifetime>(), 0, age_system);
kayan_schedule_add(level, systems, "move", kayan_mask_of<position>() | kayan_mask_of<velocity>(), 0, move_system);
It's a vec of pointers for the same reason as w.tables in 2.4. Each entry is its own allocation, so a pointer to one is a pointer into level, which never goes stale.
Step 4: a frame
kayan_progress: proc(w: *kayan_world in level, systems: *const vec<*kayan_scheduled in level>, dt: f32) -> void = {
for (s: u64 = 0; s < systems*.length; s += 1) {
sys: *kayan_scheduled in level = kayan_scheduled_at(systems, s);
kayan_query_update(w, sys*.query&);
for (i: u64 = 0; i < sys*.query.tables*.length; i += 1) {
it: kayan_iter = { .world = w, .table = kayan_query_table(w, sys*.query&, i), .dt = dt };
if (it.table*.count > 0) {
sys*.tables += 1;
sys*.rows += it.table*.count;
sys*.system*.run(it&);
}
}
}
}
kayan_progress is flecs' name for the same thing. For each system in order, for each of its tables, one call. The game loop becomes:
for (frame: u32 = 1; frame <= 8; frame += 1) {
kayan_progress(w, systems, dt);
Order is data flow
Systems run one after another, never interleaved, and each sees everything the earlier ones wrote this frame. So the order is a statement about data: a system that reads what another writes runs after it. Move before collide means bullets hit where they are now, not where they were. Damage before death means a drone dies in the frame its health hits zero, not the next. 3.4's schedule has nine systems, and each position in it is one of these decisions.
Bigger engines go further. Each system declares what it reads and writes, and the scheduler uses that to run systems that don't conflict at the same time, on other threads. Underneath, the order is still a list like this one.
Run it
the schedule, run in this order every frame:
burn: all {health burning}, none {}; tables: 1, entities: 1
table 2 {position velocity health burning}: 2:1
age: all {lifetime}, none {}; tables: 1, entities: 3
table 3 {position velocity damage lifetime}: 4:1 5:1 6:1
move: all {position velocity}, none {}; tables: 3, entities: 6
table 1 {position velocity health}: 1:1 3:1
table 2 {position velocity health burning}: 2:1
table 3 {position velocity damage lifetime}: 4:1 5:1 6:1
...
after frame 8, 2.00 seconds:
0:1 at ( 0.00, 0.00)
1:1 at (18.00, 4.00) health 100
3:1 at (26.00, -4.00) health 100
2:1 at (22.00, 0.00) health 84, burning
4:1 at (17.00, 0.00) lifetime -1.50
5:1 at (17.00, 4.00) lifetime -1.00
6:1 at (17.00, -4.00) lifetime -0.50
8 frames, one call per table per frame:
burn called 8 times, over 8 rows
age called 8 times, over 24 rows
move called 24 times, over 48 rows
3 bullets are out of time and still in the world
move made 24 calls in 8 frames, three tables a frame, and each call is one tight loop. Call overhead is per table, never per entity.
And the last line is the cliffhanger. Each bullet's lifetime ran out, and nothing removed it, because a system in this chapter changes values, never structure. Despawning from inside a system looks like a one-line fix. It isn't, and 3.3 shows why.
Try
- Swap
burnandmovein the schedule. Does anything change? Now think of a pair where it would. - Add a
healsystem for drones that aren't burning (all {health}, none {burning}). - Build in Release and find
integratein the assembly. Is it vectorized? (Lesson 1 shows how to look.) - Take
noaliasand[unsafe]offage, and theunsafe { }off its system. Now the whole system is checked. What did the optimiser lose? For a kernel with one slice, was there anything to lose?
1// 3.2 Systems2//3// 3.1's movement loop had the shape of every system: bring a query up to4// date, then for each of its tables take the columns as slices and loop over5// the rows. This chapter names that shape and splits it in two:6//7// the kernel a plain loop over plain slices, like lesson 1's move_soa. It8// knows nothing about tables, entities or the world.9// the system a query, and a small proc kayan calls on each table the10// query matches. It takes the table's columns and calls the11// kernel.12//13// A frame runs the systems in order, each over all of its tables. The order is14// the schedule, and it is written down once, as data.15//16// A system splits once more, along a line the checker17// draws: its name and its proc are pointers the checker cannot place, so they18// live in a `kayan_system` made once by a trusted proc and never written19// again; everything that changes as the frames go, the query and the counts,20// lives beside it in a `kayan_scheduled`, which is plain checked data.21 22import "std/print.rin"23import "std/vec.rin"24import "std/hashmap.rin"25import "std/slice.rin"26 27level: region;28 29vec2: struct = {30 x: f32;31 y: f32;32}33 34position: struct = { at: vec2; }35velocity: struct = { per_second: vec2; }36health: struct = { points: f32; max: f32; }37damage: struct = { amount: f32; }38lifetime: struct = { seconds: f32; }39// A field tagged "hidden" is bookkeeping the inspector does not show.40turret: struct = { cooldown: f32; timer: f32 @ "hidden"; }41burning: struct = { per_second: f32; }42 43component: enum = {44 position,45 velocity,46 health,47 damage,48 lifetime,49 turret,50 burning,51}52 53component_of: proc<position>() -> component = { return component.position; }54component_of: proc<velocity>() -> component = { return component.velocity; }55component_of: proc<health>() -> component = { return component.health; }56component_of: proc<damage>() -> component = { return component.damage; }57component_of: proc<lifetime>() -> component = { return component.lifetime; }58component_of: proc<turret>() -> component = { return component.turret; }59component_of: proc<burning>() -> component = { return component.burning; }60 61// ---------------------------------------------------------------------------62// Entities (Part 1)63// ---------------------------------------------------------------------------64 65kayan_entity: struct = {66 index: u32;67 generation: u32;68}69 70// `missing`: the entity is alive but has no such component.71kayan_status: enum[must_use] = {72 ok,73 dead,74 missing,75}76 77// Where an entity lives: which table, and which row of it.78kayan_slot: struct = {79 generation: u32;80 table: u32;81 row: u32;82}83 84// ---------------------------------------------------------------------------85// The byte core: columns (2.2), and the region they grow into86// ---------------------------------------------------------------------------87 88// The region's arena, kept where the trusted procs can reach it: the world89// remembers where it was made, as a vec does, so nothing above this layer90// hands an arena around.91kayan_heap: struct = {92 arena: ^memops_arena;93}94 95kayan_heap_make: proc(r: region R) -> *kayan_heap in R = {96 unsafe {97 h: *kayan_heap in R = cast(memops_arena_push_zero(r, sizeof(kayan_heap), alignof(kayan_heap)), *kayan_heap in R);98 h*.arena = r;99 return h;100 }101}102 103// One component's values for every row of a table, as bytes: `size` of them104// a row, `capacity` rows allocated (the table knows how many are in use).105kayan_column: struct = {106 data: []u8 in level;107 size: u64;108 align: u64;109}110 111kayan_column_make: proc(h: *kayan_heap in level, info: *const reflect in permanent) -> *kayan_column in level = {112 unsafe {113 c: *kayan_column in level = cast(memops_arena_push_zero(h*.arena[0]&, sizeof(kayan_column), alignof(kayan_column)), *kayan_column in level);114 c*.size = info*.size;115 c*.align = info*.align;116 return c;117 }118}119 120// Room for `capacity` rows, the first `count` copied over. The old storage is121// left in the region: a slice taken before the grow still points at valid122// memory, and stale values.123kayan_column_grow: proc(h: *kayan_heap, c: *kayan_column, count: u64, capacity: u64) -> void = {124 unsafe {125 raw: ^u8 = cast(memops_arena_push_zero(h*.arena[0]&, c*.size * capacity, c*.align), ^u8);126 bytes: []u8 in level = cast(raw[0..c*.size * capacity], []u8 in level);127 // Slicing checks both ends: a count past either block stops the program.128 kept: u64 = c*.size * count;129 slice<u8>copy(bytes[0..kept], c*.data[0..kept]);130 c*.data = bytes;131 }132}133 134// One row's bytes, bounds-checked like any slice.135kayan_column_bytes: proc(c: *kayan_column, row: u64) -> []u8 in level = {136 return c*.data[row * c*.size..(row + 1) * c*.size];137}138 139// The typed view: the column's first `count` rows as `[]T`. This is the one140// place bytes become a type, so it takes nothing on trust. T must be the141// column's size, and plain data: any bytes are then a valid T, so reading the142// wrong type gives wrong numbers, never a bad pointer. The rows must fit, which143// the slice knows and no field can lie about, and the bytes must be aligned for144// T. The view is `[]T in level`: the bytes are in the region, so the view is145// good memory for as long as the region is. After the column grows it is the146// old block, still readable, no longer the column: take the view again.147kayan_column_slice: proc<T>(c: *const kayan_column, count: u64) -> []T in level = {148 if (c*.size != sizeof(T)) {149 printfmt("kayan: a column of {}-byte rows read as a {}-byte type\n", c*.size, sizeof(T));150 exit(1);151 }152 if (!reflect_is_plain(T<>&)) {153 printfmt("kayan: a column holds plain data, and this {}-byte type is not\n", sizeof(T));154 exit(1);155 }156 if (count > c*.data.length / sizeof(T)) {157 printfmt("kayan: a view of {} rows on a column with room for {}\n", count, c*.data.length / sizeof(T));158 exit(1);159 }160 unsafe {161 if (cast(c*.data.data, usize) % alignof(T) != 0) {162 printfmt("kayan: a column's bytes are not aligned to {}\n", alignof(T));163 exit(1);164 }165 raw: ^T = cast(c*.data.data, ^T);166 return cast(raw[0..count], []T in level);167 }168}169 170// Copying and zeroing a row are loops over the row's bytes: checked code.171kayan_column_copy_row: proc(c: *kayan_column, to: u64, from: u64) -> void = {172 dst: []u8 = kayan_column_bytes(c, to);173 src: []u8 = kayan_column_bytes(c, from);174 for (i: u64 = 0; i < dst.length; i += 1) {175 dst[i] = src[i];176 }177}178 179kayan_column_zero_row: proc(c: *kayan_column, row: u64) -> void = {180 bytes: []u8 = kayan_column_bytes(c, row);181 for (i: u64 = 0; i < bytes.length; i += 1) {182 bytes[i] = 0;183 }184}185 186// ---------------------------------------------------------------------------187// The table188// ---------------------------------------------------------------------------189 190// `columns[id]` is a column when the signature has bit `id`, and null191// otherwise. Count and capacity are the table's, shared by its columns, so192// they can never disagree; `entities` grows on its own and always has `count`193// rows.194//195// The edges: `add_edges[id]` is the table this one's entities move to when196// they gain component `id`, and `remove_edges[id]` when they lose it. Each is197// the table's index plus one, so 0, which a new table starts with, means "not198// known yet".199kayan_table: struct = {200 signature: u64;201 count: u64;202 capacity: u64;203 entities: *vec<kayan_entity> in level;204 columns: [64]option<*kayan_column in level>;205 add_edges: [64]u32;206 remove_edges: [64]u32;207}208 209// A zeroed table in the region, with an empty entities vec: the world fills210// in the signature and the columns.211kayan_table_new: proc(h: *kayan_heap in level) -> *kayan_table in level = {212 unsafe {213 t: *kayan_table in level = cast(memops_arena_push_zero(h*.arena[0]&, sizeof(kayan_table), alignof(kayan_table)), *kayan_table in level);214 v: *vec<kayan_entity> in level = cast(memops_arena_push_zero(h*.arena[0]&, sizeof(vec<kayan_entity>), alignof(vec<kayan_entity>)), *vec<kayan_entity> in level);215 v*.arena = h*.arena;216 t*.entities = v;217 return t;218 }219}220 221// The world: the registry, the slots and free list, and every table, with a222// map from signature to table index so finding one is a lookup, not a search.223// Tables are referred to by index, so a table can be named in a slot; each224// table is its own allocation, so a pointer to one stays good when the next225// is made.226kayan_world: struct = {227 heap: *kayan_heap in level;228 infos: [64]option<*const reflect in permanent>;229 component_count: u32;230 tables: *vec<*kayan_table in level> in level;231 by_signature: *hashmap<u64, u32> in level;232 slots: *vec<kayan_slot> in level;233 free: *vec<u32> in level;234 live: u64;235}236 237kayan_bit: proc(id: u32) -> u64 = {238 return 1'u64 shl cast(id, u64);239}240 241kayan_id_of: proc<T>() -> u32 = {242 return cast(component_of<T>(), u32);243}244 245kayan_mask_of: proc<T>() -> u64 = {246 return kayan_bit(kayan_id_of<T>());247}248 249// The reflect records live for the whole program, so `permanent` is their250// region and the world may keep them.251kayan_register: proc(w: *kayan_world, id: u32, info: *const reflect in permanent) -> void = {252 if (id >= 64) {253 printfmt("kayan: component id {} is past the 64 a signature can hold\n", id);254 exit(1);255 }256 w*.infos[id] = info;257 if (id + 1 > w*.component_count) {258 w*.component_count = id + 1;259 }260}261 262// A registered component's record. The checker needs the null test; the263// return after exit is never reached.264kayan_info: proc(w: *const kayan_world, id: u32) -> *const reflect in permanent = {265 info: option<*const reflect in permanent> = w*.infos[id];266 if (info) {267 return info;268 }269 printfmt("kayan: component {} was never registered\n", id);270 exit(1);271 return kayan_entity<>&;272}273 274kayan_table_has: proc(t: *const kayan_table, id: u32) -> b32 = {275 return (t*.signature & kayan_bit(id)) != 0;276}277 278// A table for a signature: one empty column per component in it, sized from279// the registry. Every component in the signature must be registered.280kayan_table_make: proc(w: *kayan_world, signature: u64) -> *kayan_table in level = {281 t: *kayan_table in level = kayan_table_new(w*.heap);282 t*.signature = signature;283 for (id: u32 = 0; id < 64; id += 1) {284 if ((signature & kayan_bit(id)) != 0) {285 if (id >= w*.component_count) {286 printfmt("kayan: signature {} uses component {}, which was never registered\n", signature, id);287 exit(1);288 }289 t*.columns[id] = kayan_column_make(w*.heap, kayan_info(w, id));290 }291 }292 return t;293}294 295// Double the room in every column at once.296kayan_table_grow: proc(w: *kayan_world, t: *kayan_table) -> void = {297 capacity: u64 = t*.capacity == 0 ? 4 : t*.capacity * 2;298 for (id: u32 = 0; id < w*.component_count; id += 1) {299 c: option<*kayan_column in level> = t*.columns[id];300 if (c) {301 kayan_column_grow(w*.heap, c, t*.count, capacity);302 }303 }304 t*.capacity = capacity;305}306 307// A new row at the end, owned by `e`, with every component zeroed.308kayan_table_add_row: proc(w: *kayan_world, t: *kayan_table, e: kayan_entity) -> u64 = {309 if (t*.count == t*.capacity) {310 kayan_table_grow(w, t);311 }312 row: u64 = t*.count;313 t*.count += 1;314 _ = vec<kayan_entity>push(t*.entities, e);315 for (id: u32 = 0; id < w*.component_count; id += 1) {316 c: option<*kayan_column in level> = t*.columns[id];317 if (c) {318 kayan_column_zero_row(c, row);319 }320 }321 return row;322}323 324// Who owns each of the table's rows. It only reads, and the checker sees that,325// so a system holds this and two or three columns of one table at once.326kayan_entities: proc(t: *const kayan_table) -> []kayan_entity in level = {327 return vec<kayan_entity>items(t*.entities);328}329 330// 1.4's swap-remove, for every column: the last row is copied over `row`, and331// the slot of the entity that owned the last row is told where it went.332kayan_table_remove_row: proc(w: *kayan_world, t: *kayan_table, row: u64) -> void = {333 last: u64 = t*.count - 1;334 if (row != last) {335 for (id: u32 = 0; id < w*.component_count; id += 1) {336 c: option<*kayan_column in level> = t*.columns[id];337 if (c) {338 kayan_column_copy_row(c, row, last);339 }340 }341 moved: kayan_entity = vec<kayan_entity>at(t*.entities, last)*;342 vec<kayan_entity>at(t*.entities, row)* = moved;343 vec<kayan_slot>at(w*.slots, cast(moved.index, u64))*.row = cast(row, u32);344 }345 _ = vec<kayan_entity>pop(t*.entities);346 t*.count = last;347}348 349// The typed view of one of the table's columns: T's rows, all of them in use.350// Asking for a component the table does not have is a bug, so it stops. The351// slice is good memory for as long as the region, but when the table grows it is352// the old block, no longer the column: take it again, and 3.3 makes that a rule.353kayan_slice: proc<T>(t: *const kayan_table) -> []T in level = {354 id: u32 = kayan_id_of<T>();355 c: option<*kayan_column in level> = t*.columns[id];356 if (!c) {357 printfmt("kayan: the table has no column for component {}\n", id);358 exit(1);359 }360 return kayan_column_slice<T>(c, t*.count);361}362 363kayan_table_at: proc(w: *const kayan_world, index: u32) -> *kayan_table in level = {364 return vec<*kayan_table in level>at(w*.tables, cast(index, u64))*;365}366 367// The table for a signature: found if it exists, made if it does not. This is368// where archetypes come from.369kayan_table_for: proc(w: *kayan_world, signature: u64) -> u32 = {370 found: option<*u32> = hashmap<u64, u32>get(w*.by_signature, signature);371 if (found) {372 return found*;373 }374 index: u32 = cast(w*.tables*.length, u32);375 _ = vec<*kayan_table in level>push(w*.tables, kayan_table_make(w, signature));376 _ = hashmap<u64, u32>set(w*.by_signature, signature, index);377 return index;378}379 380// ---------------------------------------------------------------------------381// The world382// ---------------------------------------------------------------------------383 384kayan_world_make: proc(r: region level) -> *kayan_world in level = {385 return new<kayan_world>(r, {386 .heap = kayan_heap_make(r),387 .tables = vec<*kayan_table in level>make(r, 8),388 .by_signature = hashmap<u64, u32>make(r),389 .slots = vec<kayan_slot>make(r, 64),390 .free = vec<u32>make(r, 16),391 });392}393 394kayan_alive: proc(w: *const kayan_world, e: kayan_entity) -> b32 = {395 if (cast(e.index, u64) >= w*.slots*.length) {396 return false;397 }398 return vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.generation == e.generation;399}400 401// A new entity with the components in `signature`, all zeroed, in the table402// for that signature.403kayan_spawn: proc(w: *kayan_world, signature: u64) -> kayan_entity = {404 e: kayan_entity = {};405 if (w*.free*.length > 0) {406 e.index = vec<u32>pop(w*.free);407 } else {408 e.index = cast(w*.slots*.length, u32);409 _ = vec<kayan_slot>push(w*.slots, { .generation = 1 });410 }411 e.generation = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.generation;412 table_index: u32 = kayan_table_for(w, signature);413 t: *kayan_table in level = kayan_table_at(w, table_index);414 row: u64 = kayan_table_add_row(w, t, e);415 // The slot is taken after the calls that change the world: a pointer into416 // `slots` from before them may be stale, and the checker says so.417 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));418 slot*.table = table_index;419 slot*.row = cast(row, u32);420 w*.live += 1;421 return e;422}423 424kayan_despawn: proc(w: *kayan_world, e: kayan_entity) -> kayan_status = {425 if (!kayan_alive(w, e)) {426 return kayan_status.dead;427 }428 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));429 table: u32 = slot*.table;430 row: u32 = slot*.row;431 kayan_table_remove_row(w, kayan_table_at(w, table), cast(row, u64));432 slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));433 slot*.generation += 1;434 _ = vec<u32>push(w*.free, e.index);435 w*.live -= 1;436 return kayan_status.ok;437}438 439// The components a live entity has: its table's signature. 0 for a dead one.440kayan_signature: proc(w: *const kayan_world, e: kayan_entity) -> u64 = {441 if (!kayan_alive(w, e)) {442 return 0;443 }444 return kayan_table_at(w, vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.table)*.signature;445}446 447// ---------------------------------------------------------------------------448// Moving between tables449// ---------------------------------------------------------------------------450 451// The table an entity of table `from` moves to when it gains component `id`.452// The first time, it is found by signature, which can make the table; the453// answer is written on both tables, as from's add edge and the destination's454// remove edge, so neither direction is looked up again.455kayan_table_with: proc(w: *kayan_world, from: u32, id: u32) -> u32 = {456 edge: u32 = kayan_table_at(w, from)*.add_edges[id];457 if (edge != 0) {458 return edge - 1;459 }460 to: u32 = kayan_table_for(w, kayan_table_at(w, from)*.signature | kayan_bit(id));461 kayan_table_at(w, from)*.add_edges[id] = to + 1;462 kayan_table_at(w, to)*.remove_edges[id] = from + 1;463 return to;464}465 466// The same for losing component `id`.467kayan_table_without: proc(w: *kayan_world, from: u32, id: u32) -> u32 = {468 edge: u32 = kayan_table_at(w, from)*.remove_edges[id];469 if (edge != 0) {470 return edge - 1;471 }472 to: u32 = kayan_table_for(w, kayan_table_at(w, from)*.signature & ~kayan_bit(id));473 kayan_table_at(w, from)*.remove_edges[id] = to + 1;474 kayan_table_at(w, to)*.add_edges[id] = from + 1;475 return to;476}477 478// Move a live entity to table `to_index`. Components both tables have are479// copied, byte for byte; components only the new table has start zeroed;480// components only the old table has are dropped with its old row.481kayan_move: proc(w: *kayan_world, e: kayan_entity, to_index: u32) -> void = {482 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));483 from_index: u32 = slot*.table;484 if (from_index == to_index) {485 return;486 }487 from_row: u64 = cast(slot*.row, u64);488 from: *kayan_table in level = kayan_table_at(w, from_index);489 to: *kayan_table in level = kayan_table_at(w, to_index);490 to_row: u64 = kayan_table_add_row(w, to, e);491 for (id: u32 = 0; id < w*.component_count; id += 1) {492 fc: option<*kayan_column in level> = from*.columns[id];493 tc: option<*kayan_column in level> = to*.columns[id];494 if (fc && tc) {495 dst: []u8 = kayan_column_bytes(tc, to_row);496 src: []u8 = kayan_column_bytes(fc, from_row);497 for (i: u64 = 0; i < dst.length; i += 1) {498 dst[i] = src[i];499 }500 }501 }502 kayan_table_remove_row(w, from, from_row);503 slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));504 slot*.table = to_index;505 slot*.row = cast(to_row, u32);506}507 508// ---------------------------------------------------------------------------509// The byte layer, for tools510// ---------------------------------------------------------------------------511 512// e's component `id` as its bytes: an empty slice if e is dead or has no such513// component. The typed layer is for game code, which knows its types; this is514// for tools that work from reflect records instead.515kayan_get_bytes: proc(w: *const kayan_world, e: kayan_entity, id: u32) -> []u8 in level = {516 none: []u8 in level = {};517 if (!kayan_alive(w, e) || id >= w*.component_count) {518 return none;519 }520 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));521 table: u32 = slot*.table;522 row: u32 = slot*.row;523 t: *kayan_table in level = kayan_table_at(w, table);524 c: option<*kayan_column in level> = t*.columns[id];525 if (!c) {526 return none;527 }528 return kayan_column_bytes(c, cast(row, u64));529}530 531// ---------------------------------------------------------------------------532// The typed layer533// ---------------------------------------------------------------------------534 535// e's T, or null if e is dead or has no T. Handle, slot, table, row, typed536// view: the same path as 1.4's get_position, for any component. `option<*T>` says the537// null is possible, so the caller tests before it reads. The pointer is good538// until the next structural change, so use it now and keep the handle.539kayan_get: proc<T>(w: *const kayan_world, e: kayan_entity) -> option<*T in level> = {540 if (!kayan_alive(w, e)) {541 return null;542 }543 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));544 table: u32 = slot*.table;545 row: u32 = slot*.row;546 t: *kayan_table in level = kayan_table_at(w, table);547 if (!kayan_table_has(t, kayan_id_of<T>())) {548 return null;549 }550 rows: []T in level = kayan_slice<T>(t);551 return rows[cast(row, u64)]&;552}553 554kayan_has: proc<T>(w: *const kayan_world, e: kayan_entity) -> b32 = {555 return (kayan_signature(w, e) & kayan_mask_of<T>()) != 0;556}557 558// Write e's T. It must already have one: giving an entity a component it does559// not have changes its table, which is 2.6.560kayan_set: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> kayan_status = {561 if (!kayan_alive(w, e)) {562 return kayan_status.dead;563 }564 p: option<*T> = kayan_get<T>(w, e);565 if (!p) {566 return kayan_status.missing;567 }568 p* = value;569 return kayan_status.ok;570}571 572// Give e a T. If it already has one, this is kayan_set; if not, it moves to573// the table with T and then sets it.574kayan_add: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> kayan_status = {575 if (!kayan_alive(w, e)) {576 return kayan_status.dead;577 }578 if (!kayan_has<T>(w, e)) {579 from: u32 = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.table;580 kayan_move(w, e, kayan_table_with(w, from, kayan_id_of<T>()));581 }582 return kayan_set<T>(w, e, value);583}584 585// Take e's T away, moving it to the table without T.586kayan_remove: proc<T>(w: *kayan_world, e: kayan_entity) -> kayan_status = {587 if (!kayan_alive(w, e)) {588 return kayan_status.dead;589 }590 if (!kayan_has<T>(w, e)) {591 return kayan_status.missing;592 }593 from: u32 = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.table;594 kayan_move(w, e, kayan_table_without(w, from, kayan_id_of<T>()));595 return kayan_status.ok;596}597 598// The invariant, over every table: each row belongs to a live entity whose599// slot points back at that table and row, and the rows of all tables add up to600// the live count.601kayan_verify: proc(w: *const kayan_world) -> u64 = {602 problems: u64 = 0;603 rows: u64 = 0;604 for (ti: u64 = 0; ti < w*.tables*.length; ti += 1) {605 t: *kayan_table in level = kayan_table_at(w, cast(ti, u32));606 for (row: u64 = 0; row < t*.count; row += 1) {607 // Taken row by row: kayan_alive takes the world, and the checker608 // does not know that it leaves the rows where they are.609 es: []kayan_entity = kayan_entities(t);610 e: kayan_entity = es[row];611 if (!kayan_alive(w, e)) {612 printfmt("kayan_verify: table {} row {} holds dead entity {}:{}\n", ti, row, e.index, e.generation);613 problems += 1;614 continue;615 }616 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));617 if (cast(slot*.table, u64) != ti || cast(slot*.row, u64) != row) {618 printfmt("kayan_verify: {}:{} is in table {} row {}, its slot says table {} row {}\n",619 e.index, e.generation, ti, row, slot*.table, slot*.row);620 problems += 1;621 }622 }623 rows += t*.count;624 }625 if (rows != w*.live) {626 printfmt("kayan_verify: {} rows in tables, {} live entities\n", rows, w*.live);627 problems += 1;628 }629 return problems;630}631 632kayan_print_signature: proc(w: *const kayan_world, signature: u64) -> void = {633 printfmt("{");634 first: b32 = true;635 for (id: u32 = 0; id < w*.component_count; id += 1) {636 if ((signature & kayan_bit(id)) != 0) {637 if (!first) {638 printfmt(" ");639 }640 printfmt("{}", kayan_info(w, id)*.name);641 first = false;642 }643 }644 printfmt("}");645}646 647kayan_print_table: proc(w: *const kayan_world, ti: u32) -> void = {648 t: *kayan_table in level = kayan_table_at(w, ti);649 printfmt(" table {} ", ti);650 kayan_print_signature(w, t*.signature);651 printfmt(":");652 es: []kayan_entity = kayan_entities(t);653 for (row: u64 = 0; row < es.length; row += 1) {654 e: kayan_entity = es[row];655 printfmt(" {}:{}", e.index, e.generation);656 }657 printfmt("\n");658}659 660kayan_print_tables: proc(w: *const kayan_world) -> void = {661 for (ti: u32 = 0; ti < cast(w*.tables*.length, u32); ti += 1) {662 kayan_print_table(w, ti);663 }664}665 666// ---------------------------------------------------------------------------667// The inspector: reflect records, no component types668// ---------------------------------------------------------------------------669 670kayan_indent: proc(depth: u32) -> void = {671 for (i: u32 = 0; i < depth; i += 1) {672 printfmt(" ");673 }674}675 676// One value of a builtin type, read from its bytes by its type's name. The677// bytes may sit at any address, so the read is a copy into a local: this is the678// inspector's one unsafe block.679kayan_print_builtin: proc(type_name: ^const char, at: []const u8) -> void = {680 if (at.length < 4) {681 printfmt("<{} bytes>", at.length);682 return;683 }684 unsafe {685 if (reflect_cstr_equal(type_name, "f32")) {686 f: f32 = 0.0;687 memcpy(f&, at.data, 4);688 printfmt("{:.2}", cast(f, f64));689 } else if (reflect_cstr_equal(type_name, "u32")) {690 u: u32 = 0;691 memcpy(u&, at.data, 4);692 printfmt("{}", u);693 } else if (reflect_cstr_equal(type_name, "i32")) {694 i: i32 = 0;695 memcpy(i&, at.data, 4);696 printfmt("{}", i);697 } else {698 printfmt("<{}>", type_name);699 }700 }701}702 703// Every field of a struct whose bytes are `base`, nested structs indented704// under their field. A field's bytes are cut from `base` by its offset and705// size, both from the reflect record.706kayan_print_fields: proc(type: *const reflect, base: []const u8, depth: u32) -> void = {707 for (i: u64 = 0; i < type*.count; i += 1) {708 f: option<*const reflect_field> = reflect_field_at(type, i);709 if (!f) {710 continue;711 }712 if (reflect_field_has_attr(f, "hidden")) {713 continue;714 }715 at: []const u8 = base[f*.offset..f*.offset + f*.size];716 kayan_indent(depth);717 printfmt("{}:", f*.name);718 info: option<*const reflect> = reflect_field_info(f);719 if (info && info*.kind == reflect_kind_struct) {720 printfmt("\n");721 kayan_print_fields(info, at, depth + 1);722 } else {723 printfmt(" ");724 kayan_print_builtin(f*.type, at);725 printfmt("\n");726 }727 }728}729 730kayan_inspect: proc(w: *const kayan_world, e: kayan_entity) -> void = {731 if (!kayan_alive(w, e)) {732 printfmt("{}:{} (dead)\n", e.index, e.generation);733 return;734 }735 printfmt("{}:{} ", e.index, e.generation);736 signature: u64 = kayan_signature(w, e);737 kayan_print_signature(w, signature);738 printfmt("\n");739 for (id: u32 = 0; id < w*.component_count; id += 1) {740 if ((signature & kayan_bit(id)) != 0) {741 info: *const reflect in permanent = kayan_info(w, id);742 printfmt(" {}\n", info*.name);743 bytes: []u8 = kayan_get_bytes(w, e, id);744 kayan_print_fields(info, bytes, 2);745 }746 }747}748 749// Every table: rows in use out of room, and per column its row size and its750// bytes in use out of bytes allocated. The memory a table holds and does not751// use is the cost of doubling on growth.752kayan_print_layout: proc(w: *const kayan_world) -> void = {753 for (ti: u32 = 0; ti < cast(w*.tables*.length, u32); ti += 1) {754 t: *kayan_table in level = kayan_table_at(w, ti);755 printfmt("table {} ", ti);756 kayan_print_signature(w, t*.signature);757 printfmt(": {} rows, room for {}\n", cast(t*.count, u32), cast(t*.capacity, u32));758 for (id: u32 = 0; id < w*.component_count; id += 1) {759 c: option<*kayan_column in level> = t*.columns[id];760 if (c) {761 size: u64 = c*.size;762 printfmt(" {:-9} {:2} bytes a row, {:4} of {:4} bytes in use\n", kayan_info(w, id)*.name, cast(size, u32),763 cast(size * t*.count, u32), cast(size * t*.capacity, u32));764 }765 }766 }767}768 769// ---------------------------------------------------------------------------770// Queries771// ---------------------------------------------------------------------------772 773// The tables a system runs over: every table whose signature has all of `all`774// and none of `none`, kept by index. Tables are only ever added, at the end of775// w.tables, so the query remembers how many it has checked and, brought up to776// date, looks at the new ones only.777kayan_query: struct = {778 all: u64;779 none: u64;780 tables: *vec<u32> in level;781 checked: u64;782}783 784// The list of matched tables lives in the world's region, so a query says785// which region it is made in.786kayan_query_init: proc(r: region level, all: u64, none: u64) -> kayan_query = {787 return { .all = all, .none = none, .tables = vec<u32>make(r, 4) };788}789 790kayan_matches: proc(signature: u64, all: u64, none: u64) -> b32 = {791 return (signature & all) == all && (signature & none) == 0;792}793 794// Check the tables made since the query last looked. With none new, this is795// one comparison, so call it before every walk.796kayan_query_update: proc(w: *const kayan_world, q: *kayan_query) -> void = {797 for (ti: u64 = q*.checked; ti < w*.tables*.length; ti += 1) {798 if (kayan_matches(kayan_table_at(w, cast(ti, u32))*.signature, q*.all, q*.none)) {799 _ = vec<u32>push(q*.tables, cast(ti, u32));800 }801 }802 q*.checked = w*.tables*.length;803}804 805// The query's i-th table.806kayan_query_table: proc(w: *const kayan_world, q: *const kayan_query, i: u64) -> *kayan_table in level = {807 return kayan_table_at(w, vec<u32>at(q*.tables, i)*);808}809 810// How many entities the query matches: the rows of its tables.811kayan_query_count: proc(w: *const kayan_world, q: *kayan_query) -> u64 = {812 kayan_query_update(w, q);813 n: u64 = 0;814 for (i: u64 = 0; i < q*.tables*.length; i += 1) {815 n += kayan_query_table(w, q, i)*.count;816 }817 return n;818}819 820kayan_print_query: proc(w: *const kayan_world, name: ^const char, q: *kayan_query) -> void = {821 kayan_query_update(w, q);822 printfmt("{}: all ", name);823 kayan_print_signature(w, q*.all);824 printfmt(", none ");825 kayan_print_signature(w, q*.none);826 printfmt("; tables: {}, entities: {}\n", q*.tables*.length, kayan_query_count(w, q));827 for (i: u64 = 0; i < q*.tables*.length; i += 1) {828 kayan_print_table(w, vec<u32>at(q*.tables, i)*);829 }830}831 832// ---------------------------------------------------------------------------833// Systems834// ---------------------------------------------------------------------------835 836// What a system gets for each table it runs on: the world, the table, and the837// frame's time step. One struct rather than three parameters, so that adding838// to it later changes no system's signature. Both pointers name their region,839// so the struct can be made anywhere level is open.840kayan_iter: struct = {841 world: *kayan_world in level;842 table: *kayan_table in level;843 dt: f32;844}845 846kayan_system_proc: alias = *proc(it: *kayan_iter) -> void;847 848// A system is code: a name, and the proc run on each table its query matches.849// A C string and a proc pointer are both pointers without a region, so a850// struct holding them is a carrier: safe code reaches it only through a851// pointer into a region, made by a trusted proc. Neither changes after it is852// made, so nothing is lost.853kayan_system: struct = {854 name: ^const char;855 run: kayan_system_proc;856}857 858kayan_system_make: proc(r: region level, name: ^const char, run: kayan_system_proc) -> *kayan_system in level = {859 unsafe {860 s: *kayan_system in level = cast(memops_arena_push_zero(r, sizeof(kayan_system), alignof(kayan_system)), *kayan_system in level);861 s*.name = name;862 s*.run = run;863 return s;864 }865}866 867// A system in the schedule: which system, the query that finds its tables,868// and what it has run over so far. `tables` and `rows` count the calls.869kayan_scheduled: struct = {870 system: *kayan_system in level;871 query: kayan_query;872 tables: u64;873 rows: u64;874}875 876// The schedule is a list of these, in the order the frame runs them. Each877// entry is its own allocation so a pointer to one is a pointer into level,878// which the checker never counts as stale.879kayan_schedule_add: proc(r: region level, systems: *vec<*kayan_scheduled in level>, name: ^const char,880 all: u64, none: u64, run: kayan_system_proc) -> void = {881 _ = vec<*kayan_scheduled in level>push(systems, new<kayan_scheduled>(r, {882 .system = kayan_system_make(r, name, run),883 .query = kayan_query_init(r, all, none),884 }));885}886 887kayan_scheduled_at: proc(systems: *const vec<*kayan_scheduled in level>, s: u64) -> *kayan_scheduled in level = {888 return vec<*kayan_scheduled in level>at_const(systems, s)*;889}890 891// One frame: every system in order, each over every table it matches that has892// rows. A system sees everything the systems before it wrote this frame.893kayan_progress: proc(w: *kayan_world in level, systems: *const vec<*kayan_scheduled in level>, dt: f32) -> void = {894 for (s: u64 = 0; s < systems*.length; s += 1) {895 sys: *kayan_scheduled in level = kayan_scheduled_at(systems, s);896 kayan_query_update(w, sys*.query&);897 for (i: u64 = 0; i < sys*.query.tables*.length; i += 1) {898 it: kayan_iter = { .world = w, .table = kayan_query_table(w, sys*.query&, i), .dt = dt };899 if (it.table*.count > 0) {900 sys*.tables += 1;901 sys*.rows += it.table*.count;902 sys*.system*.run(it&);903 }904 }905 }906}907 908kayan_print_schedule: proc(w: *const kayan_world, systems: *const vec<*kayan_scheduled in level>) -> void = {909 for (s: u64 = 0; s < systems*.length; s += 1) {910 sys: *kayan_scheduled in level = kayan_scheduled_at(systems, s);911 kayan_print_query(w, sys*.system*.name, sys*.query&);912 }913}914 915// ---------------------------------------------------------------------------916// The program917// ---------------------------------------------------------------------------918 919// Kernels: plain loops over plain slices, with components for elements.920// noalias on what each one writes, as in lessons 1 to 7.921integrate: proc[unsafe](ps: noalias []position, vs: []const velocity, dt: f32) -> void = {922 for (i: u64 = 0; i < ps.length; i += 1) {923 ps[i].at.x += vs[i].per_second.x * dt;924 ps[i].at.y += vs[i].per_second.y * dt;925 }926}927 928burn: proc[unsafe](hs: noalias []health, fires: []const burning, dt: f32) -> void = {929 for (i: u64 = 0; i < hs.length; i += 1) {930 hs[i].points -= fires[i].per_second * dt;931 }932}933 934age: proc[unsafe](ls: noalias []lifetime, dt: f32) -> void = {935 for (i: u64 = 0; i < ls.length; i += 1) {936 ls[i].seconds -= dt;937 }938}939 940// Systems: the table's columns, handed to a kernel.941move_system: proc(it: *kayan_iter) -> void = {942 unsafe { integrate(kayan_slice<position>(it*.table), kayan_slice<velocity>(it*.table), it*.dt); }943}944 945burn_system: proc(it: *kayan_iter) -> void = {946 unsafe { burn(kayan_slice<health>(it*.table), kayan_slice<burning>(it*.table), it*.dt); }947}948 949age_system: proc(it: *kayan_iter) -> void = {950 unsafe { age(kayan_slice<lifetime>(it*.table), it*.dt); }951}952 953spawn_turret: proc(w: *kayan_world, x: f32, y: f32) -> kayan_entity = {954 e: kayan_entity = kayan_spawn(w, kayan_mask_of<position>() | kayan_mask_of<turret>());955 _ = kayan_set<position>(w, e, { .at = { .x = x, .y = y } });956 _ = kayan_set<turret>(w, e, { .cooldown = 0.5, .timer = 0.0 });957 return e;958}959 960spawn_drone: proc(w: *kayan_world, x: f32, y: f32) -> kayan_entity = {961 e: kayan_entity = kayan_spawn(w, kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<health>());962 _ = kayan_set<position>(w, e, { .at = { .x = x, .y = y } });963 _ = kayan_set<velocity>(w, e, { .per_second = { .x = -1.0, .y = 0.0 } });964 _ = kayan_set<health>(w, e, { .points = 100.0, .max = 100.0 });965 return e;966}967 968spawn_bullet: proc(w: *kayan_world, x: f32, y: f32, seconds: f32) -> kayan_entity = {969 e: kayan_entity = kayan_spawn(w, kayan_mask_of<position>() | kayan_mask_of<velocity>()970 | kayan_mask_of<damage>() | kayan_mask_of<lifetime>());971 _ = kayan_set<position>(w, e, { .at = { .x = x, .y = y } });972 _ = kayan_set<velocity>(w, e, { .per_second = { .x = 8.0, .y = 0.0 } });973 _ = kayan_set<damage>(w, e, { .amount = 25.0 });974 _ = kayan_set<lifetime>(w, e, { .seconds = seconds });975 return e;976}977 978// Every entity, with the components this program cares about. Each value is979// copied out of its pointer before the next lookup: a lookup takes the world,980// and a pointer into it does not survive that in the checker's eyes.981print_state: proc(w: *const kayan_world) -> void = {982 for (ti: u32 = 0; ti < cast(w*.tables*.length, u32); ti += 1) {983 t: *kayan_table in level = kayan_table_at(w, ti);984 for (row: u64 = 0; row < t*.count; row += 1) {985 es: []kayan_entity = kayan_entities(t);986 e: kayan_entity = es[row];987 p: option<*position> = kayan_get<position>(w, e);988 if (!p) {989 continue;990 }991 at: vec2 = p*.at;992 printfmt(" {}:{} at ({:5.2}, {:5.2})", e.index, e.generation, cast(at.x, f64), cast(at.y, f64));993 h: option<*health> = kayan_get<health>(w, e);994 if (h) {995 printfmt(" health {:.0}", cast(h*.points, f64));996 }997 if (kayan_has<burning>(w, e)) {998 printfmt(", burning");999 }1000 l: option<*lifetime> = kayan_get<lifetime>(w, e);1001 if (l) {1002 printfmt(" lifetime {:5.2}", cast(l*.seconds, f64));1003 }1004 printfmt("\n");1005 }1006 }1007}1008 1009main: proc() -> i32 = {1010 level: region = {1011 w: *kayan_world in level = kayan_world_make(level);1012 kayan_register(w, kayan_id_of<position>(), position<>&);1013 kayan_register(w, kayan_id_of<velocity>(), velocity<>&);1014 kayan_register(w, kayan_id_of<health>(), health<>&);1015 kayan_register(w, kayan_id_of<damage>(), damage<>&);1016 kayan_register(w, kayan_id_of<lifetime>(), lifetime<>&);1017 kayan_register(w, kayan_id_of<turret>(), turret<>&);1018 kayan_register(w, kayan_id_of<burning>(), burning<>&);1019 1020 _ = spawn_turret(w, 0.0, 0.0);1021 _ = spawn_drone(w, 20.0, 4.0);1022 hot: kayan_entity = spawn_drone(w, 24.0, 0.0);1023 _ = spawn_drone(w, 28.0, -4.0);1024 _ = kayan_add<burning>(w, hot, { .per_second = 8.0 });1025 _ = spawn_bullet(w, 1.0, 0.0, 0.5);1026 _ = spawn_bullet(w, 1.0, 4.0, 1.0);1027 _ = spawn_bullet(w, 1.0, -4.0, 1.5);1028 1029 // The schedule. The order is the data flow: a system that reads what1030 // another writes runs after it.1031 systems: *vec<*kayan_scheduled in level> in level = vec<*kayan_scheduled in level>make(level, 3);1032 kayan_schedule_add(level, systems, "burn", kayan_mask_of<health>() | kayan_mask_of<burning>(), 0, burn_system);1033 kayan_schedule_add(level, systems, "age", kayan_mask_of<lifetime>(), 0, age_system);1034 kayan_schedule_add(level, systems, "move", kayan_mask_of<position>() | kayan_mask_of<velocity>(), 0, move_system);1035 printfmt("the schedule, run in this order every frame:\n");1036 kayan_print_schedule(w, systems);1037 printfmt("\nat the start:\n");1038 print_state(w);1039 1040 dt: f32 = 0.25;1041 for (frame: u32 = 1; frame <= 8; frame += 1) {1042 kayan_progress(w, systems, dt);1043 if (frame % 4 == 0) {1044 printfmt("\nafter frame {}, {:.2} seconds:\n", frame, cast(cast(frame, f32) * dt, f64));1045 print_state(w);1046 }1047 }1048 1049 printfmt("\n8 frames, one call per table per frame:\n");1050 for (s: u64 = 0; s < 3; s += 1) {1051 sys: *kayan_scheduled in level = kayan_scheduled_at(systems, s);1052 printfmt(" {:-4} called {:2} times, over {:2} rows\n", sys*.system*.name, cast(sys*.tables, u32), cast(sys*.rows, u32));1053 }1054 1055 // A system may change values, never structure, so nothing has removed the1056 // bullets whose time is up.1057 expired: u64 = 0;1058 ager: *kayan_scheduled in level = kayan_scheduled_at(systems, 1);1059 for (i: u64 = 0; i < ager*.query.tables*.length; i += 1) {1060 ls: []lifetime = kayan_slice<lifetime>(kayan_query_table(w, ager*.query&, i));1061 for (row: u64 = 0; row < ls.length; row += 1) {1062 if (ls[row].seconds <= 0.0) {1063 expired += 1;1064 }1065 }1066 }1067 printfmt("{} bullets are out of time and still in the world\n", expired);1068 printfmt("kayan_verify: {} problems\n", kayan_verify(w));1069 }1070 return 0;1071}What systems prints. tools/check.py fails the chapter if a single byte differs.
the schedule, run in this order every frame:
burn: all {health burning}, none {}; tables: 1, entities: 1
table 2 {position velocity health burning}: 2:1
age: all {lifetime}, none {}; tables: 1, entities: 3
table 3 {position velocity damage lifetime}: 4:1 5:1 6:1
move: all {position velocity}, none {}; tables: 3, entities: 6
table 1 {position velocity health}: 1:1 3:1
table 2 {position velocity health burning}: 2:1
table 3 {position velocity damage lifetime}: 4:1 5:1 6:1
at the start:
0:1 at ( 0.00, 0.00)
1:1 at (20.00, 4.00) health 100
3:1 at (28.00, -4.00) health 100
2:1 at (24.00, 0.00) health 100, burning
4:1 at ( 1.00, 0.00) lifetime 0.50
5:1 at ( 1.00, 4.00) lifetime 1.00
6:1 at ( 1.00, -4.00) lifetime 1.50
after frame 4, 1.00 seconds:
0:1 at ( 0.00, 0.00)
1:1 at (19.00, 4.00) health 100
3:1 at (27.00, -4.00) health 100
2:1 at (23.00, 0.00) health 92, burning
4:1 at ( 9.00, 0.00) lifetime -0.50
5:1 at ( 9.00, 4.00) lifetime 0.00
6:1 at ( 9.00, -4.00) lifetime 0.50
after frame 8, 2.00 seconds:
0:1 at ( 0.00, 0.00)
1:1 at (18.00, 4.00) health 100
3:1 at (26.00, -4.00) health 100
2:1 at (22.00, 0.00) health 84, burning
4:1 at (17.00, 0.00) lifetime -1.50
5:1 at (17.00, 4.00) lifetime -1.00
6:1 at (17.00, -4.00) lifetime -0.50
8 frames, one call per table per frame:
burn called 8 times, over 8 rows
age called 8 times, over 24 rows
move called 24 times, over 48 rows
3 bullets are out of time and still in the world
kayan_verify: 0 problems
Build an ECS · part 3, systems · 3.3
Commands
ecs/src/3.systems/3.3.commands/commands.rin
In the dojo: 5. a command buffer, 7. an arena, each on its own, to watch and to practise.
3.2's systems change values, never structure, so bullets whose time ran out stay in the world. The obvious fix is to despawn them from inside the system. It's wrong, and seeing exactly why is most of this chapter.
Step 1: the bug
expire_now_system: proc(it: *kayan_iter) -> void = {
t: *kayan_table in level = it*.table;
for (row: u64 = 0; row < t*.count; row += 1) {
ls: []lifetime = kayan_slice<lifetime>(t);
es: []kayan_entity = kayan_entities(t);
if (ls[row].seconds <= 0.0) {
_ = kayan_despawn(it*.world, es[row]);
}
}
}
Four bullets, three of them out of time:
1. an expire system that despawns as it goes
lifetimes: 0:1 0.00 1:1 1.00 2:1 0.00 3:1 0.00
after one frame:
lifetimes: 3:1 0.00 1:1 1.00
kayan_verify: 0 problems. The world is fine; the loop missed one.
Row by row:
row 0: 0:1 is out of time. Despawn: the last row, 3:1, moves into row 0.
row 1: 1:1 has time left.
row 2: 2:1 is out of time. Despawn: it's the last row, so nothing moves.
row 3: the table has 2 rows now. Stop.
3:1 moved into row 0, which the loop had already passed, so it was never looked at. The world is perfectly consistent (kayan_verify agrees). The loop is what's wrong: a despawn swap-removes from the table the loop is walking.
The checker lets this through, and it isn't wrong to. The slices are taken again each time round, every read is in bounds, and nothing touches bad memory. A missed row is a logic bug, and that's the kind of bug a loop over indices leaves you with. No memory checker is coming to help here. The fix has to be a design.
Walking backwards fixes this case (lesson 2 does it), because the row that moves in has already been looked at. But it doesn't fix the general problem:
- Adding or removing a component moves the entity to another table. If the query matches that table too and visits it later, the entity is processed twice.
- That move adds a row, and if the destination table is full, its columns grow and move. A slice into that table is then looking at the old block, which the region still owns, so nothing crashes and every write through it is lost. This is 2.5's bug, from inside a loop. A slice is a borrow, and the checker ends a borrow when it can see the world changing beneath it. Inside a system it can't always: the table and the world both arrive through
it, and it has no way to connect them. So the region keeps this safe, and something else has to keep it right. - One system's despawn can land in a table another system is halfway through, if a system despawns entities it found through a second query.
Step 2: record now, apply later
So a system doesn't change structure. It records a command saying what should change, and the world applies every command after the frame's last system:
during the frame values change; tables, rows and pointers do not
after the frame spawns, despawns, adds and removes, in order
kayan_command_kind: enum = {
spawn,
despawn,
add,
remove,
}
kayan_command: struct = {
kind: kayan_command_kind;
entity: kayan_entity;
spawn: u32;
id: u32;
signature: u64;
value: []u8 in frame;
}
A command names the entity, and for an add or remove the component id. An add carries its value as bytes. The bytes, and the list of commands, are only needed until the end of the frame, so they get memory that ends with the frame. That's a second region:
level: region;
frame: region in level;
in level says frame opens and ends inside level. So memory in frame may point into level, since level will still be there. The other way round is refused, and that's what makes this shape safe. Try to keep a command in the world, and it doesn't compile:
type error: this points into region 'frame', which may end before the memory in region 'level' that holds it
So no command can outlive its frame, by construction, and no one has to remember to check. The buffer is the commands and the arena their values are copied into:
kayan_commands: struct = {
heap: *kayan_heap in frame;
list: *vec<kayan_command> in frame;
}
kayan_commands_make: proc(r: region frame) -> *kayan_commands in frame = {
return new<kayan_commands>(r, { .heap = kayan_heap_make(r), .list = vec<kayan_command>make(r, 16) });
}
and a frame opens the region, makes the buffer, runs, and ends:
run_frame: proc(w: *kayan_world in level, systems: *const vec<*kayan_scheduled in level>, dt: f32) -> void = {
frame: region = {
kayan_progress(w, systems, dt, kayan_commands_make(frame));
}
}
There's no line that drops the commands. The end of that block is it, whatever they were. A loop variable can't be called frame any more, either, now that it names a region, so these chapters count frame_no.
Systems reach the buffer through their iterator, which gains two fields:
kayan_iter: struct = {
world: *kayan_world in level;
table: *kayan_table in level;
dt: f32;
commands: *kayan_commands in frame;
ctx: option<*kayan_query in level>;
}
Step 3: recording
The typed procs a system calls copy the value into the frame's arena and append a command:
kayan_defer_value: proc<T>(h: *kayan_heap in frame, value: T) -> []u8 in frame = {
unsafe {
data: ^u8 = cast(memops_arena_push(h*.arena[0]&, sizeof(T), alignof(T)), ^u8);
memcpy(data, value&, sizeof(T));
return cast(data[0..sizeof(T)], []u8 in frame);
}
}
That's the command buffer's one trusted proc, because a value's bytes are a cast. Its signature says what matters: the bytes are in frame, so they get the same protection as the command that holds them.
kayan_defer_add: proc<T>(cmds: *kayan_commands, e: kayan_entity, value: T) -> void = {
_ = kayan_defer(cmds, { .kind = kayan_command_kind.add, .entity = e, .id = kayan_id_of<T>(),
.value = kayan_defer_value<T>(cmds*.heap, value) });
}
The fixed expire system is the broken one with one word changed:
expire_system: proc(it: *kayan_iter) -> void = {
ls: []lifetime = kayan_slice<lifetime>(it*.table);
es: []kayan_entity = kayan_entities(it*.table);
for (row: u64 = 0; row < es.length; row += 1) {
if (ls[row].seconds <= 0.0) {
kayan_defer_despawn(it*.commands, es[row]);
}
}
}
Now the table doesn't change while the loop runs, so the loop can take the entities as a slice up front, like any other column.
Step 4: applying, by id
Commands hold bytes and ids, not types. So applying an add needs 2.6's add by id:
kayan_add_id: proc(w: *kayan_world, e: kayan_entity, id: u32, value: []const u8) -> kayan_status = {
if (!kayan_alive(w, e)) {
return kayan_status.dead;
}
from: u32 = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.table;
if (!kayan_table_has(kayan_table_at(w, from), id)) {
kayan_move(w, e, kayan_table_with(w, from, id));
}
bytes: []u8 = kayan_get_bytes(w, e, id);
if (value.length != bytes.length) {
printfmt("kayan: a {}-byte value for a component of {} bytes\n", value.length, bytes.length);
exit(1);
}
for (i: u64 = 0; i < bytes.length; i += 1) {
bytes[i] = value[i];
}
return kayan_status.ok;
}
It's the same move-then-write as before, with the write a checked loop over the value's bytes. Once that exists, the typed kayan_remove<T> from 2.6 shrinks to a one-line wrapper. That's the pattern from the start of the course: byte fiddling written once, a type-safe generic layer on top.
kayan_remove: proc<T>(w: *kayan_world, e: kayan_entity) -> kayan_status = {
return kayan_remove_id(w, e, kayan_id_of<T>());
}
kayan_add<T> keeps its 2.6 body instead of calling this one. To go through kayan_add_id it would have to turn its T into bytes, and that's a cast, which would make it another trusted proc. It moves the entity and calls kayan_set<T> instead, and stays checked. The boundary between typed and untyped code is somewhere you want as few crossings as possible.
kayan_flush walks the list in order and applies each command. It's called at the end of kayan_progress:
kayan_flush(w, cmds);
}
Step 5: commands that arrive too late
By the time a command is applied, its entity may be gone. The usual case is two systems both asking to despawn the same thing. Applying in order handles that for free: the first despawn works, and the second finds a dead handle. Part 1's generations make that check one comparison:
if (status == kayan_status.ok) {
w*.applied += 1;
} else {
w*.skipped += 1;
}
Step 6: spawning from a system
A turret firing is a spawn in the middle of a frame. The new entity doesn't exist until the commands are applied, so there's no handle to give its components to. kayan_defer_spawn returns a pending spawn instead: the spawn command's place in the list. Components are given to that:
shot: kayan_pending = kayan_defer_spawn(it*.commands, bullet_signature());
kayan_pending_set<position>(it*.commands, shot, ps[row]);
kayan_pending_set<velocity>(it*.commands, shot, { .per_second = { .x = 8.0, .y = 0.0 } });
kayan_pending_set<damage>(it*.commands, shot, { .amount = 25.0 });
kayan_pending_set<lifetime>(it*.commands, shot, { .seconds = 2.0 });
When the flush reaches the spawn, it makes the entity and writes the handle into the command. The adds after it look the handle up there. The bullet is spawned with its full signature, so the adds land in the table it's already in: no moves. And it doesn't move until the next frame, because no system ran over it this one.
The flush allocates in level, deep inside a frame, with no level in sight. This is why 3.1 had the world remember its region.
Step 7: a system with two queries
A bullet that reaches a target damages it. The hit system walks the bullets (its own query) and, for each, the targets (a second query). It reaches the second query through a new field, ctx, which kayan passes through untouched:
targets: option<*kayan_query in level> = it*.ctx;
if (!targets) {
return;
}
ctx is a typed pointer rather than a *void, because the checker has to know what it points at. A system with no second query gets null, so the ones that use it test it first.
The damage is a value, so it's written now. The bullet's despawn is structure, so it's recorded. Walking a second query in the middle of a frame is safe because of commands: no table is made and no column moves until the frame is over. The slices in this system stay true for the whole frame, and that's the guarantee no checker gave us: the design did.
Run it
2. an expire system that records despawns
lifetimes: 0:1 0.00 1:1 1.00 2:1 0.00 3:1 0.00
despawn 0:1
despawn 2:1
despawn 3:1
after one frame:
lifetimes: 1:1 1.00
All three. Then the turret, with two targets 0.25 apart in its line of fire and a third out of it:
3. a turret, two targets in its line of fire, one out of it
frame 1
spawn {position velocity damage lifetime} -> 4:1
frame 2
frame 3
spawn {position velocity damage lifetime} -> 5:1
frame 4
despawn 4:1
despawn 4:1: skipped, dead
frame 5
spawn {position velocity damage lifetime} -> 4:2
...
health: 1:1 50, 2:1 50, 3:1 100
commands applied: 17, skipped: 2
In frame 4, bullet 4:1 is close enough to both targets. Both lose 25, and both hits record a despawn. The first applies; the second finds 4:1 already dead and is skipped. In frame 5 the next bullet gets number 4 back with generation 2: Part 1, working.
17 commands, for three bullets and two despawns: each spawn is five, the spawn and its four components.
Try
- In
expire_now_system, takelsandesonce, before the loop, and loop tols.lengthinstead oft*.count. It still compiles, and now all three get despawned. Work out why. (Hint: what's in the rows past the end of the table?) Is that loop correct? - Write a system that adds
burningdirectly (not deferred) to every target it walks. Count how many times it visits each entity. - Flush after each system instead of after the frame. You'll have to empty the list after each flush, with
vec<kayan_command>clear(cmds*.list), since it's no longer the region's end that lets the commands go. What does that change about what later systems see? (flecs calls these sync points.) - Add a field
kept: option<*kayan_commands in frame>;tokayan_world, and compile. Which rule did you just run into, and what bug is it saving you from?
1// 3.3 Commands2//3// 3.2's systems change values, never structure, so the bullets whose time ran4// out are still in the world. Despawning them from inside the system is the5// obvious fix, and it is wrong: the system is walking the very table a6// despawn swap-removes from, so the last row moves into a row the walk has7// already passed, and is never looked at. Adding or removing a component is8// worse: the entity moves to another table the query may visit later, so the9// walk sees it twice.10//11// So a system does not change structure. It records the change, a command,12// and the world applies every command after the frame's last system:13//14// during the frame values change; tables, rows and pointers do not15// after the frame spawns, despawns, adds and removes, in order16//17// The second half of this file is a command buffer: the record, the procs18// systems call to fill it, and the flush that empties it.19//20// The buffer lives in a region of its own, `frame`, opened21// around each frame and cleared at its end: the commands and the values they22// carry are frame memory, the world is level memory, and the checker keeps23// the two apart. A pointer into frame cannot be kept in the world, so no24// command can outlive the frame that recorded it.25 26import "std/print.rin"27import "std/vec.rin"28import "std/hashmap.rin"29import "std/slice.rin"30 31level: region;32frame: region in level;33 34vec2: struct = {35 x: f32;36 y: f32;37}38 39position: struct = { at: vec2; }40velocity: struct = { per_second: vec2; }41health: struct = { points: f32; max: f32; }42damage: struct = { amount: f32; }43lifetime: struct = { seconds: f32; }44// A field tagged "hidden" is bookkeeping the inspector does not show.45turret: struct = { cooldown: f32; timer: f32 @ "hidden"; }46burning: struct = { per_second: f32; }47 48component: enum = {49 position,50 velocity,51 health,52 damage,53 lifetime,54 turret,55 burning,56}57 58component_of: proc<position>() -> component = { return component.position; }59component_of: proc<velocity>() -> component = { return component.velocity; }60component_of: proc<health>() -> component = { return component.health; }61component_of: proc<damage>() -> component = { return component.damage; }62component_of: proc<lifetime>() -> component = { return component.lifetime; }63component_of: proc<turret>() -> component = { return component.turret; }64component_of: proc<burning>() -> component = { return component.burning; }65 66// ---------------------------------------------------------------------------67// Entities (Part 1)68// ---------------------------------------------------------------------------69 70kayan_entity: struct = {71 index: u32;72 generation: u32;73}74 75// `missing`: the entity is alive but has no such component.76kayan_status: enum[must_use] = {77 ok,78 dead,79 missing,80}81 82// Where an entity lives: which table, and which row of it.83kayan_slot: struct = {84 generation: u32;85 table: u32;86 row: u32;87}88 89// A structural change recorded during a frame and applied after it (3.3).90// `entity` is who it applies to; a spawn's is filled in when it is applied. An91// add to an entity spawned in the same frame has no entity to name yet, so it92// names its spawn instead: `spawn` is that command's index plus one. The93// value's bytes are frame memory, like the command itself.94kayan_command_kind: enum = {95 spawn,96 despawn,97 add,98 remove,99}100 101kayan_command: struct = {102 kind: kayan_command_kind;103 entity: kayan_entity;104 spawn: u32;105 id: u32;106 signature: u64;107 value: []u8 in frame;108}109 110// ---------------------------------------------------------------------------111// The byte core: columns (2.2), and the region they grow into112// ---------------------------------------------------------------------------113 114// The region's arena, kept where the trusted procs can reach it: the world115// remembers where it was made, as a vec does, so nothing above this layer116// hands an arena around.117kayan_heap: struct = {118 arena: ^memops_arena;119}120 121kayan_heap_make: proc(r: region R) -> *kayan_heap in R = {122 unsafe {123 h: *kayan_heap in R = cast(memops_arena_push_zero(r, sizeof(kayan_heap), alignof(kayan_heap)), *kayan_heap in R);124 h*.arena = r;125 return h;126 }127}128 129// One component's values for every row of a table, as bytes: `size` of them130// a row, `capacity` rows allocated (the table knows how many are in use).131kayan_column: struct = {132 data: []u8 in level;133 size: u64;134 align: u64;135}136 137kayan_column_make: proc(h: *kayan_heap in level, info: *const reflect in permanent) -> *kayan_column in level = {138 unsafe {139 c: *kayan_column in level = cast(memops_arena_push_zero(h*.arena[0]&, sizeof(kayan_column), alignof(kayan_column)), *kayan_column in level);140 c*.size = info*.size;141 c*.align = info*.align;142 return c;143 }144}145 146// Room for `capacity` rows, the first `count` copied over. The old storage is147// left in the region: a slice taken before the grow still points at valid148// memory, and stale values.149kayan_column_grow: proc(h: *kayan_heap, c: *kayan_column, count: u64, capacity: u64) -> void = {150 unsafe {151 raw: ^u8 = cast(memops_arena_push_zero(h*.arena[0]&, c*.size * capacity, c*.align), ^u8);152 bytes: []u8 in level = cast(raw[0..c*.size * capacity], []u8 in level);153 // Slicing checks both ends: a count past either block stops the program.154 kept: u64 = c*.size * count;155 slice<u8>copy(bytes[0..kept], c*.data[0..kept]);156 c*.data = bytes;157 }158}159 160// One row's bytes, bounds-checked like any slice.161kayan_column_bytes: proc(c: *kayan_column, row: u64) -> []u8 in level = {162 return c*.data[row * c*.size..(row + 1) * c*.size];163}164 165// The typed view: the column's first `count` rows as `[]T`. This is the one166// place bytes become a type, so it takes nothing on trust. T must be the167// column's size, and plain data: any bytes are then a valid T, so reading the168// wrong type gives wrong numbers, never a bad pointer. The rows must fit, which169// the slice knows and no field can lie about, and the bytes must be aligned for170// T. The view is `[]T in level`: the bytes are in the region, so the view is171// good memory for as long as the region is. After the column grows it is the172// old block, still readable, no longer the column: take the view again.173kayan_column_slice: proc<T>(c: *const kayan_column, count: u64) -> []T in level = {174 if (c*.size != sizeof(T)) {175 printfmt("kayan: a column of {}-byte rows read as a {}-byte type\n", c*.size, sizeof(T));176 exit(1);177 }178 if (!reflect_is_plain(T<>&)) {179 printfmt("kayan: a column holds plain data, and this {}-byte type is not\n", sizeof(T));180 exit(1);181 }182 if (count > c*.data.length / sizeof(T)) {183 printfmt("kayan: a view of {} rows on a column with room for {}\n", count, c*.data.length / sizeof(T));184 exit(1);185 }186 unsafe {187 if (cast(c*.data.data, usize) % alignof(T) != 0) {188 printfmt("kayan: a column's bytes are not aligned to {}\n", alignof(T));189 exit(1);190 }191 raw: ^T = cast(c*.data.data, ^T);192 return cast(raw[0..count], []T in level);193 }194}195 196// Copying and zeroing a row are loops over the row's bytes: checked code.197kayan_column_copy_row: proc(c: *kayan_column, to: u64, from: u64) -> void = {198 dst: []u8 = kayan_column_bytes(c, to);199 src: []u8 = kayan_column_bytes(c, from);200 for (i: u64 = 0; i < dst.length; i += 1) {201 dst[i] = src[i];202 }203}204 205kayan_column_zero_row: proc(c: *kayan_column, row: u64) -> void = {206 bytes: []u8 = kayan_column_bytes(c, row);207 for (i: u64 = 0; i < bytes.length; i += 1) {208 bytes[i] = 0;209 }210}211 212// ---------------------------------------------------------------------------213// The table214// ---------------------------------------------------------------------------215 216// `columns[id]` is a column when the signature has bit `id`, and null217// otherwise. Count and capacity are the table's, shared by its columns, so218// they can never disagree; `entities` grows on its own and always has `count`219// rows.220//221// The edges: `add_edges[id]` is the table this one's entities move to when222// they gain component `id`, and `remove_edges[id]` when they lose it. Each is223// the table's index plus one, so 0, which a new table starts with, means "not224// known yet".225kayan_table: struct = {226 signature: u64;227 count: u64;228 capacity: u64;229 entities: *vec<kayan_entity> in level;230 columns: [64]option<*kayan_column in level>;231 add_edges: [64]u32;232 remove_edges: [64]u32;233}234 235// A zeroed table in the region, with an empty entities vec: the world fills236// in the signature and the columns.237kayan_table_new: proc(h: *kayan_heap in level) -> *kayan_table in level = {238 unsafe {239 t: *kayan_table in level = cast(memops_arena_push_zero(h*.arena[0]&, sizeof(kayan_table), alignof(kayan_table)), *kayan_table in level);240 v: *vec<kayan_entity> in level = cast(memops_arena_push_zero(h*.arena[0]&, sizeof(vec<kayan_entity>), alignof(vec<kayan_entity>)), *vec<kayan_entity> in level);241 v*.arena = h*.arena;242 t*.entities = v;243 return t;244 }245}246 247// The world: the registry, the slots and free list, and every table, with a248// map from signature to table index so finding one is a lookup, not a search.249// Tables are referred to by index, so a table can be named in a slot; each250// table is its own allocation, so a pointer to one stays good when the next251// is made.252kayan_world: struct = {253 heap: *kayan_heap in level;254 infos: [64]option<*const reflect in permanent>;255 component_count: u32;256 tables: *vec<*kayan_table in level> in level;257 by_signature: *hashmap<u64, u32> in level;258 slots: *vec<kayan_slot> in level;259 free: *vec<u32> in level;260 live: u64;261 // What the flushes did, and whether to print each command as it is applied.262 applied: u64;263 skipped: u64;264 trace: b32;265}266 267kayan_bit: proc(id: u32) -> u64 = {268 return 1'u64 shl cast(id, u64);269}270 271kayan_id_of: proc<T>() -> u32 = {272 return cast(component_of<T>(), u32);273}274 275kayan_mask_of: proc<T>() -> u64 = {276 return kayan_bit(kayan_id_of<T>());277}278 279// The reflect records live for the whole program, so `permanent` is their280// region and the world may keep them.281kayan_register: proc(w: *kayan_world, id: u32, info: *const reflect in permanent) -> void = {282 if (id >= 64) {283 printfmt("kayan: component id {} is past the 64 a signature can hold\n", id);284 exit(1);285 }286 w*.infos[id] = info;287 if (id + 1 > w*.component_count) {288 w*.component_count = id + 1;289 }290}291 292// A registered component's record. The checker needs the null test; the293// return after exit is never reached.294kayan_info: proc(w: *const kayan_world, id: u32) -> *const reflect in permanent = {295 info: option<*const reflect in permanent> = w*.infos[id];296 if (info) {297 return info;298 }299 printfmt("kayan: component {} was never registered\n", id);300 exit(1);301 return kayan_entity<>&;302}303 304kayan_table_has: proc(t: *const kayan_table, id: u32) -> b32 = {305 return (t*.signature & kayan_bit(id)) != 0;306}307 308// A table for a signature: one empty column per component in it, sized from309// the registry. Every component in the signature must be registered.310kayan_table_make: proc(w: *kayan_world, signature: u64) -> *kayan_table in level = {311 t: *kayan_table in level = kayan_table_new(w*.heap);312 t*.signature = signature;313 for (id: u32 = 0; id < 64; id += 1) {314 if ((signature & kayan_bit(id)) != 0) {315 if (id >= w*.component_count) {316 printfmt("kayan: signature {} uses component {}, which was never registered\n", signature, id);317 exit(1);318 }319 t*.columns[id] = kayan_column_make(w*.heap, kayan_info(w, id));320 }321 }322 return t;323}324 325// Double the room in every column at once.326kayan_table_grow: proc(w: *kayan_world, t: *kayan_table) -> void = {327 capacity: u64 = t*.capacity == 0 ? 4 : t*.capacity * 2;328 for (id: u32 = 0; id < w*.component_count; id += 1) {329 c: option<*kayan_column in level> = t*.columns[id];330 if (c) {331 kayan_column_grow(w*.heap, c, t*.count, capacity);332 }333 }334 t*.capacity = capacity;335}336 337// A new row at the end, owned by `e`, with every component zeroed.338kayan_table_add_row: proc(w: *kayan_world, t: *kayan_table, e: kayan_entity) -> u64 = {339 if (t*.count == t*.capacity) {340 kayan_table_grow(w, t);341 }342 row: u64 = t*.count;343 t*.count += 1;344 _ = vec<kayan_entity>push(t*.entities, e);345 for (id: u32 = 0; id < w*.component_count; id += 1) {346 c: option<*kayan_column in level> = t*.columns[id];347 if (c) {348 kayan_column_zero_row(c, row);349 }350 }351 return row;352}353 354// Who owns each of the table's rows. It only reads, and the checker sees that,355// so a system holds this and two or three columns of one table at once.356kayan_entities: proc(t: *const kayan_table) -> []kayan_entity in level = {357 return vec<kayan_entity>items(t*.entities);358}359 360// 1.4's swap-remove, for every column: the last row is copied over `row`, and361// the slot of the entity that owned the last row is told where it went.362kayan_table_remove_row: proc(w: *kayan_world, t: *kayan_table, row: u64) -> void = {363 last: u64 = t*.count - 1;364 if (row != last) {365 for (id: u32 = 0; id < w*.component_count; id += 1) {366 c: option<*kayan_column in level> = t*.columns[id];367 if (c) {368 kayan_column_copy_row(c, row, last);369 }370 }371 moved: kayan_entity = vec<kayan_entity>at(t*.entities, last)*;372 vec<kayan_entity>at(t*.entities, row)* = moved;373 vec<kayan_slot>at(w*.slots, cast(moved.index, u64))*.row = cast(row, u32);374 }375 _ = vec<kayan_entity>pop(t*.entities);376 t*.count = last;377}378 379// The typed view of one of the table's columns: T's rows, all of them in use.380// Asking for a component the table does not have is a bug, so it stops. The381// slice is good memory for as long as the region, but when the table grows it is382// the old block, no longer the column: take it again, and 3.3 makes that a rule.383kayan_slice: proc<T>(t: *const kayan_table) -> []T in level = {384 id: u32 = kayan_id_of<T>();385 c: option<*kayan_column in level> = t*.columns[id];386 if (!c) {387 printfmt("kayan: the table has no column for component {}\n", id);388 exit(1);389 }390 return kayan_column_slice<T>(c, t*.count);391}392 393kayan_table_at: proc(w: *const kayan_world, index: u32) -> *kayan_table in level = {394 return vec<*kayan_table in level>at(w*.tables, cast(index, u64))*;395}396 397// The table for a signature: found if it exists, made if it does not. This is398// where archetypes come from.399kayan_table_for: proc(w: *kayan_world, signature: u64) -> u32 = {400 found: option<*u32> = hashmap<u64, u32>get(w*.by_signature, signature);401 if (found) {402 return found*;403 }404 index: u32 = cast(w*.tables*.length, u32);405 _ = vec<*kayan_table in level>push(w*.tables, kayan_table_make(w, signature));406 _ = hashmap<u64, u32>set(w*.by_signature, signature, index);407 return index;408}409 410// ---------------------------------------------------------------------------411// The world412// ---------------------------------------------------------------------------413 414kayan_world_make: proc(r: region level) -> *kayan_world in level = {415 return new<kayan_world>(r, {416 .heap = kayan_heap_make(r),417 .tables = vec<*kayan_table in level>make(r, 8),418 .by_signature = hashmap<u64, u32>make(r),419 .slots = vec<kayan_slot>make(r, 64),420 .free = vec<u32>make(r, 16),421 });422}423 424kayan_alive: proc(w: *const kayan_world, e: kayan_entity) -> b32 = {425 if (cast(e.index, u64) >= w*.slots*.length) {426 return false;427 }428 return vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.generation == e.generation;429}430 431// A new entity with the components in `signature`, all zeroed, in the table432// for that signature.433kayan_spawn: proc(w: *kayan_world, signature: u64) -> kayan_entity = {434 e: kayan_entity = {};435 if (w*.free*.length > 0) {436 e.index = vec<u32>pop(w*.free);437 } else {438 e.index = cast(w*.slots*.length, u32);439 _ = vec<kayan_slot>push(w*.slots, { .generation = 1 });440 }441 e.generation = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.generation;442 table_index: u32 = kayan_table_for(w, signature);443 t: *kayan_table in level = kayan_table_at(w, table_index);444 row: u64 = kayan_table_add_row(w, t, e);445 // The slot is taken after the calls that change the world: a pointer into446 // `slots` from before them may be stale, and the checker says so.447 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));448 slot*.table = table_index;449 slot*.row = cast(row, u32);450 w*.live += 1;451 return e;452}453 454kayan_despawn: proc(w: *kayan_world, e: kayan_entity) -> kayan_status = {455 if (!kayan_alive(w, e)) {456 return kayan_status.dead;457 }458 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));459 table: u32 = slot*.table;460 row: u32 = slot*.row;461 kayan_table_remove_row(w, kayan_table_at(w, table), cast(row, u64));462 slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));463 slot*.generation += 1;464 _ = vec<u32>push(w*.free, e.index);465 w*.live -= 1;466 return kayan_status.ok;467}468 469// The components a live entity has: its table's signature. 0 for a dead one.470kayan_signature: proc(w: *const kayan_world, e: kayan_entity) -> u64 = {471 if (!kayan_alive(w, e)) {472 return 0;473 }474 return kayan_table_at(w, vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.table)*.signature;475}476 477// ---------------------------------------------------------------------------478// Moving between tables479// ---------------------------------------------------------------------------480 481// The table an entity of table `from` moves to when it gains component `id`.482// The first time, it is found by signature, which can make the table; the483// answer is written on both tables, as from's add edge and the destination's484// remove edge, so neither direction is looked up again.485kayan_table_with: proc(w: *kayan_world, from: u32, id: u32) -> u32 = {486 edge: u32 = kayan_table_at(w, from)*.add_edges[id];487 if (edge != 0) {488 return edge - 1;489 }490 to: u32 = kayan_table_for(w, kayan_table_at(w, from)*.signature | kayan_bit(id));491 kayan_table_at(w, from)*.add_edges[id] = to + 1;492 kayan_table_at(w, to)*.remove_edges[id] = from + 1;493 return to;494}495 496// The same for losing component `id`.497kayan_table_without: proc(w: *kayan_world, from: u32, id: u32) -> u32 = {498 edge: u32 = kayan_table_at(w, from)*.remove_edges[id];499 if (edge != 0) {500 return edge - 1;501 }502 to: u32 = kayan_table_for(w, kayan_table_at(w, from)*.signature & ~kayan_bit(id));503 kayan_table_at(w, from)*.remove_edges[id] = to + 1;504 kayan_table_at(w, to)*.add_edges[id] = from + 1;505 return to;506}507 508// Move a live entity to table `to_index`. Components both tables have are509// copied, byte for byte; components only the new table has start zeroed;510// components only the old table has are dropped with its old row.511kayan_move: proc(w: *kayan_world, e: kayan_entity, to_index: u32) -> void = {512 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));513 from_index: u32 = slot*.table;514 if (from_index == to_index) {515 return;516 }517 from_row: u64 = cast(slot*.row, u64);518 from: *kayan_table in level = kayan_table_at(w, from_index);519 to: *kayan_table in level = kayan_table_at(w, to_index);520 to_row: u64 = kayan_table_add_row(w, to, e);521 for (id: u32 = 0; id < w*.component_count; id += 1) {522 fc: option<*kayan_column in level> = from*.columns[id];523 tc: option<*kayan_column in level> = to*.columns[id];524 if (fc && tc) {525 dst: []u8 = kayan_column_bytes(tc, to_row);526 src: []u8 = kayan_column_bytes(fc, from_row);527 for (i: u64 = 0; i < dst.length; i += 1) {528 dst[i] = src[i];529 }530 }531 }532 kayan_table_remove_row(w, from, from_row);533 slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));534 slot*.table = to_index;535 slot*.row = cast(to_row, u32);536}537 538// ---------------------------------------------------------------------------539// The byte layer, for tools and for commands540// ---------------------------------------------------------------------------541 542// Give e component `id`, with `value` for its bytes: 2.6's add, by id instead543// of by type, because a command holds its value as bytes. The copy is a loop544// over two slices the checker has sized.545kayan_add_id: proc(w: *kayan_world, e: kayan_entity, id: u32, value: []const u8) -> kayan_status = {546 if (!kayan_alive(w, e)) {547 return kayan_status.dead;548 }549 from: u32 = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.table;550 if (!kayan_table_has(kayan_table_at(w, from), id)) {551 kayan_move(w, e, kayan_table_with(w, from, id));552 }553 bytes: []u8 = kayan_get_bytes(w, e, id);554 if (value.length != bytes.length) {555 printfmt("kayan: a {}-byte value for a component of {} bytes\n", value.length, bytes.length);556 exit(1);557 }558 for (i: u64 = 0; i < bytes.length; i += 1) {559 bytes[i] = value[i];560 }561 return kayan_status.ok;562}563 564kayan_remove_id: proc(w: *kayan_world, e: kayan_entity, id: u32) -> kayan_status = {565 if (!kayan_alive(w, e)) {566 return kayan_status.dead;567 }568 from: u32 = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.table;569 if (!kayan_table_has(kayan_table_at(w, from), id)) {570 return kayan_status.missing;571 }572 kayan_move(w, e, kayan_table_without(w, from, id));573 return kayan_status.ok;574}575 576// e's component `id` as its bytes: an empty slice if e is dead or has no such577// component. The typed layer is for game code, which knows its types; this is578// for tools that work from reflect records instead.579kayan_get_bytes: proc(w: *const kayan_world, e: kayan_entity, id: u32) -> []u8 in level = {580 none: []u8 in level = {};581 if (!kayan_alive(w, e) || id >= w*.component_count) {582 return none;583 }584 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));585 table: u32 = slot*.table;586 row: u32 = slot*.row;587 t: *kayan_table in level = kayan_table_at(w, table);588 c: option<*kayan_column in level> = t*.columns[id];589 if (!c) {590 return none;591 }592 return kayan_column_bytes(c, cast(row, u64));593}594 595// ---------------------------------------------------------------------------596// The typed layer597// ---------------------------------------------------------------------------598 599// e's T, or null if e is dead or has no T. Handle, slot, table, row, typed600// view: the same path as 1.4's get_position, for any component. `option<*T>` says the601// null is possible, so the caller tests before it reads. The pointer is good602// until the next structural change, so use it now and keep the handle.603kayan_get: proc<T>(w: *const kayan_world, e: kayan_entity) -> option<*T in level> = {604 if (!kayan_alive(w, e)) {605 return null;606 }607 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));608 table: u32 = slot*.table;609 row: u32 = slot*.row;610 t: *kayan_table in level = kayan_table_at(w, table);611 if (!kayan_table_has(t, kayan_id_of<T>())) {612 return null;613 }614 rows: []T in level = kayan_slice<T>(t);615 return rows[cast(row, u64)]&;616}617 618kayan_has: proc<T>(w: *const kayan_world, e: kayan_entity) -> b32 = {619 return (kayan_signature(w, e) & kayan_mask_of<T>()) != 0;620}621 622// Write e's T. It must already have one: giving an entity a component it does623// not have changes its table, which is 2.6.624kayan_set: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> kayan_status = {625 if (!kayan_alive(w, e)) {626 return kayan_status.dead;627 }628 p: option<*T> = kayan_get<T>(w, e);629 if (!p) {630 return kayan_status.missing;631 }632 p* = value;633 return kayan_status.ok;634}635 636// Give e a T. If it already has one, this is kayan_set; if not, it moves to637// the table with T and then sets it.638kayan_add: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> kayan_status = {639 if (!kayan_alive(w, e)) {640 return kayan_status.dead;641 }642 if (!kayan_has<T>(w, e)) {643 from: u32 = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.table;644 kayan_move(w, e, kayan_table_with(w, from, kayan_id_of<T>()));645 }646 return kayan_set<T>(w, e, value);647}648 649// Take e's T away: the byte proc, with the type turned into an id.650kayan_remove: proc<T>(w: *kayan_world, e: kayan_entity) -> kayan_status = {651 return kayan_remove_id(w, e, kayan_id_of<T>());652}653 654// The invariant, over every table: each row belongs to a live entity whose655// slot points back at that table and row, and the rows of all tables add up to656// the live count.657kayan_verify: proc(w: *const kayan_world) -> u64 = {658 problems: u64 = 0;659 rows: u64 = 0;660 for (ti: u64 = 0; ti < w*.tables*.length; ti += 1) {661 t: *kayan_table in level = kayan_table_at(w, cast(ti, u32));662 for (row: u64 = 0; row < t*.count; row += 1) {663 // Taken row by row: kayan_alive takes the world, and the checker664 // does not know that it leaves the rows where they are.665 es: []kayan_entity = kayan_entities(t);666 e: kayan_entity = es[row];667 if (!kayan_alive(w, e)) {668 printfmt("kayan_verify: table {} row {} holds dead entity {}:{}\n", ti, row, e.index, e.generation);669 problems += 1;670 continue;671 }672 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));673 if (cast(slot*.table, u64) != ti || cast(slot*.row, u64) != row) {674 printfmt("kayan_verify: {}:{} is in table {} row {}, its slot says table {} row {}\n",675 e.index, e.generation, ti, row, slot*.table, slot*.row);676 problems += 1;677 }678 }679 rows += t*.count;680 }681 if (rows != w*.live) {682 printfmt("kayan_verify: {} rows in tables, {} live entities\n", rows, w*.live);683 problems += 1;684 }685 return problems;686}687 688kayan_print_signature: proc(w: *const kayan_world, signature: u64) -> void = {689 printfmt("{");690 first: b32 = true;691 for (id: u32 = 0; id < w*.component_count; id += 1) {692 if ((signature & kayan_bit(id)) != 0) {693 if (!first) {694 printfmt(" ");695 }696 printfmt("{}", kayan_info(w, id)*.name);697 first = false;698 }699 }700 printfmt("}");701}702 703kayan_print_table: proc(w: *const kayan_world, ti: u32) -> void = {704 t: *kayan_table in level = kayan_table_at(w, ti);705 printfmt(" table {} ", ti);706 kayan_print_signature(w, t*.signature);707 printfmt(":");708 es: []kayan_entity = kayan_entities(t);709 for (row: u64 = 0; row < es.length; row += 1) {710 e: kayan_entity = es[row];711 printfmt(" {}:{}", e.index, e.generation);712 }713 printfmt("\n");714}715 716kayan_print_tables: proc(w: *const kayan_world) -> void = {717 for (ti: u32 = 0; ti < cast(w*.tables*.length, u32); ti += 1) {718 kayan_print_table(w, ti);719 }720}721 722// ---------------------------------------------------------------------------723// The inspector: reflect records, no component types724// ---------------------------------------------------------------------------725 726kayan_indent: proc(depth: u32) -> void = {727 for (i: u32 = 0; i < depth; i += 1) {728 printfmt(" ");729 }730}731 732// One value of a builtin type, read from its bytes by its type's name. The733// bytes may sit at any address, so the read is a copy into a local: this is the734// inspector's one unsafe block.735kayan_print_builtin: proc(type_name: ^const char, at: []const u8) -> void = {736 if (at.length < 4) {737 printfmt("<{} bytes>", at.length);738 return;739 }740 unsafe {741 if (reflect_cstr_equal(type_name, "f32")) {742 f: f32 = 0.0;743 memcpy(f&, at.data, 4);744 printfmt("{:.2}", cast(f, f64));745 } else if (reflect_cstr_equal(type_name, "u32")) {746 u: u32 = 0;747 memcpy(u&, at.data, 4);748 printfmt("{}", u);749 } else if (reflect_cstr_equal(type_name, "i32")) {750 i: i32 = 0;751 memcpy(i&, at.data, 4);752 printfmt("{}", i);753 } else {754 printfmt("<{}>", type_name);755 }756 }757}758 759// Every field of a struct whose bytes are `base`, nested structs indented760// under their field. A field's bytes are cut from `base` by its offset and761// size, both from the reflect record.762kayan_print_fields: proc(type: *const reflect, base: []const u8, depth: u32) -> void = {763 for (i: u64 = 0; i < type*.count; i += 1) {764 f: option<*const reflect_field> = reflect_field_at(type, i);765 if (!f) {766 continue;767 }768 if (reflect_field_has_attr(f, "hidden")) {769 continue;770 }771 at: []const u8 = base[f*.offset..f*.offset + f*.size];772 kayan_indent(depth);773 printfmt("{}:", f*.name);774 info: option<*const reflect> = reflect_field_info(f);775 if (info && info*.kind == reflect_kind_struct) {776 printfmt("\n");777 kayan_print_fields(info, at, depth + 1);778 } else {779 printfmt(" ");780 kayan_print_builtin(f*.type, at);781 printfmt("\n");782 }783 }784}785 786kayan_inspect: proc(w: *const kayan_world, e: kayan_entity) -> void = {787 if (!kayan_alive(w, e)) {788 printfmt("{}:{} (dead)\n", e.index, e.generation);789 return;790 }791 printfmt("{}:{} ", e.index, e.generation);792 signature: u64 = kayan_signature(w, e);793 kayan_print_signature(w, signature);794 printfmt("\n");795 for (id: u32 = 0; id < w*.component_count; id += 1) {796 if ((signature & kayan_bit(id)) != 0) {797 info: *const reflect in permanent = kayan_info(w, id);798 printfmt(" {}\n", info*.name);799 bytes: []u8 = kayan_get_bytes(w, e, id);800 kayan_print_fields(info, bytes, 2);801 }802 }803}804 805// Every table: rows in use out of room, and per column its row size and its806// bytes in use out of bytes allocated. The memory a table holds and does not807// use is the cost of doubling on growth.808kayan_print_layout: proc(w: *const kayan_world) -> void = {809 for (ti: u32 = 0; ti < cast(w*.tables*.length, u32); ti += 1) {810 t: *kayan_table in level = kayan_table_at(w, ti);811 printfmt("table {} ", ti);812 kayan_print_signature(w, t*.signature);813 printfmt(": {} rows, room for {}\n", cast(t*.count, u32), cast(t*.capacity, u32));814 for (id: u32 = 0; id < w*.component_count; id += 1) {815 c: option<*kayan_column in level> = t*.columns[id];816 if (c) {817 size: u64 = c*.size;818 printfmt(" {:-9} {:2} bytes a row, {:4} of {:4} bytes in use\n", kayan_info(w, id)*.name, cast(size, u32),819 cast(size * t*.count, u32), cast(size * t*.capacity, u32));820 }821 }822 }823}824 825// ---------------------------------------------------------------------------826// Queries827// ---------------------------------------------------------------------------828 829// The tables a system runs over: every table whose signature has all of `all`830// and none of `none`, kept by index. Tables are only ever added, at the end of831// w.tables, so the query remembers how many it has checked and, brought up to832// date, looks at the new ones only.833kayan_query: struct = {834 all: u64;835 none: u64;836 tables: *vec<u32> in level;837 checked: u64;838}839 840// The list of matched tables lives in the world's region, so a query says841// which region it is made in.842kayan_query_init: proc(r: region level, all: u64, none: u64) -> kayan_query = {843 return { .all = all, .none = none, .tables = vec<u32>make(r, 4) };844}845 846kayan_matches: proc(signature: u64, all: u64, none: u64) -> b32 = {847 return (signature & all) == all && (signature & none) == 0;848}849 850// Check the tables made since the query last looked. With none new, this is851// one comparison, so call it before every walk.852kayan_query_update: proc(w: *const kayan_world, q: *kayan_query) -> void = {853 for (ti: u64 = q*.checked; ti < w*.tables*.length; ti += 1) {854 if (kayan_matches(kayan_table_at(w, cast(ti, u32))*.signature, q*.all, q*.none)) {855 _ = vec<u32>push(q*.tables, cast(ti, u32));856 }857 }858 q*.checked = w*.tables*.length;859}860 861// The query's i-th table.862kayan_query_table: proc(w: *const kayan_world, q: *const kayan_query, i: u64) -> *kayan_table in level = {863 return kayan_table_at(w, vec<u32>at(q*.tables, i)*);864}865 866// How many entities the query matches: the rows of its tables.867kayan_query_count: proc(w: *const kayan_world, q: *kayan_query) -> u64 = {868 kayan_query_update(w, q);869 n: u64 = 0;870 for (i: u64 = 0; i < q*.tables*.length; i += 1) {871 n += kayan_query_table(w, q, i)*.count;872 }873 return n;874}875 876kayan_print_query: proc(w: *const kayan_world, name: ^const char, q: *kayan_query) -> void = {877 kayan_query_update(w, q);878 printfmt("{}: all ", name);879 kayan_print_signature(w, q*.all);880 printfmt(", none ");881 kayan_print_signature(w, q*.none);882 printfmt("; tables: {}, entities: {}\n", q*.tables*.length, kayan_query_count(w, q));883 for (i: u64 = 0; i < q*.tables*.length; i += 1) {884 kayan_print_table(w, vec<u32>at(q*.tables, i)*);885 }886}887 888// ---------------------------------------------------------------------------889// Systems890// ---------------------------------------------------------------------------891 892// ---------------------------------------------------------------------------893// Commands894// ---------------------------------------------------------------------------895 896// The frame's command buffer: the commands, and the frame arena their values897// are copied into. Made at the start of each frame, in the frame region, and898// gone with it.899kayan_commands: struct = {900 heap: *kayan_heap in frame;901 list: *vec<kayan_command> in frame;902}903 904kayan_commands_make: proc(r: region frame) -> *kayan_commands in frame = {905 return new<kayan_commands>(r, { .heap = kayan_heap_make(r), .list = vec<kayan_command>make(r, 16) });906}907 908// A spawn that has been recorded and not applied. There is no entity yet, so909// this is the spawn command's place in the list; give it components with910// kayan_pending_set.911kayan_pending: struct = {912 command: u32;913}914 915kayan_defer: proc(cmds: *kayan_commands, c: kayan_command) -> u32 = {916 index: u32 = cast(cmds*.list*.length, u32);917 _ = vec<kayan_command>push(cmds*.list, c);918 return index;919}920 921// A copy of `value`, as bytes, in the frame arena. The one trusted proc of922// the command buffer: a value's bytes are a cast.923kayan_defer_value: proc<T>(h: *kayan_heap in frame, value: T) -> []u8 in frame = {924 unsafe {925 data: ^u8 = cast(memops_arena_push(h*.arena[0]&, sizeof(T), alignof(T)), ^u8);926 memcpy(data, value&, sizeof(T));927 return cast(data[0..sizeof(T)], []u8 in frame);928 }929}930 931kayan_defer_spawn: proc(cmds: *kayan_commands, signature: u64) -> kayan_pending = {932 p: kayan_pending = {};933 p.command = kayan_defer(cmds, { .kind = kayan_command_kind.spawn, .signature = signature });934 return p;935}936 937kayan_pending_set: proc<T>(cmds: *kayan_commands, p: kayan_pending, value: T) -> void = {938 _ = kayan_defer(cmds, { .kind = kayan_command_kind.add, .spawn = p.command + 1, .id = kayan_id_of<T>(),939 .value = kayan_defer_value<T>(cmds*.heap, value) });940}941 942kayan_defer_despawn: proc(cmds: *kayan_commands, e: kayan_entity) -> void = {943 _ = kayan_defer(cmds, { .kind = kayan_command_kind.despawn, .entity = e });944}945 946kayan_defer_add: proc<T>(cmds: *kayan_commands, e: kayan_entity, value: T) -> void = {947 _ = kayan_defer(cmds, { .kind = kayan_command_kind.add, .entity = e, .id = kayan_id_of<T>(),948 .value = kayan_defer_value<T>(cmds*.heap, value) });949}950 951kayan_defer_remove: proc<T>(cmds: *kayan_commands, e: kayan_entity) -> void = {952 _ = kayan_defer(cmds, { .kind = kayan_command_kind.remove, .entity = e, .id = kayan_id_of<T>() });953}954 955// One applied command. The components given to a new entity are part of its956// spawn line, so they are not printed on their own.957kayan_print_command: proc(w: *const kayan_world, c: *const kayan_command, status: kayan_status) -> void = {958 if (c*.spawn != 0 && status == kayan_status.ok) {959 return;960 }961 printfmt(" {} ", c*.kind);962 if (c*.kind == kayan_command_kind.spawn) {963 kayan_print_signature(w, c*.signature);964 printfmt(" -> ");965 } else if (c*.kind != kayan_command_kind.despawn) {966 printfmt("{}, ", kayan_info(w, c*.id)*.name);967 }968 printfmt("{}:{}", c*.entity.index, c*.entity.generation);969 if (status != kayan_status.ok) {970 printfmt(": skipped, {}", status);971 }972 printfmt("\n");973}974 975// Apply the frame's commands in the order they were recorded. A command for976// an entity that died after it was recorded (despawned by an earlier command,977// most likely) does nothing, and is counted. Letting the commands go is the978// frame region's end, not a line here.979kayan_flush: proc(w: *kayan_world, cmds: *kayan_commands) -> void = {980 for (i: u64 = 0; i < cmds*.list*.length; i += 1) {981 c: *kayan_command = vec<kayan_command>at(cmds*.list, i);982 if (c*.spawn != 0) {983 c*.entity = vec<kayan_command>at(cmds*.list, cast(c*.spawn - 1, u64))*.entity;984 }985 status: kayan_status = kayan_status.ok;986 if (c*.kind == kayan_command_kind.spawn) {987 c*.entity = kayan_spawn(w, c*.signature);988 } else if (c*.kind == kayan_command_kind.despawn) {989 status = kayan_despawn(w, c*.entity);990 } else if (c*.kind == kayan_command_kind.add) {991 status = kayan_add_id(w, c*.entity, c*.id, c*.value);992 } else {993 status = kayan_remove_id(w, c*.entity, c*.id);994 }995 if (status == kayan_status.ok) {996 w*.applied += 1;997 } else {998 w*.skipped += 1;999 }1000 if (w*.trace) {1001 kayan_print_command(w, c, status);1002 }1003 }1004}1005 1006// ---------------------------------------------------------------------------1007// Systems1008// ---------------------------------------------------------------------------1009 1010// What a system gets for each table it runs on: the world, the table, the1011// frame's time step, the frame's command buffer, and `ctx`: whatever the1012// system needs beyond its own tables, here another query. It is a typed1013// pointer, not a `*void`: the checker has to know what it points at, and the1014// systems that use it test it. Every pointer names its region, so the struct1015// can be made wherever frame is open.1016kayan_iter: struct = {1017 world: *kayan_world in level;1018 table: *kayan_table in level;1019 dt: f32;1020 commands: *kayan_commands in frame;1021 ctx: option<*kayan_query in level>;1022}1023 1024kayan_system_proc: alias = *proc(it: *kayan_iter) -> void;1025 1026// A system is code: a name, and the proc run on each table its query matches.1027// A C string and a proc pointer are both pointers without a region, so a1028// struct holding them is a carrier: safe code reaches it only through a1029// pointer into a region, made by a trusted proc. Neither changes after it is1030// made, so nothing is lost.1031kayan_system: struct = {1032 name: ^const char;1033 run: kayan_system_proc;1034}1035 1036kayan_system_make: proc(r: region level, name: ^const char, run: kayan_system_proc) -> *kayan_system in level = {1037 unsafe {1038 s: *kayan_system in level = cast(memops_arena_push_zero(r, sizeof(kayan_system), alignof(kayan_system)), *kayan_system in level);1039 s*.name = name;1040 s*.run = run;1041 return s;1042 }1043}1044 1045// A system in the schedule: which system, the query that finds its tables,1046// and what it has run over so far. `tables` and `rows` count the calls.1047kayan_scheduled: struct = {1048 system: *kayan_system in level;1049 query: kayan_query;1050 ctx: option<*kayan_query in level>;1051 tables: u64;1052 rows: u64;1053}1054 1055// The schedule is a list of these, in the order the frame runs them. Each1056// entry is its own allocation so a pointer to one is a pointer into level,1057// which the checker never counts as stale.1058kayan_schedule_add: proc(r: region level, systems: *vec<*kayan_scheduled in level>, name: ^const char,1059 all: u64, none: u64, run: kayan_system_proc, ctx: option<*kayan_query in level>) -> void = {1060 _ = vec<*kayan_scheduled in level>push(systems, new<kayan_scheduled>(r, {1061 .system = kayan_system_make(r, name, run),1062 .query = kayan_query_init(r, all, none),1063 .ctx = ctx,1064 }));1065}1066 1067kayan_scheduled_at: proc(systems: *const vec<*kayan_scheduled in level>, s: u64) -> *kayan_scheduled in level = {1068 return vec<*kayan_scheduled in level>at_const(systems, s)*;1069}1070 1071// One frame: every system in order, each over every table it matches that has1072// rows, then the commands they recorded. A system sees every value the1073// systems before it wrote this frame, and no structural change until the1074// next. The caller opens the frame region and makes the buffer in it.1075kayan_progress: proc(w: *kayan_world in level, systems: *const vec<*kayan_scheduled in level>, dt: f32,1076 cmds: *kayan_commands in frame) -> void = {1077 for (s: u64 = 0; s < systems*.length; s += 1) {1078 sys: *kayan_scheduled in level = kayan_scheduled_at(systems, s);1079 kayan_query_update(w, sys*.query&);1080 for (i: u64 = 0; i < sys*.query.tables*.length; i += 1) {1081 it: kayan_iter = { .world = w, .table = kayan_query_table(w, sys*.query&, i), .dt = dt,1082 .commands = cmds, .ctx = sys*.ctx };1083 if (it.table*.count > 0) {1084 sys*.tables += 1;1085 sys*.rows += it.table*.count;1086 sys*.system*.run(it&);1087 }1088 }1089 }1090 kayan_flush(w, cmds);1091}1092 1093kayan_print_schedule: proc(w: *const kayan_world, systems: *const vec<*kayan_scheduled in level>) -> void = {1094 for (s: u64 = 0; s < systems*.length; s += 1) {1095 sys: *kayan_scheduled in level = kayan_scheduled_at(systems, s);1096 kayan_print_query(w, sys*.system*.name, sys*.query&);1097 }1098}1099 1100// ---------------------------------------------------------------------------1101// The program1102// ---------------------------------------------------------------------------1103 1104bullet_signature: proc() -> u64 = {1105 return kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<damage>() | kayan_mask_of<lifetime>();1106}1107 1108spawn_bullet: proc(w: *kayan_world, x: f32, seconds: f32) -> kayan_entity = {1109 e: kayan_entity = kayan_spawn(w, bullet_signature());1110 _ = kayan_set<position>(w, e, { .at = { .x = x, .y = 0.0 } });1111 _ = kayan_set<velocity>(w, e, { .per_second = { .x = 8.0, .y = 0.0 } });1112 _ = kayan_set<damage>(w, e, { .amount = 25.0 });1113 _ = kayan_set<lifetime>(w, e, { .seconds = seconds });1114 return e;1115}1116 1117spawn_target: proc(w: *kayan_world, x: f32, y: f32) -> kayan_entity = {1118 e: kayan_entity = kayan_spawn(w, kayan_mask_of<position>() | kayan_mask_of<health>());1119 _ = kayan_set<position>(w, e, { .at = { .x = x, .y = y } });1120 _ = kayan_set<health>(w, e, { .points = 100.0, .max = 100.0 });1121 return e;1122}1123 1124// The wrong way: despawn as the loop goes. Row `row` is swap-removed, the last1125// row moves into it, and the loop moves on to row + 1. The checker lets this1126// through: the despawn takes the world, and the slice is taken again each1127// time round, so nothing it can see is stale. The row it misses is a logic1128// bug, which is the kind a loop over indices leaves you with.1129expire_now_system: proc(it: *kayan_iter) -> void = {1130 t: *kayan_table in level = it*.table;1131 for (row: u64 = 0; row < t*.count; row += 1) {1132 ls: []lifetime = kayan_slice<lifetime>(t);1133 es: []kayan_entity = kayan_entities(t);1134 if (ls[row].seconds <= 0.0) {1135 _ = kayan_despawn(it*.world, es[row]);1136 }1137 }1138}1139 1140// The right way: record the despawn. The table does not change until the1141// frame is over, so every row is looked at once.1142expire_system: proc(it: *kayan_iter) -> void = {1143 ls: []lifetime = kayan_slice<lifetime>(it*.table);1144 es: []kayan_entity = kayan_entities(it*.table);1145 for (row: u64 = 0; row < es.length; row += 1) {1146 if (ls[row].seconds <= 0.0) {1147 kayan_defer_despawn(it*.commands, es[row]);1148 }1149 }1150}1151 1152age_system: proc(it: *kayan_iter) -> void = {1153 ls: []lifetime = kayan_slice<lifetime>(it*.table);1154 for (row: u64 = 0; row < ls.length; row += 1) {1155 ls[row].seconds -= it*.dt;1156 }1157}1158 1159move_system: proc(it: *kayan_iter) -> void = {1160 ps: []position = kayan_slice<position>(it*.table);1161 vs: []velocity = kayan_slice<velocity>(it*.table);1162 for (row: u64 = 0; row < ps.length; row += 1) {1163 ps[row].at.x += vs[row].per_second.x * it*.dt;1164 ps[row].at.y += vs[row].per_second.y * it*.dt;1165 }1166}1167 1168// A turret whose timer is up spawns a bullet where it stands. The bullet does1169// not exist until the frame ends, so it first moves next frame.1170turret_system: proc(it: *kayan_iter) -> void = {1171 ps: []position = kayan_slice<position>(it*.table);1172 ts: []turret = kayan_slice<turret>(it*.table);1173 for (row: u64 = 0; row < ts.length; row += 1) {1174 if (ts[row].timer <= 0.0) {1175 shot: kayan_pending = kayan_defer_spawn(it*.commands, bullet_signature());1176 kayan_pending_set<position>(it*.commands, shot, ps[row]);1177 kayan_pending_set<velocity>(it*.commands, shot, { .per_second = { .x = 8.0, .y = 0.0 } });1178 kayan_pending_set<damage>(it*.commands, shot, { .amount = 25.0 });1179 kayan_pending_set<lifetime>(it*.commands, shot, { .seconds = 2.0 });1180 ts[row].timer += ts[row].cooldown;1181 }1182 ts[row].timer -= it*.dt;1183 }1184}1185 1186// A bullet within half a unit of a target hits it. The damage is a value, so1187// it is written now; the bullet's despawn is structure, so it is recorded. The1188// targets are another query, which the system gets through its ctx. During a1189// frame no table is made or moved, so walking a second query is safe.1190hit_system: proc(it: *kayan_iter) -> void = {1191 w: *kayan_world in level = it*.world;1192 targets: option<*kayan_query in level> = it*.ctx;1193 if (!targets) {1194 return;1195 }1196 ps: []position = kayan_slice<position>(it*.table);1197 ds: []damage = kayan_slice<damage>(it*.table);1198 es: []kayan_entity = kayan_entities(it*.table);1199 kayan_query_update(w, targets);1200 for (b: u64 = 0; b < es.length; b += 1) {1201 for (i: u64 = 0; i < targets*.tables*.length; i += 1) {1202 t: *kayan_table in level = kayan_query_table(w, targets, i);1203 tps: []position = kayan_slice<position>(t);1204 ths: []health = kayan_slice<health>(t);1205 for (row: u64 = 0; row < tps.length; row += 1) {1206 dx: f32 = tps[row].at.x - ps[b].at.x;1207 dy: f32 = tps[row].at.y - ps[b].at.y;1208 if (dx * dx + dy * dy < 0.25) {1209 ths[row].points -= ds[b].amount;1210 kayan_defer_despawn(it*.commands, es[b]);1211 }1212 }1213 }1214 }1215}1216 1217make_world: proc(r: region level) -> *kayan_world in level = {1218 w: *kayan_world in level = kayan_world_make(r);1219 kayan_register(w, kayan_id_of<position>(), position<>&);1220 kayan_register(w, kayan_id_of<velocity>(), velocity<>&);1221 kayan_register(w, kayan_id_of<health>(), health<>&);1222 kayan_register(w, kayan_id_of<damage>(), damage<>&);1223 kayan_register(w, kayan_id_of<lifetime>(), lifetime<>&);1224 kayan_register(w, kayan_id_of<turret>(), turret<>&);1225 kayan_register(w, kayan_id_of<burning>(), burning<>&);1226 return w;1227}1228 1229// Four bullets, three of them out of time.1230four_bullets: proc(w: *kayan_world) -> void = {1231 _ = spawn_bullet(w, 1.0, 0.0);1232 _ = spawn_bullet(w, 2.0, 1.0);1233 _ = spawn_bullet(w, 3.0, 0.0);1234 _ = spawn_bullet(w, 4.0, 0.0);1235}1236 1237print_lifetimes: proc(w: *kayan_world) -> void = {1238 t: *kayan_table in level = kayan_table_at(w, kayan_table_for(w, bullet_signature()));1239 ls: []lifetime = kayan_slice<lifetime>(t);1240 es: []kayan_entity = kayan_entities(t);1241 printfmt(" lifetimes:");1242 for (row: u64 = 0; row < t*.count; row += 1) {1243 e: kayan_entity = es[row];1244 l: lifetime = ls[row];1245 printfmt(" {}:{} {:.2}", e.index, e.generation, cast(l.seconds, f64));1246 }1247 printfmt("\n");1248}1249 1250// One frame of a world: the frame region opens, the buffer is made in it, the1251// systems run and their commands are applied, and the region ends.1252run_frame: proc(w: *kayan_world in level, systems: *const vec<*kayan_scheduled in level>, dt: f32) -> void = {1253 frame: region = {1254 kayan_progress(w, systems, dt, kayan_commands_make(frame));1255 }1256}1257 1258health_of: proc(w: *const kayan_world, e: kayan_entity) -> f32 = {1259 h: option<*health> = kayan_get<health>(w, e);1260 return h ? h*.points : 0.0;1261}1262 1263main: proc() -> i32 = {1264 level: region = {1265 lifetime_only: u64 = kayan_mask_of<lifetime>();1266 1267 printfmt("1. an expire system that despawns as it goes\n");1268 a: *kayan_world in level = make_world(level);1269 four_bullets(a);1270 print_lifetimes(a);1271 wrong: *vec<*kayan_scheduled in level> in level = vec<*kayan_scheduled in level>make(level, 1);1272 kayan_schedule_add(level, wrong, "expire", lifetime_only, 0, expire_now_system, null);1273 run_frame(a, wrong, 0.25);1274 printfmt("after one frame:\n");1275 print_lifetimes(a);1276 printfmt("kayan_verify: {} problems. The world is fine; the loop missed one.\n", kayan_verify(a));1277 1278 printfmt("\n2. an expire system that records despawns\n");1279 b: *kayan_world in level = make_world(level);1280 b*.trace = true;1281 four_bullets(b);1282 print_lifetimes(b);1283 right: *vec<*kayan_scheduled in level> in level = vec<*kayan_scheduled in level>make(level, 1);1284 kayan_schedule_add(level, right, "expire", lifetime_only, 0, expire_system, null);1285 run_frame(b, right, 0.25);1286 printfmt("after one frame:\n");1287 print_lifetimes(b);1288 1289 printfmt("\n3. a turret, two targets in its line of fire, one out of it\n");1290 c: *kayan_world in level = make_world(level);1291 c*.trace = true;1292 gun: kayan_entity = kayan_spawn(c, kayan_mask_of<position>() | kayan_mask_of<turret>());1293 _ = kayan_set<turret>(c, gun, { .cooldown = 0.5, .timer = 0.0 });1294 near: kayan_entity = spawn_target(c, 4.0, 0.0);1295 far: kayan_entity = spawn_target(c, 4.25, 0.0);1296 aside: kayan_entity = spawn_target(c, 4.0, 3.0);1297 targets: *kayan_query in level = new<kayan_query>(level, kayan_query_init(level, kayan_mask_of<position>() | kayan_mask_of<health>(), 0));1298 systems: *vec<*kayan_scheduled in level> in level = vec<*kayan_scheduled in level>make(level, 5);1299 kayan_schedule_add(level, systems, "turret", kayan_mask_of<position>() | kayan_mask_of<turret>(), 0, turret_system, null);1300 kayan_schedule_add(level, systems, "age", lifetime_only, 0, age_system, null);1301 kayan_schedule_add(level, systems, "expire", lifetime_only, 0, expire_system, null);1302 kayan_schedule_add(level, systems, "hit", kayan_mask_of<position>() | kayan_mask_of<damage>(), 0, hit_system, targets);1303 kayan_schedule_add(level, systems, "move", kayan_mask_of<position>() | kayan_mask_of<velocity>(), 0, move_system, null);1304 for (frame_no: u32 = 1; frame_no <= 6; frame_no += 1) {1305 printfmt("frame {}\n", frame_no);1306 run_frame(c, systems, 0.25);1307 }1308 printfmt("health: {}:{} {:.0}, {}:{} {:.0}, {}:{} {:.0}\n",1309 near.index, near.generation, cast(health_of(c, near), f64),1310 far.index, far.generation, cast(health_of(c, far), f64),1311 aside.index, aside.generation, cast(health_of(c, aside), f64));1312 printfmt("commands applied: {}, skipped: {}\n", c*.applied, c*.skipped);1313 kayan_print_tables(c);1314 printfmt("kayan_verify: {} problems\n", kayan_verify(c));1315 }1316 return 0;1317}What commands prints. tools/check.py fails the chapter if a single byte differs.
1. an expire system that despawns as it goes
lifetimes: 0:1 0.00 1:1 1.00 2:1 0.00 3:1 0.00
after one frame:
lifetimes: 3:1 0.00 1:1 1.00
kayan_verify: 0 problems. The world is fine; the loop missed one.
2. an expire system that records despawns
lifetimes: 0:1 0.00 1:1 1.00 2:1 0.00 3:1 0.00
despawn 0:1
despawn 2:1
despawn 3:1
after one frame:
lifetimes: 1:1 1.00
3. a turret, two targets in its line of fire, one out of it
frame 1
spawn {position velocity damage lifetime} -> 4:1
frame 2
frame 3
spawn {position velocity damage lifetime} -> 5:1
frame 4
despawn 4:1
despawn 4:1: skipped, dead
frame 5
spawn {position velocity damage lifetime} -> 4:2
frame 6
despawn 5:1
despawn 5:1: skipped, dead
health: 1:1 50, 2:1 50, 3:1 100
commands applied: 17, skipped: 2
table 0 {position turret}: 0:1
table 1 {position health}: 1:1 2:1 3:1
table 2 {position velocity damage lifetime}: 4:2
kayan_verify: 0 problems
Build an ECS · part 3, systems · 3.4
A game
ecs/src/3.systems/3.4.game/game.rin
In the dojo: this chapter puts the building blocks together; the dojo has each one on its own, to watch and to practise.
Everything together. Three lanes, a turret at the left end of each, and drones coming from the right in waves:
- a turret fires down its lane while there's a drone in it;
- a bullet that reaches a drone damages it;
- a drone below half health catches fire;
- a drone with no health left is destroyed;
- a drone that gets to the left end has reached the base.
The top of the file is kayan, as 3.3 left it, with one type changed: ctx now points at the game. The game is the bottom: components, kernels, systems, a schedule and a level script. Every rule of the game is a system over a query. Every structural change a system makes is a command.
The pieces
The components are the same seven from 2.1. A drone is a signature:
drone_signature: proc() -> u64 = {
return kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<health>();
}
A burning drone is that plus burning, which puts it in another table. There is no drone type anywhere in the program. A drone is whatever lives in a table with these columns.
What the systems share (the drones query, the clock, the score) is one struct, handed to the systems that need it through ctx:
game: struct = {
drones: kayan_query;
time: f32;
fired: u32;
hits: u32;
fires: u32;
kills: u32;
breaches: u32;
came: u32;
}
It lives in level memory, beside the world, so an iterator can hold it:
g: *game in level = new<game>(level, {
.drones = kayan_query_init(level, kayan_mask_of<position>() | kayan_mask_of<health>(), 0),
});
Not every system has a game to look at, so the iterator's field is an option<*game in level>, and a system that wants it starts by testing it:
ignite_system: proc(it: *kayan_iter) -> void = {
g: option<*game in level> = it*.ctx;
if (!g) {
return;
}
The schedule
kayan_schedule_add(level, systems, "turret", shooting, 0, turret_system, g);
kayan_schedule_add(level, systems, "age", kayan_mask_of<lifetime>(), 0, age_system, null);
kayan_schedule_add(level, systems, "expire", kayan_mask_of<lifetime>(), 0, expire_system, null);
kayan_schedule_add(level, systems, "move", movers, 0, move_system, null);
kayan_schedule_add(level, systems, "hit", bullets, 0, hit_system, g);
kayan_schedule_add(level, systems, "ignite", kayan_mask_of<health>(), kayan_mask_of<burning>(), ignite_system, g);
kayan_schedule_add(level, systems, "burn", kayan_mask_of<health>() | kayan_mask_of<burning>(), 0, burn_system, null);
kayan_schedule_add(level, systems, "die", kayan_mask_of<health>(), 0, die_system, g);
kayan_schedule_add(level, systems, "breach", kayan_mask_of<position>() | kayan_mask_of<health>(), 0, breach_system, g);
Read it as the frame, top to bottom. Each line's position is a data-flow decision (3.2):
movebeforehit: bullets hit where they are, not where they were.hitbeforeignite,burnanddie: a drone takes damage, catches fire and is destroyed in the same frame its health drops.diebeforebreach: only a drone that survived the frame can reach the base.breachalso checkshealth > 0itself, because both run in the same frame anddie's despawn hasn't been applied yet.ignitehasnone {burning}: a drone catches fire once. The add is a command, so next frame it's in the burning table and the query no longer matches it.
The rules
Each one is a few lines. The ones that only change values call kernels, with the noalias promise from 3.2 kept right there at the call: a health column and a burning column can't overlap.
burn_system: proc(it: *kayan_iter) -> void = {
unsafe { burn(kayan_slice<health>(it*.table), kayan_slice<burning>(it*.table), it*.dt); }
}
The ones that change structure record commands:
if (hs[row].points < 0.5 * hs[row].max && hs[row].points > 0.0) {
kayan_defer_add<burning>(it*.commands, es[row], { .per_second = 10.0 });
g*.fires += 1;
}
The turret is the most involved. It asks the drones query whether anything is ahead of it in its lane, then spawns a bullet with a pending spawn (3.3):
if (ts[row].timer <= 0.0 && drone_ahead(it*.world, g, ps[row].at.x, ps[row].at.y)) {
hit is 3.3's hit system, with one extra thought about speed. A bullet moves a unit a frame and a fast drone a quarter, so they close by at most 1.25 a frame. A hit window of ±0.75 is 1.5 wide, so no bullet can pass through a drone between two frames:
if (dx * dx < 0.5625 && dy * dy < 0.25) {
The level
The waves aren't a system. They're the level script, and it runs between frames, in the game loop, where spawning directly is safe:
for (frame_no: u32 = 0; frame_no < frames; frame_no += 1) {
g*.time = cast(frame_no, f32) * dt;
if (frame_no % 32 == 0 && frame_no / 32 < 8) {
send_wave(w, g, frame_no / 32);
}
run_frame(w, systems, dt);
problems += kayan_verify(w);
run_frame opens the frame region and closes it again, 512 times, so every command and every copied value from the whole game ends with the frame that recorded it. Level memory holds what lasts.
kayan_verify runs after every frame: 512 checks of Part 1's invariant over the whole world.
Drawing by table
The picture of the lanes walks a query of everything with a position. What an entity is drawn as comes from its table's signature, decided once per table, not per entity:
glyph_of: proc(signature: u64) -> c8 = {
if ((signature & kayan_mask_of<turret>()) != 0) {
return 'T';
}
if ((signature & kayan_mask_of<burning>()) != 0) {
return '*';
}
if ((signature & kayan_mask_of<health>()) != 0) {
return 'd';
}
return '-';
}
In an OOP engine this would be a virtual draw() per object. Here it's a property of a table, and the inner loop is a plain loop over positions.
Exact on purpose
Every speed and time step is a power of two: dt is 1/16, drones move 2 or 4 units a second, bullets 16, fire burns 10 a second (0.625 a frame). So every position, health and timer is exactly representable, and no rounding ever happens. That's why a Debug build, a Release build, gcc, clang and a CPU that fuses multiply-adds all print the same game. Real games give that up for nicer numbers. Rounding differences usually don't matter, but they do the day you want a replay or a lockstep network game.
Run it
0.00s wave 1: 3:1, 2 units a second
2.00s wave 2: 7:1 8:1, 2 units a second
4.00s 12 entities in 4 tables
2 T......-...........-........d....
1 T..........-...........-*........
0 T......-...........-........d....
4.00s wave 3: 9:2 10:2 14:1, 2 units a second
4.00s 3:1 destroyed
...
28.00s 6 entities in 4 tables
2 T.*..............................
1 T.*..............................
0 T.*..............................
28.12s 15:4 destroyed
28.31s 4:6 destroyed
28.31s 5:7 destroyed
...
32.00 seconds, 512 frames; kayan_verify after every one: 0 problems
drones: 28 came, 27 destroyed, 1 reached the base, 0 still flying
bullets: 110 fired, 111 hits; fires started: 28
entities: at most 27 alive at once; commands: 716 applied, 1 skipped
The last three drones burn down one unit short of the base. There's one more hit than bullets: once, a bullet reached two drones in the same frame. That's also the one skipped command, its second despawn.
the schedule, and what each system ran over:
turret called 512 times, over 1536 rows
age called 409 times, over 1622 rows
expire called 409 times, over 1622 rows
move called 1018 times, over 5051 rows
hit called 409 times, over 1622 rows
ignite called 443 times, over 3145 rows
burn called 166 times, over 284 rows
die called 609 times, over 3429 rows
breach called 609 times, over 3429 rows
And the whole game ran on four tables:
the tables at the end:
table 0 {position turret}: 0:1 1:1 2:1
table 1 {position velocity health}:
table 2 {position velocity damage lifetime}:
table 3 {position velocity health burning}:
What you built
Part 1 entities handles, generations, slots, swap-remove, kayan_verify
Part 2 components ids, columns, tables, archetypes, typed access,
moving between tables, the archetype graph, reflection
Part 3 systems queries, kernels and schedules, commands, a game
That's a working archetype ECS, about 790 lines of kayan without comments. It's the design flecs is built on (tables, the archetype graph, cached queries, command buffers), and Bevy and Unity DOTS are close cousins, minus their years of features.
And it's memory safe, with very little left on trust. Here's all of it. Kayan has eight unsafe { } blocks, each a place where bytes meet types or where memory gets its region: 3.1's six, plus kayan_system_make for the proc pointers (3.2) and kayan_defer_value for a value's bytes (3.3). The game adds three noalias kernels, each called from a one-line unsafe { } that keeps its promise. Everything else is checked, and that's the rest of the file: the world, the tables, the queries, the commands, and every other rule of the game. And the blocks that turn bytes into types don't take the checked code's word for anything: 2.2's rule holds to the end, so a mistake up here stops the game or gives it wrong numbers.
The safety and the data-oriented design turned out to want the same things. Memory that ends all at once is what a level and a frame already are. Handles, not pointers, were already how an ECS names things. Commands were already how it keeps tables still while a system walks them. Very little of this design exists only to satisfy the checker.
Things worth adding next, roughly in order of how much they teach:
- Relationships:
child_of(parent)as a component, flecs' big idea. - Change detection: a per-column version number, so a system can skip tables nothing wrote to.
- Parallel systems: systems that don't write what others read can run at the same time. The schedule already holds what you need to decide that.
- Removing empty tables, and the queries' table lists with them.
- Serialization: the reflect records already describe every byte; write the tables out and read them back.
Try
- Add a fourth lane. What changes? (Only the level:
lanes, the picture.) - Make bullets pierce: they don't despawn on a hit, but each drone can only be hit by a bullet once. Where does "once" live? (A component on the bullet? On the drone?)
- Add a
shieldcomponent that absorbs the next hit, then removes itself. Which systems change? - Print
kayan_inspectof a drone the frame it catches fire.
1// 3.4 A game2//3// Everything together. Three lanes, a turret at the left end of each, and4// drones coming from the right in waves. A turret fires down its lane while a5// drone is in it; a bullet that reaches a drone damages it; a drone below half6// health catches fire, and one with no health left is destroyed. A drone that7// gets to the left end has reached the base.8//9// kayan entities, tables, queries, systems and commands: 1.1 to 3.3,10// unchanged11// the game components, kernels, systems and a schedule: the end of this12// file13//14// Every rule of the game is a system over a query, and every structural change15// a system makes goes through a command. The level script that sends the waves16// runs between frames, where spawning directly is safe.17//18// Every speed and time step is a power of two, so the arithmetic is exact and19// a Debug and a Release build print the same game.20//21// The game's state is a `game` in level memory, and the systems reach it22// through the typed `ctx` on their iterator. The `unsafe` blocks are kayan's23// byte core, as in 3.1 to 3.3, and the calls to the game's three kernels, whose24// `noalias` is a promise the checker cannot check. The rest is checked.25 26import "std/print.rin"27import "std/vec.rin"28import "std/hashmap.rin"29import "std/slice.rin"30 31level: region;32frame: region in level;33 34vec2: struct = {35 x: f32;36 y: f32;37}38 39position: struct = { at: vec2; }40velocity: struct = { per_second: vec2; }41health: struct = { points: f32; max: f32; }42damage: struct = { amount: f32; }43lifetime: struct = { seconds: f32; }44// A field tagged "hidden" is bookkeeping the inspector does not show.45turret: struct = { cooldown: f32; timer: f32 @ "hidden"; }46burning: struct = { per_second: f32; }47 48component: enum = {49 position,50 velocity,51 health,52 damage,53 lifetime,54 turret,55 burning,56}57 58component_of: proc<position>() -> component = { return component.position; }59component_of: proc<velocity>() -> component = { return component.velocity; }60component_of: proc<health>() -> component = { return component.health; }61component_of: proc<damage>() -> component = { return component.damage; }62component_of: proc<lifetime>() -> component = { return component.lifetime; }63component_of: proc<turret>() -> component = { return component.turret; }64component_of: proc<burning>() -> component = { return component.burning; }65 66// ---------------------------------------------------------------------------67// Entities (Part 1)68// ---------------------------------------------------------------------------69 70kayan_entity: struct = {71 index: u32;72 generation: u32;73}74 75// `missing`: the entity is alive but has no such component.76kayan_status: enum[must_use] = {77 ok,78 dead,79 missing,80}81 82// Where an entity lives: which table, and which row of it.83kayan_slot: struct = {84 generation: u32;85 table: u32;86 row: u32;87}88 89// A structural change recorded during a frame and applied after it (3.3).90// `entity` is who it applies to; a spawn's is filled in when it is applied. An91// add to an entity spawned in the same frame has no entity to name yet, so it92// names its spawn instead: `spawn` is that command's index plus one. The93// value's bytes are frame memory, like the command itself.94kayan_command_kind: enum = {95 spawn,96 despawn,97 add,98 remove,99}100 101kayan_command: struct = {102 kind: kayan_command_kind;103 entity: kayan_entity;104 spawn: u32;105 id: u32;106 signature: u64;107 value: []u8 in frame;108}109 110// ---------------------------------------------------------------------------111// The byte core: columns (2.2), and the region they grow into112// ---------------------------------------------------------------------------113 114// The region's arena, kept where the trusted procs can reach it: the world115// remembers where it was made, as a vec does, so nothing above this layer116// hands an arena around.117kayan_heap: struct = {118 arena: ^memops_arena;119}120 121kayan_heap_make: proc(r: region R) -> *kayan_heap in R = {122 unsafe {123 h: *kayan_heap in R = cast(memops_arena_push_zero(r, sizeof(kayan_heap), alignof(kayan_heap)), *kayan_heap in R);124 h*.arena = r;125 return h;126 }127}128 129// One component's values for every row of a table, as bytes: `size` of them130// a row, `capacity` rows allocated (the table knows how many are in use).131kayan_column: struct = {132 data: []u8 in level;133 size: u64;134 align: u64;135}136 137kayan_column_make: proc(h: *kayan_heap in level, info: *const reflect in permanent) -> *kayan_column in level = {138 unsafe {139 c: *kayan_column in level = cast(memops_arena_push_zero(h*.arena[0]&, sizeof(kayan_column), alignof(kayan_column)), *kayan_column in level);140 c*.size = info*.size;141 c*.align = info*.align;142 return c;143 }144}145 146// Room for `capacity` rows, the first `count` copied over. The old storage is147// left in the region: a slice taken before the grow still points at valid148// memory, and stale values.149kayan_column_grow: proc(h: *kayan_heap, c: *kayan_column, count: u64, capacity: u64) -> void = {150 unsafe {151 raw: ^u8 = cast(memops_arena_push_zero(h*.arena[0]&, c*.size * capacity, c*.align), ^u8);152 bytes: []u8 in level = cast(raw[0..c*.size * capacity], []u8 in level);153 // Slicing checks both ends: a count past either block stops the program.154 kept: u64 = c*.size * count;155 slice<u8>copy(bytes[0..kept], c*.data[0..kept]);156 c*.data = bytes;157 }158}159 160// One row's bytes, bounds-checked like any slice.161kayan_column_bytes: proc(c: *kayan_column, row: u64) -> []u8 in level = {162 return c*.data[row * c*.size..(row + 1) * c*.size];163}164 165// The typed view: the column's first `count` rows as `[]T`. This is the one166// place bytes become a type, so it takes nothing on trust. T must be the167// column's size, and plain data: any bytes are then a valid T, so reading the168// wrong type gives wrong numbers, never a bad pointer. The rows must fit, which169// the slice knows and no field can lie about, and the bytes must be aligned for170// T. The view is `[]T in level`: the bytes are in the region, so the view is171// good memory for as long as the region is. After the column grows it is the172// old block, still readable, no longer the column: take the view again.173kayan_column_slice: proc<T>(c: *const kayan_column, count: u64) -> []T in level = {174 if (c*.size != sizeof(T)) {175 printfmt("kayan: a column of {}-byte rows read as a {}-byte type\n", c*.size, sizeof(T));176 exit(1);177 }178 if (!reflect_is_plain(T<>&)) {179 printfmt("kayan: a column holds plain data, and this {}-byte type is not\n", sizeof(T));180 exit(1);181 }182 if (count > c*.data.length / sizeof(T)) {183 printfmt("kayan: a view of {} rows on a column with room for {}\n", count, c*.data.length / sizeof(T));184 exit(1);185 }186 unsafe {187 if (cast(c*.data.data, usize) % alignof(T) != 0) {188 printfmt("kayan: a column's bytes are not aligned to {}\n", alignof(T));189 exit(1);190 }191 raw: ^T = cast(c*.data.data, ^T);192 return cast(raw[0..count], []T in level);193 }194}195 196// Copying and zeroing a row are loops over the row's bytes: checked code.197kayan_column_copy_row: proc(c: *kayan_column, to: u64, from: u64) -> void = {198 dst: []u8 = kayan_column_bytes(c, to);199 src: []u8 = kayan_column_bytes(c, from);200 for (i: u64 = 0; i < dst.length; i += 1) {201 dst[i] = src[i];202 }203}204 205kayan_column_zero_row: proc(c: *kayan_column, row: u64) -> void = {206 bytes: []u8 = kayan_column_bytes(c, row);207 for (i: u64 = 0; i < bytes.length; i += 1) {208 bytes[i] = 0;209 }210}211 212// ---------------------------------------------------------------------------213// The table214// ---------------------------------------------------------------------------215 216// `columns[id]` is a column when the signature has bit `id`, and null217// otherwise. Count and capacity are the table's, shared by its columns, so218// they can never disagree; `entities` grows on its own and always has `count`219// rows.220//221// The edges: `add_edges[id]` is the table this one's entities move to when222// they gain component `id`, and `remove_edges[id]` when they lose it. Each is223// the table's index plus one, so 0, which a new table starts with, means "not224// known yet".225kayan_table: struct = {226 signature: u64;227 count: u64;228 capacity: u64;229 entities: *vec<kayan_entity> in level;230 columns: [64]option<*kayan_column in level>;231 add_edges: [64]u32;232 remove_edges: [64]u32;233}234 235// A zeroed table in the region, with an empty entities vec: the world fills236// in the signature and the columns.237kayan_table_new: proc(h: *kayan_heap in level) -> *kayan_table in level = {238 unsafe {239 t: *kayan_table in level = cast(memops_arena_push_zero(h*.arena[0]&, sizeof(kayan_table), alignof(kayan_table)), *kayan_table in level);240 v: *vec<kayan_entity> in level = cast(memops_arena_push_zero(h*.arena[0]&, sizeof(vec<kayan_entity>), alignof(vec<kayan_entity>)), *vec<kayan_entity> in level);241 v*.arena = h*.arena;242 t*.entities = v;243 return t;244 }245}246 247// The world: the registry, the slots and free list, and every table, with a248// map from signature to table index so finding one is a lookup, not a search.249// Tables are referred to by index, so a table can be named in a slot; each250// table is its own allocation, so a pointer to one stays good when the next251// is made.252kayan_world: struct = {253 heap: *kayan_heap in level;254 infos: [64]option<*const reflect in permanent>;255 component_count: u32;256 tables: *vec<*kayan_table in level> in level;257 by_signature: *hashmap<u64, u32> in level;258 slots: *vec<kayan_slot> in level;259 free: *vec<u32> in level;260 live: u64;261 // What the flushes did, and whether to print each command as it is applied.262 applied: u64;263 skipped: u64;264 trace: b32;265}266 267kayan_bit: proc(id: u32) -> u64 = {268 return 1'u64 shl cast(id, u64);269}270 271kayan_id_of: proc<T>() -> u32 = {272 return cast(component_of<T>(), u32);273}274 275kayan_mask_of: proc<T>() -> u64 = {276 return kayan_bit(kayan_id_of<T>());277}278 279// The reflect records live for the whole program, so `permanent` is their280// region and the world may keep them.281kayan_register: proc(w: *kayan_world, id: u32, info: *const reflect in permanent) -> void = {282 if (id >= 64) {283 printfmt("kayan: component id {} is past the 64 a signature can hold\n", id);284 exit(1);285 }286 w*.infos[id] = info;287 if (id + 1 > w*.component_count) {288 w*.component_count = id + 1;289 }290}291 292// A registered component's record. The checker needs the null test; the293// return after exit is never reached.294kayan_info: proc(w: *const kayan_world, id: u32) -> *const reflect in permanent = {295 info: option<*const reflect in permanent> = w*.infos[id];296 if (info) {297 return info;298 }299 printfmt("kayan: component {} was never registered\n", id);300 exit(1);301 return kayan_entity<>&;302}303 304kayan_table_has: proc(t: *const kayan_table, id: u32) -> b32 = {305 return (t*.signature & kayan_bit(id)) != 0;306}307 308// A table for a signature: one empty column per component in it, sized from309// the registry. Every component in the signature must be registered.310kayan_table_make: proc(w: *kayan_world, signature: u64) -> *kayan_table in level = {311 t: *kayan_table in level = kayan_table_new(w*.heap);312 t*.signature = signature;313 for (id: u32 = 0; id < 64; id += 1) {314 if ((signature & kayan_bit(id)) != 0) {315 if (id >= w*.component_count) {316 printfmt("kayan: signature {} uses component {}, which was never registered\n", signature, id);317 exit(1);318 }319 t*.columns[id] = kayan_column_make(w*.heap, kayan_info(w, id));320 }321 }322 return t;323}324 325// Double the room in every column at once.326kayan_table_grow: proc(w: *kayan_world, t: *kayan_table) -> void = {327 capacity: u64 = t*.capacity == 0 ? 4 : t*.capacity * 2;328 for (id: u32 = 0; id < w*.component_count; id += 1) {329 c: option<*kayan_column in level> = t*.columns[id];330 if (c) {331 kayan_column_grow(w*.heap, c, t*.count, capacity);332 }333 }334 t*.capacity = capacity;335}336 337// A new row at the end, owned by `e`, with every component zeroed.338kayan_table_add_row: proc(w: *kayan_world, t: *kayan_table, e: kayan_entity) -> u64 = {339 if (t*.count == t*.capacity) {340 kayan_table_grow(w, t);341 }342 row: u64 = t*.count;343 t*.count += 1;344 _ = vec<kayan_entity>push(t*.entities, e);345 for (id: u32 = 0; id < w*.component_count; id += 1) {346 c: option<*kayan_column in level> = t*.columns[id];347 if (c) {348 kayan_column_zero_row(c, row);349 }350 }351 return row;352}353 354// Who owns each of the table's rows. It only reads, and the checker sees that,355// so a system holds this and two or three columns of one table at once.356kayan_entities: proc(t: *const kayan_table) -> []kayan_entity in level = {357 return vec<kayan_entity>items(t*.entities);358}359 360// 1.4's swap-remove, for every column: the last row is copied over `row`, and361// the slot of the entity that owned the last row is told where it went.362kayan_table_remove_row: proc(w: *kayan_world, t: *kayan_table, row: u64) -> void = {363 last: u64 = t*.count - 1;364 if (row != last) {365 for (id: u32 = 0; id < w*.component_count; id += 1) {366 c: option<*kayan_column in level> = t*.columns[id];367 if (c) {368 kayan_column_copy_row(c, row, last);369 }370 }371 moved: kayan_entity = vec<kayan_entity>at(t*.entities, last)*;372 vec<kayan_entity>at(t*.entities, row)* = moved;373 vec<kayan_slot>at(w*.slots, cast(moved.index, u64))*.row = cast(row, u32);374 }375 _ = vec<kayan_entity>pop(t*.entities);376 t*.count = last;377}378 379// The typed view of one of the table's columns: T's rows, all of them in use.380// Asking for a component the table does not have is a bug, so it stops. The381// slice is good memory for as long as the region, but when the table grows it is382// the old block, no longer the column: take it again, and 3.3 makes that a rule.383kayan_slice: proc<T>(t: *const kayan_table) -> []T in level = {384 id: u32 = kayan_id_of<T>();385 c: option<*kayan_column in level> = t*.columns[id];386 if (!c) {387 printfmt("kayan: the table has no column for component {}\n", id);388 exit(1);389 }390 return kayan_column_slice<T>(c, t*.count);391}392 393kayan_table_at: proc(w: *const kayan_world, index: u32) -> *kayan_table in level = {394 return vec<*kayan_table in level>at(w*.tables, cast(index, u64))*;395}396 397// The table for a signature: found if it exists, made if it does not. This is398// where archetypes come from.399kayan_table_for: proc(w: *kayan_world, signature: u64) -> u32 = {400 found: option<*u32> = hashmap<u64, u32>get(w*.by_signature, signature);401 if (found) {402 return found*;403 }404 index: u32 = cast(w*.tables*.length, u32);405 _ = vec<*kayan_table in level>push(w*.tables, kayan_table_make(w, signature));406 _ = hashmap<u64, u32>set(w*.by_signature, signature, index);407 return index;408}409 410// ---------------------------------------------------------------------------411// The world412// ---------------------------------------------------------------------------413 414kayan_world_make: proc(r: region level) -> *kayan_world in level = {415 return new<kayan_world>(r, {416 .heap = kayan_heap_make(r),417 .tables = vec<*kayan_table in level>make(r, 8),418 .by_signature = hashmap<u64, u32>make(r),419 .slots = vec<kayan_slot>make(r, 64),420 .free = vec<u32>make(r, 16),421 });422}423 424kayan_alive: proc(w: *const kayan_world, e: kayan_entity) -> b32 = {425 if (cast(e.index, u64) >= w*.slots*.length) {426 return false;427 }428 return vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.generation == e.generation;429}430 431// A new entity with the components in `signature`, all zeroed, in the table432// for that signature.433kayan_spawn: proc(w: *kayan_world, signature: u64) -> kayan_entity = {434 e: kayan_entity = {};435 if (w*.free*.length > 0) {436 e.index = vec<u32>pop(w*.free);437 } else {438 e.index = cast(w*.slots*.length, u32);439 _ = vec<kayan_slot>push(w*.slots, { .generation = 1 });440 }441 e.generation = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.generation;442 table_index: u32 = kayan_table_for(w, signature);443 t: *kayan_table in level = kayan_table_at(w, table_index);444 row: u64 = kayan_table_add_row(w, t, e);445 // The slot is taken after the calls that change the world: a pointer into446 // `slots` from before them may be stale, and the checker says so.447 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));448 slot*.table = table_index;449 slot*.row = cast(row, u32);450 w*.live += 1;451 return e;452}453 454kayan_despawn: proc(w: *kayan_world, e: kayan_entity) -> kayan_status = {455 if (!kayan_alive(w, e)) {456 return kayan_status.dead;457 }458 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));459 table: u32 = slot*.table;460 row: u32 = slot*.row;461 kayan_table_remove_row(w, kayan_table_at(w, table), cast(row, u64));462 slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));463 slot*.generation += 1;464 _ = vec<u32>push(w*.free, e.index);465 w*.live -= 1;466 return kayan_status.ok;467}468 469// The components a live entity has: its table's signature. 0 for a dead one.470kayan_signature: proc(w: *const kayan_world, e: kayan_entity) -> u64 = {471 if (!kayan_alive(w, e)) {472 return 0;473 }474 return kayan_table_at(w, vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.table)*.signature;475}476 477// ---------------------------------------------------------------------------478// Moving between tables479// ---------------------------------------------------------------------------480 481// The table an entity of table `from` moves to when it gains component `id`.482// The first time, it is found by signature, which can make the table; the483// answer is written on both tables, as from's add edge and the destination's484// remove edge, so neither direction is looked up again.485kayan_table_with: proc(w: *kayan_world, from: u32, id: u32) -> u32 = {486 edge: u32 = kayan_table_at(w, from)*.add_edges[id];487 if (edge != 0) {488 return edge - 1;489 }490 to: u32 = kayan_table_for(w, kayan_table_at(w, from)*.signature | kayan_bit(id));491 kayan_table_at(w, from)*.add_edges[id] = to + 1;492 kayan_table_at(w, to)*.remove_edges[id] = from + 1;493 return to;494}495 496// The same for losing component `id`.497kayan_table_without: proc(w: *kayan_world, from: u32, id: u32) -> u32 = {498 edge: u32 = kayan_table_at(w, from)*.remove_edges[id];499 if (edge != 0) {500 return edge - 1;501 }502 to: u32 = kayan_table_for(w, kayan_table_at(w, from)*.signature & ~kayan_bit(id));503 kayan_table_at(w, from)*.remove_edges[id] = to + 1;504 kayan_table_at(w, to)*.add_edges[id] = from + 1;505 return to;506}507 508// Move a live entity to table `to_index`. Components both tables have are509// copied, byte for byte; components only the new table has start zeroed;510// components only the old table has are dropped with its old row.511kayan_move: proc(w: *kayan_world, e: kayan_entity, to_index: u32) -> void = {512 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));513 from_index: u32 = slot*.table;514 if (from_index == to_index) {515 return;516 }517 from_row: u64 = cast(slot*.row, u64);518 from: *kayan_table in level = kayan_table_at(w, from_index);519 to: *kayan_table in level = kayan_table_at(w, to_index);520 to_row: u64 = kayan_table_add_row(w, to, e);521 for (id: u32 = 0; id < w*.component_count; id += 1) {522 fc: option<*kayan_column in level> = from*.columns[id];523 tc: option<*kayan_column in level> = to*.columns[id];524 if (fc && tc) {525 dst: []u8 = kayan_column_bytes(tc, to_row);526 src: []u8 = kayan_column_bytes(fc, from_row);527 for (i: u64 = 0; i < dst.length; i += 1) {528 dst[i] = src[i];529 }530 }531 }532 kayan_table_remove_row(w, from, from_row);533 slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));534 slot*.table = to_index;535 slot*.row = cast(to_row, u32);536}537 538// ---------------------------------------------------------------------------539// The byte layer, for tools and for commands540// ---------------------------------------------------------------------------541 542// Give e component `id`, with `value` for its bytes: 2.6's add, by id instead543// of by type, because a command holds its value as bytes. The copy is a loop544// over two slices the checker has sized.545kayan_add_id: proc(w: *kayan_world, e: kayan_entity, id: u32, value: []const u8) -> kayan_status = {546 if (!kayan_alive(w, e)) {547 return kayan_status.dead;548 }549 from: u32 = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.table;550 if (!kayan_table_has(kayan_table_at(w, from), id)) {551 kayan_move(w, e, kayan_table_with(w, from, id));552 }553 bytes: []u8 = kayan_get_bytes(w, e, id);554 if (value.length != bytes.length) {555 printfmt("kayan: a {}-byte value for a component of {} bytes\n", value.length, bytes.length);556 exit(1);557 }558 for (i: u64 = 0; i < bytes.length; i += 1) {559 bytes[i] = value[i];560 }561 return kayan_status.ok;562}563 564kayan_remove_id: proc(w: *kayan_world, e: kayan_entity, id: u32) -> kayan_status = {565 if (!kayan_alive(w, e)) {566 return kayan_status.dead;567 }568 from: u32 = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.table;569 if (!kayan_table_has(kayan_table_at(w, from), id)) {570 return kayan_status.missing;571 }572 kayan_move(w, e, kayan_table_without(w, from, id));573 return kayan_status.ok;574}575 576// e's component `id` as its bytes: an empty slice if e is dead or has no such577// component. The typed layer is for game code, which knows its types; this is578// for tools that work from reflect records instead.579kayan_get_bytes: proc(w: *const kayan_world, e: kayan_entity, id: u32) -> []u8 in level = {580 none: []u8 in level = {};581 if (!kayan_alive(w, e) || id >= w*.component_count) {582 return none;583 }584 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));585 table: u32 = slot*.table;586 row: u32 = slot*.row;587 t: *kayan_table in level = kayan_table_at(w, table);588 c: option<*kayan_column in level> = t*.columns[id];589 if (!c) {590 return none;591 }592 return kayan_column_bytes(c, cast(row, u64));593}594 595// ---------------------------------------------------------------------------596// The typed layer597// ---------------------------------------------------------------------------598 599// e's T, or null if e is dead or has no T. Handle, slot, table, row, typed600// view: the same path as 1.4's get_position, for any component. `option<*T>` says the601// null is possible, so the caller tests before it reads. The pointer is good602// until the next structural change, so use it now and keep the handle.603kayan_get: proc<T>(w: *const kayan_world, e: kayan_entity) -> option<*T in level> = {604 if (!kayan_alive(w, e)) {605 return null;606 }607 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));608 table: u32 = slot*.table;609 row: u32 = slot*.row;610 t: *kayan_table in level = kayan_table_at(w, table);611 if (!kayan_table_has(t, kayan_id_of<T>())) {612 return null;613 }614 rows: []T in level = kayan_slice<T>(t);615 return rows[cast(row, u64)]&;616}617 618kayan_has: proc<T>(w: *const kayan_world, e: kayan_entity) -> b32 = {619 return (kayan_signature(w, e) & kayan_mask_of<T>()) != 0;620}621 622// Write e's T. It must already have one: giving an entity a component it does623// not have changes its table, which is 2.6.624kayan_set: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> kayan_status = {625 if (!kayan_alive(w, e)) {626 return kayan_status.dead;627 }628 p: option<*T> = kayan_get<T>(w, e);629 if (!p) {630 return kayan_status.missing;631 }632 p* = value;633 return kayan_status.ok;634}635 636// Give e a T. If it already has one, this is kayan_set; if not, it moves to637// the table with T and then sets it.638kayan_add: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> kayan_status = {639 if (!kayan_alive(w, e)) {640 return kayan_status.dead;641 }642 if (!kayan_has<T>(w, e)) {643 from: u32 = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.table;644 kayan_move(w, e, kayan_table_with(w, from, kayan_id_of<T>()));645 }646 return kayan_set<T>(w, e, value);647}648 649// Take e's T away: the byte proc, with the type turned into an id.650kayan_remove: proc<T>(w: *kayan_world, e: kayan_entity) -> kayan_status = {651 return kayan_remove_id(w, e, kayan_id_of<T>());652}653 654// The invariant, over every table: each row belongs to a live entity whose655// slot points back at that table and row, and the rows of all tables add up to656// the live count.657kayan_verify: proc(w: *const kayan_world) -> u64 = {658 problems: u64 = 0;659 rows: u64 = 0;660 for (ti: u64 = 0; ti < w*.tables*.length; ti += 1) {661 t: *kayan_table in level = kayan_table_at(w, cast(ti, u32));662 for (row: u64 = 0; row < t*.count; row += 1) {663 // Taken row by row: kayan_alive takes the world, and the checker664 // does not know that it leaves the rows where they are.665 es: []kayan_entity = kayan_entities(t);666 e: kayan_entity = es[row];667 if (!kayan_alive(w, e)) {668 printfmt("kayan_verify: table {} row {} holds dead entity {}:{}\n", ti, row, e.index, e.generation);669 problems += 1;670 continue;671 }672 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));673 if (cast(slot*.table, u64) != ti || cast(slot*.row, u64) != row) {674 printfmt("kayan_verify: {}:{} is in table {} row {}, its slot says table {} row {}\n",675 e.index, e.generation, ti, row, slot*.table, slot*.row);676 problems += 1;677 }678 }679 rows += t*.count;680 }681 if (rows != w*.live) {682 printfmt("kayan_verify: {} rows in tables, {} live entities\n", rows, w*.live);683 problems += 1;684 }685 return problems;686}687 688kayan_print_signature: proc(w: *const kayan_world, signature: u64) -> void = {689 printfmt("{");690 first: b32 = true;691 for (id: u32 = 0; id < w*.component_count; id += 1) {692 if ((signature & kayan_bit(id)) != 0) {693 if (!first) {694 printfmt(" ");695 }696 printfmt("{}", kayan_info(w, id)*.name);697 first = false;698 }699 }700 printfmt("}");701}702 703kayan_print_table: proc(w: *const kayan_world, ti: u32) -> void = {704 t: *kayan_table in level = kayan_table_at(w, ti);705 printfmt(" table {} ", ti);706 kayan_print_signature(w, t*.signature);707 printfmt(":");708 es: []kayan_entity = kayan_entities(t);709 for (row: u64 = 0; row < es.length; row += 1) {710 e: kayan_entity = es[row];711 printfmt(" {}:{}", e.index, e.generation);712 }713 printfmt("\n");714}715 716kayan_print_tables: proc(w: *const kayan_world) -> void = {717 for (ti: u32 = 0; ti < cast(w*.tables*.length, u32); ti += 1) {718 kayan_print_table(w, ti);719 }720}721 722// ---------------------------------------------------------------------------723// The inspector: reflect records, no component types724// ---------------------------------------------------------------------------725 726kayan_indent: proc(depth: u32) -> void = {727 for (i: u32 = 0; i < depth; i += 1) {728 printfmt(" ");729 }730}731 732// One value of a builtin type, read from its bytes by its type's name. The733// bytes may sit at any address, so the read is a copy into a local: this is the734// inspector's one unsafe block.735kayan_print_builtin: proc(type_name: ^const char, at: []const u8) -> void = {736 if (at.length < 4) {737 printfmt("<{} bytes>", at.length);738 return;739 }740 unsafe {741 if (reflect_cstr_equal(type_name, "f32")) {742 f: f32 = 0.0;743 memcpy(f&, at.data, 4);744 printfmt("{:.2}", cast(f, f64));745 } else if (reflect_cstr_equal(type_name, "u32")) {746 u: u32 = 0;747 memcpy(u&, at.data, 4);748 printfmt("{}", u);749 } else if (reflect_cstr_equal(type_name, "i32")) {750 i: i32 = 0;751 memcpy(i&, at.data, 4);752 printfmt("{}", i);753 } else {754 printfmt("<{}>", type_name);755 }756 }757}758 759// Every field of a struct whose bytes are `base`, nested structs indented760// under their field. A field's bytes are cut from `base` by its offset and761// size, both from the reflect record.762kayan_print_fields: proc(type: *const reflect, base: []const u8, depth: u32) -> void = {763 for (i: u64 = 0; i < type*.count; i += 1) {764 f: option<*const reflect_field> = reflect_field_at(type, i);765 if (!f) {766 continue;767 }768 if (reflect_field_has_attr(f, "hidden")) {769 continue;770 }771 at: []const u8 = base[f*.offset..f*.offset + f*.size];772 kayan_indent(depth);773 printfmt("{}:", f*.name);774 info: option<*const reflect> = reflect_field_info(f);775 if (info && info*.kind == reflect_kind_struct) {776 printfmt("\n");777 kayan_print_fields(info, at, depth + 1);778 } else {779 printfmt(" ");780 kayan_print_builtin(f*.type, at);781 printfmt("\n");782 }783 }784}785 786kayan_inspect: proc(w: *const kayan_world, e: kayan_entity) -> void = {787 if (!kayan_alive(w, e)) {788 printfmt("{}:{} (dead)\n", e.index, e.generation);789 return;790 }791 printfmt("{}:{} ", e.index, e.generation);792 signature: u64 = kayan_signature(w, e);793 kayan_print_signature(w, signature);794 printfmt("\n");795 for (id: u32 = 0; id < w*.component_count; id += 1) {796 if ((signature & kayan_bit(id)) != 0) {797 info: *const reflect in permanent = kayan_info(w, id);798 printfmt(" {}\n", info*.name);799 bytes: []u8 = kayan_get_bytes(w, e, id);800 kayan_print_fields(info, bytes, 2);801 }802 }803}804 805// Every table: rows in use out of room, and per column its row size and its806// bytes in use out of bytes allocated. The memory a table holds and does not807// use is the cost of doubling on growth.808kayan_print_layout: proc(w: *const kayan_world) -> void = {809 for (ti: u32 = 0; ti < cast(w*.tables*.length, u32); ti += 1) {810 t: *kayan_table in level = kayan_table_at(w, ti);811 printfmt("table {} ", ti);812 kayan_print_signature(w, t*.signature);813 printfmt(": {} rows, room for {}\n", cast(t*.count, u32), cast(t*.capacity, u32));814 for (id: u32 = 0; id < w*.component_count; id += 1) {815 c: option<*kayan_column in level> = t*.columns[id];816 if (c) {817 size: u64 = c*.size;818 printfmt(" {:-9} {:2} bytes a row, {:4} of {:4} bytes in use\n", kayan_info(w, id)*.name, cast(size, u32),819 cast(size * t*.count, u32), cast(size * t*.capacity, u32));820 }821 }822 }823}824 825// ---------------------------------------------------------------------------826// Queries827// ---------------------------------------------------------------------------828 829// The tables a system runs over: every table whose signature has all of `all`830// and none of `none`, kept by index. Tables are only ever added, at the end of831// w.tables, so the query remembers how many it has checked and, brought up to832// date, looks at the new ones only.833kayan_query: struct = {834 all: u64;835 none: u64;836 tables: *vec<u32> in level;837 checked: u64;838}839 840// The list of matched tables lives in the world's region, so a query says841// which region it is made in.842kayan_query_init: proc(r: region level, all: u64, none: u64) -> kayan_query = {843 return { .all = all, .none = none, .tables = vec<u32>make(r, 4) };844}845 846kayan_matches: proc(signature: u64, all: u64, none: u64) -> b32 = {847 return (signature & all) == all && (signature & none) == 0;848}849 850// Check the tables made since the query last looked. With none new, this is851// one comparison, so call it before every walk.852kayan_query_update: proc(w: *const kayan_world, q: *kayan_query) -> void = {853 for (ti: u64 = q*.checked; ti < w*.tables*.length; ti += 1) {854 if (kayan_matches(kayan_table_at(w, cast(ti, u32))*.signature, q*.all, q*.none)) {855 _ = vec<u32>push(q*.tables, cast(ti, u32));856 }857 }858 q*.checked = w*.tables*.length;859}860 861// The query's i-th table.862kayan_query_table: proc(w: *const kayan_world, q: *const kayan_query, i: u64) -> *kayan_table in level = {863 return kayan_table_at(w, vec<u32>at(q*.tables, i)*);864}865 866// How many entities the query matches: the rows of its tables.867kayan_query_count: proc(w: *const kayan_world, q: *kayan_query) -> u64 = {868 kayan_query_update(w, q);869 n: u64 = 0;870 for (i: u64 = 0; i < q*.tables*.length; i += 1) {871 n += kayan_query_table(w, q, i)*.count;872 }873 return n;874}875 876kayan_print_query: proc(w: *const kayan_world, name: ^const char, q: *kayan_query) -> void = {877 kayan_query_update(w, q);878 printfmt("{}: all ", name);879 kayan_print_signature(w, q*.all);880 printfmt(", none ");881 kayan_print_signature(w, q*.none);882 printfmt("; tables: {}, entities: {}\n", q*.tables*.length, kayan_query_count(w, q));883 for (i: u64 = 0; i < q*.tables*.length; i += 1) {884 kayan_print_table(w, vec<u32>at(q*.tables, i)*);885 }886}887 888// ---------------------------------------------------------------------------889// Systems890// ---------------------------------------------------------------------------891 892// ---------------------------------------------------------------------------893// Commands894// ---------------------------------------------------------------------------895 896// The frame's command buffer: the commands, and the frame arena their values897// are copied into. Made at the start of each frame, in the frame region, and898// gone with it.899kayan_commands: struct = {900 heap: *kayan_heap in frame;901 list: *vec<kayan_command> in frame;902}903 904kayan_commands_make: proc(r: region frame) -> *kayan_commands in frame = {905 return new<kayan_commands>(r, { .heap = kayan_heap_make(r), .list = vec<kayan_command>make(r, 16) });906}907 908// A spawn that has been recorded and not applied. There is no entity yet, so909// this is the spawn command's place in the list; give it components with910// kayan_pending_set.911kayan_pending: struct = {912 command: u32;913}914 915kayan_defer: proc(cmds: *kayan_commands, c: kayan_command) -> u32 = {916 index: u32 = cast(cmds*.list*.length, u32);917 _ = vec<kayan_command>push(cmds*.list, c);918 return index;919}920 921// A copy of `value`, as bytes, in the frame arena. The one trusted proc of922// the command buffer: a value's bytes are a cast.923kayan_defer_value: proc<T>(h: *kayan_heap in frame, value: T) -> []u8 in frame = {924 unsafe {925 data: ^u8 = cast(memops_arena_push(h*.arena[0]&, sizeof(T), alignof(T)), ^u8);926 memcpy(data, value&, sizeof(T));927 return cast(data[0..sizeof(T)], []u8 in frame);928 }929}930 931kayan_defer_spawn: proc(cmds: *kayan_commands, signature: u64) -> kayan_pending = {932 p: kayan_pending = {};933 p.command = kayan_defer(cmds, { .kind = kayan_command_kind.spawn, .signature = signature });934 return p;935}936 937kayan_pending_set: proc<T>(cmds: *kayan_commands, p: kayan_pending, value: T) -> void = {938 _ = kayan_defer(cmds, { .kind = kayan_command_kind.add, .spawn = p.command + 1, .id = kayan_id_of<T>(),939 .value = kayan_defer_value<T>(cmds*.heap, value) });940}941 942kayan_defer_despawn: proc(cmds: *kayan_commands, e: kayan_entity) -> void = {943 _ = kayan_defer(cmds, { .kind = kayan_command_kind.despawn, .entity = e });944}945 946kayan_defer_add: proc<T>(cmds: *kayan_commands, e: kayan_entity, value: T) -> void = {947 _ = kayan_defer(cmds, { .kind = kayan_command_kind.add, .entity = e, .id = kayan_id_of<T>(),948 .value = kayan_defer_value<T>(cmds*.heap, value) });949}950 951kayan_defer_remove: proc<T>(cmds: *kayan_commands, e: kayan_entity) -> void = {952 _ = kayan_defer(cmds, { .kind = kayan_command_kind.remove, .entity = e, .id = kayan_id_of<T>() });953}954 955// One applied command. The components given to a new entity are part of its956// spawn line, so they are not printed on their own.957kayan_print_command: proc(w: *const kayan_world, c: *const kayan_command, status: kayan_status) -> void = {958 if (c*.spawn != 0 && status == kayan_status.ok) {959 return;960 }961 printfmt(" {} ", c*.kind);962 if (c*.kind == kayan_command_kind.spawn) {963 kayan_print_signature(w, c*.signature);964 printfmt(" -> ");965 } else if (c*.kind != kayan_command_kind.despawn) {966 printfmt("{}, ", kayan_info(w, c*.id)*.name);967 }968 printfmt("{}:{}", c*.entity.index, c*.entity.generation);969 if (status != kayan_status.ok) {970 printfmt(": skipped, {}", status);971 }972 printfmt("\n");973}974 975// Apply the frame's commands in the order they were recorded. A command for976// an entity that died after it was recorded (despawned by an earlier command,977// most likely) does nothing, and is counted. Letting the commands go is the978// frame region's end, not a line here.979kayan_flush: proc(w: *kayan_world, cmds: *kayan_commands) -> void = {980 for (i: u64 = 0; i < cmds*.list*.length; i += 1) {981 c: *kayan_command = vec<kayan_command>at(cmds*.list, i);982 if (c*.spawn != 0) {983 c*.entity = vec<kayan_command>at(cmds*.list, cast(c*.spawn - 1, u64))*.entity;984 }985 status: kayan_status = kayan_status.ok;986 if (c*.kind == kayan_command_kind.spawn) {987 c*.entity = kayan_spawn(w, c*.signature);988 } else if (c*.kind == kayan_command_kind.despawn) {989 status = kayan_despawn(w, c*.entity);990 } else if (c*.kind == kayan_command_kind.add) {991 status = kayan_add_id(w, c*.entity, c*.id, c*.value);992 } else {993 status = kayan_remove_id(w, c*.entity, c*.id);994 }995 if (status == kayan_status.ok) {996 w*.applied += 1;997 } else {998 w*.skipped += 1;999 }1000 if (w*.trace) {1001 kayan_print_command(w, c, status);1002 }1003 }1004}1005 1006// ---------------------------------------------------------------------------1007// Systems1008// ---------------------------------------------------------------------------1009 1010// What a system gets for each table it runs on: the world, the table, the1011// frame's time step, the frame's command buffer, and `ctx`: whatever the1012// system needs beyond its own tables, here the game's state. It is a typed1013// pointer, not a `*void`: the checker has to know what it points at, and the1014// systems that use it test it. Every pointer names its region, so the struct1015// can be made wherever frame is open.1016kayan_iter: struct = {1017 world: *kayan_world in level;1018 table: *kayan_table in level;1019 dt: f32;1020 commands: *kayan_commands in frame;1021 ctx: option<*game in level>;1022}1023 1024kayan_system_proc: alias = *proc(it: *kayan_iter) -> void;1025 1026// A system is code: a name, and the proc run on each table its query matches.1027// A C string and a proc pointer are both pointers without a region, so a1028// struct holding them is a carrier: safe code reaches it only through a1029// pointer into a region, made by a trusted proc. Neither changes after it is1030// made, so nothing is lost.1031kayan_system: struct = {1032 name: ^const char;1033 run: kayan_system_proc;1034}1035 1036kayan_system_make: proc(r: region level, name: ^const char, run: kayan_system_proc) -> *kayan_system in level = {1037 unsafe {1038 s: *kayan_system in level = cast(memops_arena_push_zero(r, sizeof(kayan_system), alignof(kayan_system)), *kayan_system in level);1039 s*.name = name;1040 s*.run = run;1041 return s;1042 }1043}1044 1045// A system in the schedule: which system, the query that finds its tables,1046// and what it has run over so far. `tables` and `rows` count the calls.1047kayan_scheduled: struct = {1048 system: *kayan_system in level;1049 query: kayan_query;1050 ctx: option<*game in level>;1051 tables: u64;1052 rows: u64;1053}1054 1055// The schedule is a list of these, in the order the frame runs them. Each1056// entry is its own allocation so a pointer to one is a pointer into level,1057// which the checker never counts as stale.1058kayan_schedule_add: proc(r: region level, systems: *vec<*kayan_scheduled in level>, name: ^const char,1059 all: u64, none: u64, run: kayan_system_proc, ctx: option<*game in level>) -> void = {1060 _ = vec<*kayan_scheduled in level>push(systems, new<kayan_scheduled>(r, {1061 .system = kayan_system_make(r, name, run),1062 .query = kayan_query_init(r, all, none),1063 .ctx = ctx,1064 }));1065}1066 1067kayan_scheduled_at: proc(systems: *const vec<*kayan_scheduled in level>, s: u64) -> *kayan_scheduled in level = {1068 return vec<*kayan_scheduled in level>at_const(systems, s)*;1069}1070 1071// One frame: every system in order, each over every table it matches that has1072// rows, then the commands they recorded. A system sees every value the1073// systems before it wrote this frame, and no structural change until the1074// next. The caller opens the frame region and makes the buffer in it.1075kayan_progress: proc(w: *kayan_world in level, systems: *const vec<*kayan_scheduled in level>, dt: f32,1076 cmds: *kayan_commands in frame) -> void = {1077 for (s: u64 = 0; s < systems*.length; s += 1) {1078 sys: *kayan_scheduled in level = kayan_scheduled_at(systems, s);1079 kayan_query_update(w, sys*.query&);1080 for (i: u64 = 0; i < sys*.query.tables*.length; i += 1) {1081 it: kayan_iter = { .world = w, .table = kayan_query_table(w, sys*.query&, i), .dt = dt,1082 .commands = cmds, .ctx = sys*.ctx };1083 if (it.table*.count > 0) {1084 sys*.tables += 1;1085 sys*.rows += it.table*.count;1086 sys*.system*.run(it&);1087 }1088 }1089 }1090 kayan_flush(w, cmds);1091}1092 1093kayan_print_schedule: proc(w: *const kayan_world, systems: *const vec<*kayan_scheduled in level>) -> void = {1094 for (s: u64 = 0; s < systems*.length; s += 1) {1095 sys: *kayan_scheduled in level = kayan_scheduled_at(systems, s);1096 kayan_print_query(w, sys*.system*.name, sys*.query&);1097 }1098}1099 1100// ---------------------------------------------------------------------------1101// The game1102// ---------------------------------------------------------------------------1103 1104// What the game's systems share: the drones, for the systems that look for1105// them, and the score. Systems get it through their ctx; it lives in level,1106// so an iterator can hold it.1107game: struct = {1108 drones: kayan_query;1109 time: f32;1110 fired: u32;1111 hits: u32;1112 fires: u32;1113 kills: u32;1114 breaches: u32;1115 came: u32;1116}1117 1118// Kernels: plain loops over plain slices, as in 3.2.1119integrate: proc[unsafe](ps: noalias []position, vs: []const velocity, dt: f32) -> void = {1120 for (i: u64 = 0; i < ps.length; i += 1) {1121 ps[i].at.x += vs[i].per_second.x * dt;1122 ps[i].at.y += vs[i].per_second.y * dt;1123 }1124}1125 1126burn: proc[unsafe](hs: noalias []health, fires: []const burning, dt: f32) -> void = {1127 for (i: u64 = 0; i < hs.length; i += 1) {1128 hs[i].points -= fires[i].per_second * dt;1129 }1130}1131 1132age: proc[unsafe](ls: noalias []lifetime, dt: f32) -> void = {1133 for (i: u64 = 0; i < ls.length; i += 1) {1134 ls[i].seconds -= dt;1135 }1136}1137 1138bullet_signature: proc() -> u64 = {1139 return kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<damage>() | kayan_mask_of<lifetime>();1140}1141 1142drone_signature: proc() -> u64 = {1143 return kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<health>();1144}1145 1146// Is there a drone in the lane at height y, ahead of x?1147drone_ahead: proc(w: *const kayan_world, g: *game, x: f32, y: f32) -> b32 = {1148 for (i: u64 = 0; i < g*.drones.tables*.length; i += 1) {1149 ps: []position = kayan_slice<position>(kayan_query_table(w, g*.drones&, i));1150 for (row: u64 = 0; row < ps.length; row += 1) {1151 dy: f32 = ps[row].at.y - y;1152 if (dy * dy < 0.25 && ps[row].at.x > x) {1153 return true;1154 }1155 }1156 }1157 return false;1158}1159 1160// A turret whose timer is up fires, if it has something to fire at.1161turret_system: proc(it: *kayan_iter) -> void = {1162 g: option<*game in level> = it*.ctx;1163 if (!g) {1164 return;1165 }1166 ps: []position = kayan_slice<position>(it*.table);1167 ts: []turret = kayan_slice<turret>(it*.table);1168 kayan_query_update(it*.world, g*.drones&);1169 for (row: u64 = 0; row < ts.length; row += 1) {1170 if (ts[row].timer <= 0.0 && drone_ahead(it*.world, g, ps[row].at.x, ps[row].at.y)) {1171 shot: kayan_pending = kayan_defer_spawn(it*.commands, bullet_signature());1172 kayan_pending_set<position>(it*.commands, shot, ps[row]);1173 kayan_pending_set<velocity>(it*.commands, shot, { .per_second = { .x = 16.0, .y = 0.0 } });1174 kayan_pending_set<damage>(it*.commands, shot, { .amount = 25.0 });1175 kayan_pending_set<lifetime>(it*.commands, shot, { .seconds = 2.0 });1176 ts[row].timer = ts[row].cooldown;1177 g*.fired += 1;1178 }1179 if (ts[row].timer > 0.0) {1180 ts[row].timer -= it*.dt;1181 }1182 }1183}1184 1185age_system: proc(it: *kayan_iter) -> void = {1186 unsafe { age(kayan_slice<lifetime>(it*.table), it*.dt); }1187}1188 1189expire_system: proc(it: *kayan_iter) -> void = {1190 ls: []lifetime = kayan_slice<lifetime>(it*.table);1191 es: []kayan_entity = kayan_entities(it*.table);1192 for (row: u64 = 0; row < es.length; row += 1) {1193 if (ls[row].seconds <= 0.0) {1194 kayan_defer_despawn(it*.commands, es[row]);1195 }1196 }1197}1198 1199move_system: proc(it: *kayan_iter) -> void = {1200 unsafe { integrate(kayan_slice<position>(it*.table), kayan_slice<velocity>(it*.table), it*.dt); }1201}1202 1203// A bullet within three quarters of a unit of a drone in its lane hits it. A1204// bullet moves a unit a frame and a drone at most a quarter, so none passes1205// through a drone between two frames without being this close in one of them.1206hit_system: proc(it: *kayan_iter) -> void = {1207 w: *kayan_world in level = it*.world;1208 g: option<*game in level> = it*.ctx;1209 if (!g) {1210 return;1211 }1212 ps: []position = kayan_slice<position>(it*.table);1213 ds: []damage = kayan_slice<damage>(it*.table);1214 es: []kayan_entity = kayan_entities(it*.table);1215 kayan_query_update(w, g*.drones&);1216 for (b: u64 = 0; b < es.length; b += 1) {1217 for (i: u64 = 0; i < g*.drones.tables*.length; i += 1) {1218 t: *kayan_table in level = kayan_query_table(w, g*.drones&, i);1219 tps: []position = kayan_slice<position>(t);1220 ths: []health = kayan_slice<health>(t);1221 for (row: u64 = 0; row < tps.length; row += 1) {1222 dx: f32 = tps[row].at.x - ps[b].at.x;1223 dy: f32 = tps[row].at.y - ps[b].at.y;1224 if (dx * dx < 0.5625 && dy * dy < 0.25) {1225 ths[row].points -= ds[b].amount;1226 kayan_defer_despawn(it*.commands, es[b]);1227 g*.hits += 1;1228 }1229 }1230 }1231 }1232}1233 1234// Below half health, a drone catches fire: a component added, so a command.1235ignite_system: proc(it: *kayan_iter) -> void = {1236 g: option<*game in level> = it*.ctx;1237 if (!g) {1238 return;1239 }1240 hs: []health = kayan_slice<health>(it*.table);1241 es: []kayan_entity = kayan_entities(it*.table);1242 for (row: u64 = 0; row < es.length; row += 1) {1243 if (hs[row].points < 0.5 * hs[row].max && hs[row].points > 0.0) {1244 kayan_defer_add<burning>(it*.commands, es[row], { .per_second = 10.0 });1245 g*.fires += 1;1246 }1247 }1248}1249 1250burn_system: proc(it: *kayan_iter) -> void = {1251 unsafe { burn(kayan_slice<health>(it*.table), kayan_slice<burning>(it*.table), it*.dt); }1252}1253 1254// The entity and its values are copied out before the line is printed: a1255// print of a slice element in a loop reads as a change to the checker.1256die_system: proc(it: *kayan_iter) -> void = {1257 g: option<*game in level> = it*.ctx;1258 if (!g) {1259 return;1260 }1261 hs: []health = kayan_slice<health>(it*.table);1262 es: []kayan_entity = kayan_entities(it*.table);1263 for (row: u64 = 0; row < es.length; row += 1) {1264 e: kayan_entity = es[row];1265 points: f32 = hs[row].points;1266 if (points <= 0.0) {1267 kayan_defer_despawn(it*.commands, e);1268 g*.kills += 1;1269 printfmt("{:6.2}s {}:{} destroyed\n", cast(g*.time, f64), e.index, e.generation);1270 }1271 }1272}1273 1274// A drone still alive at the left end has reached the base.1275breach_system: proc(it: *kayan_iter) -> void = {1276 g: option<*game in level> = it*.ctx;1277 if (!g) {1278 return;1279 }1280 ps: []position = kayan_slice<position>(it*.table);1281 hs: []health = kayan_slice<health>(it*.table);1282 es: []kayan_entity = kayan_entities(it*.table);1283 for (row: u64 = 0; row < es.length; row += 1) {1284 e: kayan_entity = es[row];1285 at: vec2 = ps[row].at;1286 points: f32 = hs[row].points;1287 if (at.x <= 0.0 && points > 0.0) {1288 kayan_defer_despawn(it*.commands, e);1289 g*.breaches += 1;1290 printfmt("{:6.2}s {}:{} reached the base in lane {:.0}\n", cast(g*.time, f64),1291 e.index, e.generation, cast(at.y, f64));1292 }1293 }1294}1295 1296// ---------------------------------------------------------------------------1297// The level: waves, and a picture of the lanes1298// ---------------------------------------------------------------------------1299 1300lanes: u32 = 3;1301arena_end: f32 = 32.0;1302 1303spawn_turret: proc(w: *kayan_world, lane: u32) -> kayan_entity = {1304 e: kayan_entity = kayan_spawn(w, kayan_mask_of<position>() | kayan_mask_of<turret>());1305 _ = kayan_set<position>(w, e, { .at = { .x = 0.0, .y = cast(lane, f32) } });1306 // Each lane's turret starts a quarter second after the one below it.1307 _ = kayan_set<turret>(w, e, { .cooldown = 0.75, .timer = 0.25 * cast(lane, f32) });1308 return e;1309}1310 1311spawn_drone: proc(w: *kayan_world, x: f32, lane: u32, speed: f32) -> kayan_entity = {1312 e: kayan_entity = kayan_spawn(w, drone_signature());1313 _ = kayan_set<position>(w, e, { .at = { .x = x, .y = cast(lane, f32) } });1314 _ = kayan_set<velocity>(w, e, { .per_second = { .x = -speed, .y = 0.0 } });1315 _ = kayan_set<health>(w, e, { .points = 100.0, .max = 100.0 });1316 return e;1317}1318 1319// Wave k: `wave_rows[k]` drones, two units apart, in each lane whose bit is1320// set in `wave_lanes[k]`, all at one speed.1321wave_lanes: [8]u32 = { 2, 5, 7, 2, 7, 5, 7, 7 };1322wave_rows: [8]u32 = { 1, 1, 1, 3, 1, 2, 2, 2 };1323wave_speed: [8]f32 = { 2.0, 2.0, 2.0, 2.0, 4.0, 4.0, 2.0, 4.0 };1324 1325send_wave: proc(w: *kayan_world, g: *game, k: u32) -> void = {1326 printfmt("{:6.2}s wave {}:", cast(g*.time, f64), k + 1);1327 for (lane: u32 = 0; lane < lanes; lane += 1) {1328 if ((wave_lanes[k] & (1'u32 shl lane)) != 0) {1329 for (r: u32 = 0; r < wave_rows[k]; r += 1) {1330 e: kayan_entity = spawn_drone(w, arena_end + 2.0 * cast(r, f32), lane, wave_speed[k]);1331 printfmt(" {}:{}", e.index, e.generation);1332 g*.came += 1;1333 }1334 }1335 }1336 printfmt(", {:.0} units a second\n", cast(wave_speed[k], f64));1337}1338 1339// What an entity is drawn as follows from its table: one check per table,1340// not per entity.1341glyph_of: proc(signature: u64) -> c8 = {1342 if ((signature & kayan_mask_of<turret>()) != 0) {1343 return 'T';1344 }1345 if ((signature & kayan_mask_of<burning>()) != 0) {1346 return '*';1347 }1348 if ((signature & kayan_mask_of<health>()) != 0) {1349 return 'd';1350 }1351 return '-';1352}1353 1354// The lanes as text, lane 2 at the top. Every entity with a position, by way1355// of a query.1356draw: proc(w: *const kayan_world, g: *const game, everything: *kayan_query) -> void = {1357 cells: [3][33]c8 = {};1358 for (lane: u32 = 0; lane < lanes; lane += 1) {1359 for (x: u32 = 0; x < 33; x += 1) {1360 cells[lane][x] = '.';1361 }1362 }1363 kayan_query_update(w, everything);1364 for (i: u64 = 0; i < everything*.tables*.length; i += 1) {1365 t: *kayan_table in level = kayan_query_table(w, everything, i);1366 glyph: c8 = glyph_of(t*.signature);1367 ps: []position = kayan_slice<position>(t);1368 for (row: u64 = 0; row < ps.length; row += 1) {1369 x: i32 = cast(ps[row].at.x, i32);1370 lane: i32 = cast(ps[row].at.y, i32);1371 if (x >= 0 && x <= 32 && lane >= 0 && lane < 3) {1372 cells[lane][x] = glyph;1373 }1374 }1375 }1376 printfmt("{:6.2}s {} entities in {} tables\n", cast(g*.time, f64), cast(w*.live, u32), cast(w*.tables*.length, u32));1377 for (lane: u32 = lanes; lane > 0; lane -= 1) {1378 printfmt(" {} ", lane - 1);1379 for (x: u32 = 0; x < 33; x += 1) {1380 printfmt("{}", cells[lane - 1][x]);1381 }1382 printfmt("\n");1383 }1384}1385 1386// One frame of the world: the frame region opens, the buffer is made in it,1387// the systems run and their commands are applied, and the region ends.1388run_frame: proc(w: *kayan_world in level, systems: *const vec<*kayan_scheduled in level>, dt: f32) -> void = {1389 frame: region = {1390 kayan_progress(w, systems, dt, kayan_commands_make(frame));1391 }1392}1393 1394main: proc() -> i32 = {1395 level: region = {1396 w: *kayan_world in level = kayan_world_make(level);1397 kayan_register(w, kayan_id_of<position>(), position<>&);1398 kayan_register(w, kayan_id_of<velocity>(), velocity<>&);1399 kayan_register(w, kayan_id_of<health>(), health<>&);1400 kayan_register(w, kayan_id_of<damage>(), damage<>&);1401 kayan_register(w, kayan_id_of<lifetime>(), lifetime<>&);1402 kayan_register(w, kayan_id_of<turret>(), turret<>&);1403 kayan_register(w, kayan_id_of<burning>(), burning<>&);1404 1405 g: *game in level = new<game>(level, {1406 .drones = kayan_query_init(level, kayan_mask_of<position>() | kayan_mask_of<health>(), 0),1407 });1408 everything: kayan_query = kayan_query_init(level, kayan_mask_of<position>(), 0);1409 for (lane: u32 = 0; lane < lanes; lane += 1) {1410 _ = spawn_turret(w, lane);1411 }1412 1413 // The schedule. Each system reads what the ones before it wrote: bullets1414 // hit where they are after moving, a drone catches fire and is destroyed1415 // in the frame its health drops, and only a drone that survived the frame1416 // can reach the base.1417 shooting: u64 = kayan_mask_of<position>() | kayan_mask_of<turret>();1418 bullets: u64 = kayan_mask_of<position>() | kayan_mask_of<damage>();1419 movers: u64 = kayan_mask_of<position>() | kayan_mask_of<velocity>();1420 systems: *vec<*kayan_scheduled in level> in level = vec<*kayan_scheduled in level>make(level, 9);1421 kayan_schedule_add(level, systems, "turret", shooting, 0, turret_system, g);1422 kayan_schedule_add(level, systems, "age", kayan_mask_of<lifetime>(), 0, age_system, null);1423 kayan_schedule_add(level, systems, "expire", kayan_mask_of<lifetime>(), 0, expire_system, null);1424 kayan_schedule_add(level, systems, "move", movers, 0, move_system, null);1425 kayan_schedule_add(level, systems, "hit", bullets, 0, hit_system, g);1426 kayan_schedule_add(level, systems, "ignite", kayan_mask_of<health>(), kayan_mask_of<burning>(), ignite_system, g);1427 kayan_schedule_add(level, systems, "burn", kayan_mask_of<health>() | kayan_mask_of<burning>(), 0, burn_system, null);1428 kayan_schedule_add(level, systems, "die", kayan_mask_of<health>(), 0, die_system, g);1429 kayan_schedule_add(level, systems, "breach", kayan_mask_of<position>() | kayan_mask_of<health>(), 0, breach_system, g);1430 1431 dt: f32 = 0.0625;1432 frames: u32 = 512;1433 problems: u64 = 0;1434 most: u64 = 0;1435 for (frame_no: u32 = 0; frame_no < frames; frame_no += 1) {1436 g*.time = cast(frame_no, f32) * dt;1437 if (frame_no % 32 == 0 && frame_no / 32 < 8) {1438 send_wave(w, g, frame_no / 32);1439 }1440 run_frame(w, systems, dt);1441 problems += kayan_verify(w);1442 most = w*.live > most ? w*.live : most;1443 if (frame_no % 64 == 63) {1444 g*.time = cast(frame_no + 1, f32) * dt;1445 draw(w, g, everything&);1446 }1447 }1448 1449 kayan_query_update(w, g*.drones&);1450 flying: u64 = kayan_query_count(w, g*.drones&);1451 printfmt("\n{:.2} seconds, {} frames; kayan_verify after every one: {} problems\n",1452 cast(cast(frames, f32) * dt, f64), frames, cast(problems, u32));1453 printfmt("drones: {} came, {} destroyed, {} reached the base, {} still flying\n", g*.came, g*.kills, g*.breaches, cast(flying, u32));1454 printfmt("bullets: {} fired, {} hits; fires started: {}\n", g*.fired, g*.hits, g*.fires);1455 printfmt("entities: at most {} alive at once; commands: {} applied, {} skipped\n",1456 cast(most, u32), cast(w*.applied, u32), cast(w*.skipped, u32));1457 printfmt("\nthe schedule, and what each system ran over:\n");1458 for (s: u64 = 0; s < 9; s += 1) {1459 sys: *kayan_scheduled in level = kayan_scheduled_at(systems, s);1460 printfmt(" {:-6} called {:4} times, over {:5} rows\n", sys*.system*.name, cast(sys*.tables, u32), cast(sys*.rows, u32));1461 }1462 printfmt("\nthe tables at the end:\n");1463 kayan_print_tables(w);1464 }1465 return 0;1466}What game prints. tools/check.py fails the chapter if a single byte differs.
0.00s wave 1: 3:1, 2 units a second
2.00s wave 2: 7:1 8:1, 2 units a second
4.00s 12 entities in 4 tables
2 T......-...........-........d....
1 T..........-...........-*........
0 T......-...........-........d....
4.00s wave 3: 9:2 10:2 14:1, 2 units a second
4.00s 3:1 destroyed
5.75s 8:1 destroyed
5.75s 7:1 destroyed
6.00s wave 4: 7:2 8:2 3:3, 2 units a second
7.12s 10:2 destroyed
8.00s 15 entities in 4 tables
2 T..........-...........-d........
1 T..-...........-...........-d.d.d
0 T..........-...........-d........
8.00s wave 5: 13:3 16:2 17:1, 4 units a second
8.62s 14:1 destroyed
8.62s 9:2 destroyed
10.00s wave 6: 18:2 19:2 6:5 11:4, 4 units a second
10.00s 7:2 destroyed
11.38s 17:1 destroyed
11.38s 13:3 destroyed
11.81s 16:2 destroyed
12.00s 14 entities in 4 tables
2 T..-...........-........d.d......
1 T......-..............d.d........
0 T..-...........-........d.d......
12.00s wave 7: 16:3 5:7 14:4 15:4 7:3 4:6, 2 units a second
14.00s wave 8: 12:7 20:2 21:1 22:1 23:1 24:1, 4 units a second
14.19s 6:5 destroyed
14.19s 18:2 destroyed
14.75s 8:2 destroyed
16.00s 21 entities in 4 tables
2 T......-..d.............d.d......
1 T..........-....d.......d.d......
0 T......-..d.............d.d......
16.69s 11:4 destroyed
16.69s 19:2 destroyed
17.56s 3:3 destroyed
19.81s 23:1 destroyed
19.81s 12:7 destroyed
20.00s 14 entities in 4 tables
2 T.........d.....d.d..............
1 T..-....*.d.....d.d..............
0 T.........d.....d.d..............
20.19s 21:1 destroyed
22.31s 20:2 destroyed
22.31s 24:1 destroyed
22.44s 22:1 reached the base in lane 1
24.00s 12 entities in 4 tables
2 T..-....d.d......................
1 T......-d.d......................
0 T..-....d.d......................
25.31s 14:4 destroyed
25.56s 16:3 destroyed
25.56s 7:3 destroyed
28.00s 6 entities in 4 tables
2 T.*..............................
1 T.*..............................
0 T.*..............................
28.12s 15:4 destroyed
28.31s 4:6 destroyed
28.31s 5:7 destroyed
32.00s 3 entities in 4 tables
2 T................................
1 T................................
0 T................................
32.00 seconds, 512 frames; kayan_verify after every one: 0 problems
drones: 28 came, 27 destroyed, 1 reached the base, 0 still flying
bullets: 110 fired, 111 hits; fires started: 28
entities: at most 27 alive at once; commands: 716 applied, 1 skipped
the schedule, and what each system ran over:
turret called 512 times, over 1536 rows
age called 409 times, over 1622 rows
expire called 409 times, over 1622 rows
move called 1018 times, over 5051 rows
hit called 409 times, over 1622 rows
ignite called 443 times, over 3145 rows
burn called 166 times, over 284 rows
die called 609 times, over 3429 rows
breach called 609 times, over 3429 rows
the tables at the end:
table 0 {position turret}: 0:1 1:1 2:1
table 1 {position velocity health}:
table 2 {position velocity damage lifetime}:
table 3 {position velocity health burning}:
The dojo · thirteen building blocks
The dojo
dojo/README.md
An ECS is mostly a few small building blocks put together. None of them is specific to an ECS: a renderer's texture handles are a slot map, a particle system is swap-remove over packed arrays, undo and network replication are command buffers. The course meets each block for the first time in the middle of building something bigger, which is why its chapters can feel dense. The dojo takes the blocks out and gives each one two halves:
- a playground, to understand it: a small program that runs the block step by step and draws its insides, with one switch at the top that breaks it the way it most often breaks. Step through it in a debugger, flip the switch, watch what goes wrong.
- a kata, to own it: you write the block yourself, and a harness checks it. Reading and stepping through code feels like learning, but writing it from a blank page, getting it wrong, and doing it again a few days later is what makes it yours.
| block | what it is | in the course | |
|---|---|---|---|
| 1 | free list and generations | handles that are safe to keep after what they name is gone | 1.2, 1.3 |
| 2 | swap-remove and the back-reference | values packed with no holes, found by id: a sparse set | 1.4, 2.3, 2.6 |
| 3 | the slot map | 1 and 2 together: handles into packed values | 1.4, 2.4 |
| 4 | a type-erased column | an array of any type as bytes, viewed as a typed slice | 2.2, 2.3, 2.5 |
| 5 | a command buffer | structural changes recorded now and applied later | 3.3, 3.4 |
| 6 | bitsets | a set of components in a u64: has, all of, none of | 2.1, 3.1 |
| 7 | an arena | push, push, clear: memory for things that die together | every chapter |
| 8 | index, not pointer | links that stay right when their storage moves | 2.2, 2.4, 2.5 |
| 9 | interning | find or make, by key: a hash table that hands out dense ids | 2.4 |
| 10 | memoized edges | a slow lookup done once, then remembered on the edge | 2.6 |
| 11 | a reflection walk | reading, writing and comparing any value through its type's record | 2.7 |
| 12 | a watermark | looking only at what was added since last time | 3.1 |
| 13 | a dispatch table | behaviour as data: the schedule as a list of functions | 3.2 |
Do 1 to 5 in order the first time: 3 is 1 and 2 put together, and 5 applies its commands to something like 3. 6 to 13 come in the order the course meets them and stand on their own, except that 9, 10 and 12 work on 6's signatures. Any of them can come before the course, alongside it, or after it.
Each block
dojo/3.slot_map/
README.md the block, its invariant, the levels, the variations
play_slot_map.rin the playground
harness.rin fixed: a scenario, then a fuzz run against a model
level1.rin ... 3.rin where a kata starts
solution.rin the reference; don't peek
kata.rin yours: made by `start`, ignored by git
The harness doesn't know how your kata works, only what it must do. First it runs a short scenario, the playground's steps, and prints each as it passes. Then it runs 20,000 random operations from a seed against a simple model of the right answers, and checks after each one. A failure says which operation, which seed, and how to stop just before it in a debugger.
Levels, so the reps get harder:
- One function in an otherwise finished file: the one with the idea in it, usually the one people forget.
- Every function body, with the fields and signatures given. A proc that must return a real pointer or slice, a
_makeabove all, has a body to start from, since there's nothing shorter that compiles. Some of those are finished, and some only make the vecs: read what the comment asks for. - A blank page, with only a list of what the harness calls.
- Variations: the same harness, different insides, listed in each block's README.
Each README gives a target time for levels 1 to 3.
In the safe subset
Every kata, harness and playground here is checked by rin with its safety rules on, and so is your kata: rin compile refuses one that breaks them, and the runner prints what it said. It makes the katas look alike:
- A kata that keeps memory lives in a region.
level, which each harness opens inmainand hands to the scenario and the fuzz run, is where an arena would be. Blocks 5 and 11 also useframe, a shorter region opened once per game frame or fuzz step.dojo/dojo.rindeclares both. - **A kata is made whole by its
_makeproc**,handles_make(r: region level) -> *handles in level, and its vecs are fields of the form*vec<u32> in level. A vec knows its region, so a proc that pushes needs no region of its own. - **A proc that may find nothing returns
option<*T>**, and the caller tests it,if (p), before using it. - A pointer or slice that comes out of the kata's storage says where it points:
option<*u64 in level>,[]u64 in level. Without that the checker refuses thereturn:the result names no region, and may point into region 'level'. - A pointer into a vec is good until the vec may move. Keep one across a call that can change the vec, and the checker says
'n' may be stale: 'l', which it was taken from, has been changed since; take it again. Blocks 8 and 10 are about why. - **What safe code can't do is in
unsafe { }**, and only that: reading the command line and comparing C strings indojo.rin, three small blocks in block 4's column, the one proc in block 13 that keeps a name and a proc pointer, and a few helpers in the harnesses of 4, 11 and 13 and the playgrounds of 4, 11 and 13. The top of each file names them.
A block's README says where this changes how its kata is solved, most of all in 4 (you write three small unsafe { } blocks, and check what they rely on), 7 (the arena hands out offsets), 11 (the walk goes through the harness's helpers) and 13 (a system is kept behind a pointer into the region).
Watching a playground
python tools/dojo.py play slot_map
A playground runs one scenario, one operation per line of main, and prints what changed after each, so it works without a debugger. In one, put a breakpoint on a line of main and step over the operations. Where the block has slots there are only a few of them, in fixed arrays, so the watch window shows the whole state. From block 6 on, a verify proc runs after every operation, checks the block's invariant, and says what broke; a breakpoint inside its failure branch stops at the first operation that breaks it.
The switch is a global b32, read each time it's used, so besides editing it and running again, you can flip it mid-run: stop where the playground's header says, set it to 0 in the watch window, and continue. rin writes C, and the global keeps its name, so in gdb:
(gdb) break handles_free
(gdb) run
(gdb) set var bump_on_free = 0
(gdb) continue
Doing reps
python tools/dojo.py the blocks, your reps, and what is due
python tools/dojo.py play slot_map the playground
python tools/dojo.py start slot_map 2 a fresh kata.rin from level 2; the clock starts
python tools/dojo.py slot_map build the harness with your kata.rin and run it
A block can be named by its number, its name or the start of its name: 3, slot_map and slot are the same. While the kata fails, fix it and run it again. When it passes, the time since start goes into dojo/log.md, and the block is due again 1, 3, 7, 14 and then 30 days after each pass. python tools/dojo.py with no arguments says what is due. Spacing reps out is what makes them stick, much more than doing many in one day.
start won't overwrite a kata.rin you've changed unless you add --force. Your log is a plain markdown table; commit it or don't.
When a kata fails, the harness says where:
FAIL at operation 64 (seed 1): row 4 holds 43's value 37043, but get(43) gives 43020
debug it: build/dojo/swap_remove/harness --seed 1 --stop 64
with a breakpoint on dojo_stop_here, which runs just before that operation
Run that binary under a debugger with a breakpoint on dojo_stop_here. It stops just before the failing operation, with your structure in the state that leads to the failure. If the harness crashes instead, on a vec index past the end say, the runner runs it again with --trace 1 and prints the operation that was running. --seed N reruns a different fuzz run, and --sanitize builds with AddressSanitizer and UBSan, which block 4's unsafe { } blocks are exactly what they're for.
How it's built
Every kata file is a whole program. Yours, the solution and each level import what they use, then the harness, which has main:
import "std/memops.rin"
import "std/vec.rin"
import "std/option.rin"
import "../dojo.rin"
import "harness.rin"
So any of them builds on its own, rin compile dojo/3.slot_map/solution.rin for instance, and rin check or your editor finds every name it uses. Only the harness doesn't build alone: it calls the procs the kata defines. A file imported twice is read once, so the kata and the harness both importing dojo.rin is fine. dojo/dojo.rin holds what every harness shares: the regions level and frame, dojo_handle and dojo_status, the run's settings, the random number generator, and the reporting. The runner builds kata.rin the way tools/check.py does, rin compile and then the C compiler, with the warnings for unused parameters and locals off, since a kata in progress is full of them.
python tools/check.py dojo keeps the dojo honest: every solution must build on its own, pass its harness and print its harness.expected, level 1 must fail (its gap must be a gap), level 2 must compile, and the playgrounds must print their .expected. The harnesses were also checked against broken solutions, the mistakes each block is about, and catch each one.
The same dojo exists in Rust, in rust-dod. The random number generator is the same, so a seed is the same run in both, and the playgrounds print the same thing, except where block 4's column gets its memory: a region's arena in rin, a Vec in Rust. The katas differ where the languages do: rin's are made in a region; block 7's arena hands out offsets in both, for different reasons; and in block 8 only rin can keep a pointer across a move, so only rin's switch does. Each README says how.
The dojo · block 1
Free list and generations
dojo/1.free_list/play_free_list.rin
Handles name things that come and go: entities, textures, sounds, network connections. A handle is two numbers, index:generation, and two small ideas make it safe to keep:
- A free list. A freed index is reused before a new one is made, so the arrays an index points into stay as long as the most things ever alive at once, not the most things ever made.
- A generation per index. Freeing bumps the index's generation, so every handle made for the old life stops matching. A reused index can't be mistaken for whatever had it before.
The invariant: a handle is alive exactly when its generation is the one its index holds now. A free index already holds its next generation, which no handle has yet, so "free" needs no flag of its own. Every index is either alive or on the free list, once. Generations start at 1, so a zeroed handle, {}, is never alive.
In the course: 1.2 builds the free list and meets its bug, 1.3 adds generations, and every later chapter keeps both.
Watch it: the playground
python tools/dojo.py play free_list
It allocates four handles, frees two, reuses one, and draws both arrays after each step:
free b, then d: their generations go up now, before anyone reuses them
generations 0:1 1:2 2:1 3:2
free list [1 3]
alloc e: alloc takes from the end of the free list, so index 3, in its next life
e 3:2
Step through it: break on handles_alloc and handles_free, and watch generations and free change. Then flip the switch at the top, bump_on_free, to 0 and run it again. The freed b is still alive, the reused index comes back with the old generation so d and e are the same handle, and freeing d again frees e and puts index 3 back on the free list while e is still using it. That's the bug from 1.2, in four lines of output.
Do reps: the kata
You write handles, an allocator of handles, made in the region level:
handles_make: proc(r: region level) -> *handles in level = {
handles_alloc: proc(h: *handles) -> dojo_handle = {
handles_free: proc(h: *handles, e: dojo_handle) -> dojo_status = {
handles_alive: proc(h: *handles, e: dojo_handle) -> b32 = {
plus handles_live (how many are alive) and handles_slots (how many indices have been handed out). dojo_handle and dojo_status are given, in dojo/dojo.rin. The arrays are *vec<u32> in level, made by handles_make; they grow in the region by themselves, so handles_alloc takes no region.
| level | you write | target |
|---|---|---|
| 1 | handles_free, in an otherwise finished file | 2:00 |
| 2 | every proc body; the fields and signatures are given | 8:00 |
| 3 | the whole file, from a blank page | 10:00 |
python tools/dojo.py start free_list 2 kata.rin from level 2, and the clock starts
python tools/dojo.py free_list your kata.rin against the harness
The harness first runs a short scenario, the playground's steps, and prints each as it passes. Then it runs 20,000 random allocs and frees from a seed, against a model that remembers, per index, the last generation handed out and whether the index is free. It checks what your allocator does, not how: a generation is never handed out twice for an index, a freed index is reused before a new one is made, and handles_alive agrees with the model for every handle, live and dead. The seeds give the same runs as rust-dod's dojo.
Level 4, variations. The same harness, different insides:
- Keep the free list inside the dead slots: no
freevec, but anext_freefield in each slot that is only meaningful while the slot is free, and afirst_freeinhandles. What marks the end of the list? - 16-bit generations in the slots, stored as
u16, that wrap from 65535 back to 1, never 0. What does wrapping give up? - Reuse in first-in, first-out order, so an index rests as long as possible before it comes back. Why might a game want that?
When it fails
FAIL at operation 266 (seed 1): after free, alive() is still true for 43:1
debug it: build/dojo/free_list/harness --seed 1 --stop 266
with a breakpoint on dojo_stop_here, which runs just before that operation
python tools/dojo.py free_list built that harness with your kata. Run it under your debugger with those arguments and a breakpoint on dojo_stop_here (gdb --args build/dojo/free_list/harness --seed 1 --stop 266, then break dojo_stop_here, run). It stops just before the operation that fails; step out and into your code.
If the harness crashes instead, on an index past the end of a vec say, the runner runs it again with --trace 1 and prints the operation that was running.
The playground. The kata's harness is dojo/1.free_list/harness.rin, and the levels start from dojo/1.free_list/level1.rin to level3.rin.
1// Dojo 1, the playground: a free list and generations.2//3// Handles name things that come and go: entities, textures, sounds, network4// connections. Two small ideas make them safe to keep:5//6// * a free list: freed indices are reused before new ones are made, so the7// arrays an index points into stay as small as the most things alive;8// * a generation per index: freeing bumps it, so every handle made for the9// old life stops matching, and a reused index can't be mistaken for the10// thing that had it before.11//12// This program runs the allocator step by step and draws its two arrays after13// each step. To see it in a debugger, break on `handles_alloc` and14// `handles_free` and watch `generations` and `free` change.15//16// The switch: set bump_on_free to 0 and run it again. The reused index comes17// back with the old generation, and a dead handle is alive again. It's a18// global, read each time it's used, so a debugger can flip it mid-run too:19// stop on handles_free, set it to 0 in the watch window, and continue.20 21import "std/print.rin"22 23bump_on_free: b32 = 1;24 25handle: struct = {26 index: u32;27 generation: u32;28}29 30handles: struct = {31 // The life each index is in now, or will be in next if it is free.32 generations: [8]u32;33 count: u32;34 // Freed indices, a stack: the last one freed is the first one reused.35 free: [8]u32;36 free_count: u32;37}38 39handles_alloc: proc(h: *handles) -> handle = {40 index: u32 = 0;41 if (h*.free_count > 0) {42 h*.free_count -= 1;43 index = h*.free[h*.free_count];44 } else {45 index = h*.count;46 h*.generations[index] = 1;47 h*.count += 1;48 }49 return { .index = index, .generation = h*.generations[index] };50}51 52handles_alive: proc(h: *handles, e: handle) -> b32 = {53 return e.index < h*.count && h*.generations[e.index] == e.generation;54}55 56handles_free: proc(h: *handles, e: handle) -> b32 = {57 if (!handles_alive(h, e)) {58 return 0;59 }60 if (bump_on_free) {61 h*.generations[e.index] += 1;62 }63 h*.free[h*.free_count] = e.index;64 h*.free_count += 1;65 return 1;66}67 68// Each index with the generation it holds, then the free list.69draw: proc(h: *handles) -> void = {70 printfmt(" generations");71 for (i: u32 = 0; i < h*.count; i += 1) {72 printfmt(" {}:{}", i, h*.generations[i]);73 }74 printfmt("\n free list [");75 for (i: u32 = 0; i < h*.free_count; i += 1) {76 if (i > 0) {77 printfmt(" ");78 }79 printfmt("{}", h*.free[i]);80 }81 printfmt("]\n");82}83 84says: proc(alive: b32) -> ^const char = {85 return alive ? "alive" : "dead";86}87 88main: proc() -> i32 = {89 h: handles = {};90 printfmt("alloc a, b, c, d: four new indices, each in its first life\n");91 a: handle = handles_alloc(h&);92 b: handle = handles_alloc(h&);93 c: handle = handles_alloc(h&);94 d: handle = handles_alloc(h&);95 printfmt(" a {}:{} b {}:{} c {}:{} d {}:{}\n", a.index, a.generation, b.index, b.generation,96 c.index, c.generation, d.index, d.generation);97 draw(h&);98 99 printfmt("free b, then d: their generations go up now, before anyone reuses them\n");100 _ = handles_free(h&, b);101 _ = handles_free(h&, d);102 draw(h&);103 104 printfmt("alloc e: alloc takes from the end of the free list, so index 3, in its next life\n");105 e: handle = handles_alloc(h&);106 printfmt(" e {}:{}\n", e.index, e.generation);107 draw(h&);108 109 printfmt("the old handles\n");110 printfmt(" d {}:{} is {}\n", d.index, d.generation, says(handles_alive(h&, d)));111 printfmt(" e {}:{} is {}\n", e.index, e.generation, says(handles_alive(h&, e)));112 printfmt(" b {}:{} is {}\n", b.index, b.generation, says(handles_alive(h&, b)));113 if (d.index == e.index && d.generation == e.generation) {114 printfmt(" d and e are the same handle: whoever kept d now reaches e\n");115 } else {116 printfmt(" d and e share index {} but not a generation, so d can never reach e\n", e.index);117 }118 119 printfmt("free d again\n");120 if (handles_free(h&, d)) {121 printfmt(" freed: d named e, so e is gone\n");122 } else {123 printfmt(" refused: d is dead, nothing changes\n");124 }125 draw(h&);126 127 printfmt("alloc f, g: index 1 first, then a new one\n");128 f: handle = handles_alloc(h&);129 g: handle = handles_alloc(h&);130 printfmt(" f {}:{} g {}:{}\n", f.index, f.generation, g.index, g.generation);131 draw(h&);132 printfmt(" c {}:{} was never freed: still {}\n", c.index, c.generation, says(handles_alive(h&, c)));133 return 0;134}What play_free_list prints. tools/check.py fails the playground if a single byte differs.
alloc a, b, c, d: four new indices, each in its first life
a 0:1 b 1:1 c 2:1 d 3:1
generations 0:1 1:1 2:1 3:1
free list []
free b, then d: their generations go up now, before anyone reuses them
generations 0:1 1:2 2:1 3:2
free list [1 3]
alloc e: alloc takes from the end of the free list, so index 3, in its next life
e 3:2
generations 0:1 1:2 2:1 3:2
free list [1]
the old handles
d 3:1 is dead
e 3:2 is alive
b 1:1 is dead
d and e share index 3 but not a generation, so d can never reach e
free d again
refused: d is dead, nothing changes
generations 0:1 1:2 2:1 3:2
free list [1]
alloc f, g: index 1 first, then a new one
f 1:2 g 4:1
generations 0:1 1:2 2:1 3:2 4:1
free list []
c 2:1 was never freed: still alive
The dojo · block 2
Swap-remove and the back-reference
dojo/2.swap_remove/play_swap_remove.rin
Systems loop over values, so values are kept packed: one after another, no holes, nothing to skip. Removing one from the middle would leave a hole, so the last value moves into it instead. That's swap-remove: one copy, however many values there are, at the price of order.
Values move, so an id can't be the position of its value. Two arrays keep track:
- sparse, by id:
rows[id]says which row id's value is in now; - packed, by row:
ids[row]is the id that owns the row. This is the back-reference: when the last row moves into a hole, it says whose entry inrowsto fix.
The whole structure is what EnTT calls a sparse set.
The invariant: the rows are packed, and the two arrays point at each other: for every row, rows[ids[row]] names that row, and an id with no value has no row.
In the course: 1.4 packs entities into rows this way, 2.3 does it for every column of a table, and 2.6 does it when an entity moves between tables.
Watch it: the playground
python tools/dojo.py play swap_remove
It inserts four values, removes one from the middle, and draws both sides after each step:
remove 7, in row 1: the last row, 5's, moves into the hole
row 0 1 2
ids 3 5 1
values 30 50 10
rows 1->2 3->0 5->1 (id -> row)
Step through it: break on sparse_set_remove, and watch ids and rows as the last row moves. Then flip the switch at the top, fix_back_reference, to 0 and run it again. Id 5's row still says 3, past the end, so get 5 finds nothing. Then insert 9 puts 9's value in row 3, and get 5 returns 90, another id's value, with nothing to say it's wrong. That's the bug every packed array invites, and the reason the back-reference exists.
Do reps: the kata
You write sparse_set, made in the region level, with values, ids and rows each a *vec<...> in level. The heart of it is sparse_set_remove:
sparse_set_remove: proc(s: *sparse_set, id: u32) -> option<u64> = {
if (sparse_set_get(s, id) == null) {
return option<u64>none();
}
entry: *u32 = vec<u32>at(s*.rows, cast(id, u64));
row: u64 = cast(entry* - 1, u64);
entry* = 0;
value: u64 = vec<u64>at(s*.values, row)*;
last_value: u64 = vec<u64>pop(s*.values);
last_id: u32 = vec<u32>pop(s*.ids);
if (row < s*.values*.length) {
vec<u64>at(s*.values, row)* = last_value;
vec<u32>at(s*.ids, row)* = last_id;
vec<u32>at(s*.rows, cast(last_id, u64))* = cast(row, u32) + 1;
}
return option<u64>some(value);
}
That's the reference, so don't read it before level 1 if you'd rather not know. vec has no swap-remove of its own: the last element is popped, and written over the hole unless the hole was the last row. The full API:
sparse_set_insert: proc(s: *sparse_set, id: u32, value: u64) -> option<u64> = {
sparse_set_get: proc(s: *sparse_set, id: u32) -> option<*u64 in level> = {
plus sparse_set_make(r), sparse_set_len, sparse_set_values (the packed values, in row order, a []u64 in level) and sparse_set_ids (ids[row] owns values[row], a []u32 in level).
| level | you write | target |
|---|---|---|
| 1 | sparse_set_remove, in an otherwise finished file | 2:00 |
| 2 | every proc body; the fields and signatures are given | 8:00 |
| 3 | the whole file, from a blank page | 10:00 |
python tools/dojo.py start swap_remove 2
python tools/dojo.py swap_remove
The harness runs the playground's steps as a scenario, then 20,000 random inserts and removes over a small range of ids, so the same ids come and go many times, against a plain array from id to value. Every 64 operations it checks the packed rows themselves: as many as the model has, and each row's owner holding exactly that row's value. The seeds give the same runs as rust-dod's dojo.
rin notes. Where Rust's kata returns Option<T> and has get_mut, this one uses std's option<u64> for what insert and remove give back (the old value, or none), and one sparse_set_get that returns an option<*u64 in level>, a pointer to the value, null for none, to read or to change, the way kayan_get<T> does in the course. The caller tests it, if (current), before using it. The in level is there, on sparse_set_values and sparse_set_ids too, because the pointer comes out of a vec in level, and the checker refuses a result that doesn't say so. rows[id] holds the row plus one, so a new entry, 0, means "no row", and growing rows is appending zeros.
Level 4, variations. The same harness, different insides:
- Pages:
rowssplit into pages of 64 ids, each page made only when an id in it gets a value. A big id no longer means a bigrows. This is what EnTT does, with a pointer per page, null for a page not made yet. Heresparse_set_inserthas no region to make a page in, and a struct can't keep one, so a page is a struct holding[64]u32in avecof pages, and avec<u32>by page number holds one more than its place there, 0 for a page not made yet. - A sentinel:
rows[id]holds the row itself, with0xFFFFFFFFfor "no row". What does growingrowscost now? - A flag: a second
vec<b32>,has_row, besiderows. Compare the memory with the plus-one encoding.
And one that fails on purpose: a remove that keeps order, shifting every later row down and fixing each one's back-reference. The scenario stops it at the first ids line. Why does the harness insist on swap-remove's order, and what does keeping order cost?
When it fails
A forgotten back-reference often shows up as a crash in a sparse_set_get that is right, not in the sparse_set_remove that is wrong: get follows the stale row past the end, and vec<u64>at stops the program.
rin runtime: vec index out of bounds
the harness crashed in the scenario, after the last step printed above
The last step printed is the last that passed, so the bug is in what ran after it. In the fuzz run, the runner runs the harness again with --trace 1 and prints the operation that was running when it crashed. A failed check gives the operation and the seed:
FAIL at operation 64 (seed 1): row 4 holds 43's value 37043, but get(43) gives 43020
debug it: build/dojo/swap_remove/harness --seed 1 --stop 64
with a breakpoint on dojo_stop_here, which runs just before that operation
Run the harness under your debugger with those arguments and a breakpoint on dojo_stop_here (gdb --args build/dojo/swap_remove/harness --seed 1 --stop 64, then break dojo_stop_here, run). It stops just before the operation that fails; step out and into your code.
The playground. The kata's harness is dojo/2.swap_remove/harness.rin, and the levels start from dojo/2.swap_remove/level1.rin to level3.rin.
1// Dojo 2, the playground: swap-remove and the back-reference.2//3// Systems loop over values, so values are kept packed: one after another, no4// holes, nothing to skip. Removing one from the middle would leave a hole, so5// the last value moves into it instead. That's swap-remove: one copy, however6// many values there are. The price is that values move, so an id can't be the7// position of its value. Two arrays keep track:8//9// * sparse, by id: rows[id] is the row id's value is in now;10// * packed, by row: ids[row] is the id that owns the row, the11// back-reference. When the last row moves, it says whose entry in `rows`12// to fix.13//14// This is what EnTT calls a sparse set, and what the ECS course's tables do15// in 1.4 and 2.3.16//17// This program inserts four values, removes one from the middle, and draws18// both sides after each step. To see it in a debugger, break on19// `sparse_set_remove` and watch `ids` and `rows` as the last row moves.20//21// The switch: set fix_back_reference to 0 and run it again. The moved id's row22// still points where its value used to be, and the next insert writes another23// id's value there. It's a global, read each time it's used, so a debugger can24// flip it mid-run too: stop on sparse_set_remove, set it to 0 in the watch25// window, and continue.26 27import "std/print.rin"28 29fix_back_reference: b32 = 1;30 31sparse_set: struct = {32 values: [8]u64;33 ids: [8]u32;34 count: u32;35 // rows[id] is one more than id's row, so 0 means id has no value.36 rows: [16]u32;37}38 39sparse_set_insert: proc(s: *sparse_set, id: u32, value: u64) -> void = {40 s*.rows[id] = s*.count + 1;41 s*.values[s*.count] = value;42 s*.ids[s*.count] = id;43 s*.count += 1;44}45 46sparse_set_remove: proc(s: *sparse_set, id: u32) -> void = {47 if (s*.rows[id] == 0) {48 return;49 }50 row: u32 = s*.rows[id] - 1;51 s*.rows[id] = 0;52 last: u32 = s*.count - 1;53 s*.values[row] = s*.values[last];54 s*.ids[row] = s*.ids[last];55 s*.count = last;56 if (row < last && fix_back_reference) {57 s*.rows[s*.ids[row]] = row + 1;58 }59}60 61// id's value through its row, or 0 for nothing. A row past the end reads62// nothing; a row that is wrong but in range reads someone else's value.63sparse_set_get: proc(s: *sparse_set, id: u32) -> u64 = {64 if (s*.rows[id] == 0 || s*.rows[id] - 1 >= s*.count) {65 return 0;66 }67 return s*.values[s*.rows[id] - 1];68}69 70draw: proc(s: *sparse_set) -> void = {71 printfmt(" row ");72 for (r: u32 = 0; r < s*.count; r += 1) {73 printfmt("{:5}", r);74 }75 printfmt("\n ids ");76 for (r: u32 = 0; r < s*.count; r += 1) {77 printfmt("{:5}", s*.ids[r]);78 }79 printfmt("\n values ");80 for (r: u32 = 0; r < s*.count; r += 1) {81 printfmt("{:5}", cast(s*.values[r], u32));82 }83 printfmt("\n rows ");84 first: b32 = 1;85 for (id: u32 = 0; id < 16; id += 1) {86 if (s*.rows[id] != 0) {87 if (!first) {88 printfmt(" ");89 }90 printfmt("{}->{}", id, s*.rows[id] - 1);91 first = 0;92 }93 }94 printfmt(" (id -> row)\n");95}96 97read_value: proc(s: *sparse_set, id: u32) -> void = {98 value: u64 = sparse_set_get(s, id);99 if (value == 0) {100 printfmt("nothing");101 } else {102 printfmt("{}", value);103 }104}105 106main: proc() -> i32 = {107 s: sparse_set = {};108 printfmt("insert 3, 7, 1, 5, each with a value of ten times its id\n");109 ids: [4]u32 = { 3, 7, 1, 5 };110 for (i: u32 = 0; i < 4; i += 1) {111 sparse_set_insert(s&, ids[i], cast(ids[i], u64) * 10);112 }113 draw(s&);114 115 printfmt("remove 7, in row 1: the last row, 5's, moves into the hole\n");116 sparse_set_remove(s&, 7);117 draw(s&);118 printfmt(" get 5: ");119 read_value(s&, 5);120 printfmt("\n get 7: ");121 read_value(s&, 7);122 printfmt("\n");123 124 printfmt("insert 9: a new row at the end\n");125 sparse_set_insert(s&, 9, 90);126 draw(s&);127 printfmt(" get 5: ");128 read_value(s&, 5);129 printfmt("\n get 9: ");130 read_value(s&, 9);131 printfmt("\n");132 133 printfmt("remove 3, in row 0: the last row, 9's, moves into it\n");134 sparse_set_remove(s&, 3);135 draw(s&);136 printfmt(" get 9: ");137 read_value(s&, 9);138 printfmt(" get 5: ");139 read_value(s&, 5);140 printfmt(" get 1: ");141 read_value(s&, 1);142 printfmt("\n");143 return 0;144}What play_swap_remove prints. tools/check.py fails the playground if a single byte differs.
insert 3, 7, 1, 5, each with a value of ten times its id
row 0 1 2 3
ids 3 7 1 5
values 30 70 10 50
rows 1->2 3->0 5->3 7->1 (id -> row)
remove 7, in row 1: the last row, 5's, moves into the hole
row 0 1 2
ids 3 5 1
values 30 50 10
rows 1->2 3->0 5->1 (id -> row)
get 5: 50
get 7: nothing
insert 9: a new row at the end
row 0 1 2 3
ids 3 5 1 9
values 30 50 10 90
rows 1->2 3->0 5->1 9->3 (id -> row)
get 5: 50
get 9: 90
remove 3, in row 0: the last row, 9's, moves into it
row 0 1 2
ids 9 5 1
values 90 50 10
rows 1->2 5->1 9->0 (id -> row)
get 9: 90 get 5: 50 get 1: 10
The dojo · block 3
The slot map
dojo/3.slot_map/play_slot_map.rin
Dojos 1 and 2, put together. Values live packed in rows, so a loop over them has no holes (dojo 2). The rest of the program keeps handles, which survive rows moving and say when their value is gone (dojo 1). In between is a slot per index:
handle 1:2 -> slots[1] { generation 2, row 0 } -> values[0]
A lookup is two steps: the handle's generation must be the slot's, and the slot says the row. A remove is both dojos' work: bump the generation and free the index, then swap-remove the row and fix the slot of whoever moved. Game engines keep textures, meshes and sounds this way, and it is the ECS's entity index.
The invariant: dojo 1's and dojo 2's together. A handle is alive exactly when its generation is its slot's; the rows are packed; and for every row, the slot of the row's handle points back at that row.
In the course: 1.4 is a slot map with an entity's components as its value, and every chapter after keeps it, with a table number added to the slot in 2.4.
Watch it: the playground
python tools/dojo.py play slot_map
It keeps three textures, removes one, reuses its index, and draws the slots and the rows after each step:
remove stone: free index 1 in its next life, and water moves into row 1
slots 0: generation 1, row 0 | 1: generation 2, free | 2: generation 1, row 1
rows 0:1 grass | 2:1 water
Step through it: break on slot_map_remove, and watch the slot, the free list and the rows change in that order. Then flip the switch at the top, check_generation, to 0 and run it again. The removed stone still finds slot 1, and reads whatever is in slot 1's row now: water, then lava. In a game, that's a sound that plays the wrong sample, or a mesh drawn with another mesh's texture.
Do reps: the kata
You write slot_map, made in the region level, with its four arrays each a *vec<...> in level. Its two-step lookup:
slot_map_get: proc(m: *slot_map, h: dojo_handle) -> option<*u64 in level> = {
if (!slot_map_contains(m, h)) {
return null;
}
row: u32 = vec<slot_map_slot>at(m*.slots, cast(h.index, u64))*.row;
return vec<u64>at(m*.values, cast(row, u64));
}
and the API around it:
slot_map_insert: proc(m: *slot_map, value: u64) -> dojo_handle = {
slot_map_remove: proc(m: *slot_map, h: dojo_handle) -> option<u64> = {
plus slot_map_make(r), slot_map_contains, slot_map_len, slot_map_values (the packed values, in row order, a []u64 in level) and slot_map_handles (handles[row] owns values[row], a []dojo_handle in level). dojo_handle comes from dojo/dojo.rin, as in dojo 1.
| level | you write | target |
|---|---|---|
| 1 | slot_map_remove, in an otherwise finished file | 3:00 |
| 2 | every proc body; the fields and signatures are given | 12:00 |
| 3 | the whole file, from a blank page | 15:00 |
python tools/dojo.py start slot_map 2
python tools/dojo.py slot_map
Level 3 is the one to come back to. Once you can write a slot map from a blank page in fifteen minutes, dojos 1 and 2 are in your hands too.
The harness runs the playground's steps as a scenario, then 20,000 random inserts, removes, reads and writes against a model that keeps, per index, the last generation handed out, whether the index is free, and its value. It checks dojo 1's rules for the handles (a generation never handed out twice for an index, a freed index reused before a new one is made, a removed handle never alive again) and dojo 2's for the rows (packed, each row's handle holding that row's value). The seeds give the same runs as rust-dod's dojo.
rin notes. As in dojo 2, slot_map_remove gives back std's option<u64> and slot_map_get returns an option<*u64 in level>, a pointer that is null for a dead handle and tested before use, to read or to change the value; Rust's kata has get and get_mut. The in level is there, and on the two slices, because the checker refuses a result that comes out of a vec in level and doesn't say so. A zeroed handle, {}, is never alive, since generations start at 1, and the scenario checks that too.
Level 4, variations. The same harness, different insides:
- Free list inside the slots: a
next_free: u32in each slot, only meaningful while the slot is free, and afirst_freeinslot_map. Nofreevec. What marks the end of the list? - One look at the slot:
slot_map_removewithoutslot_map_contains: fetch the slot once, check its generation, and use it. How many times does the reference look the slot up? - **
slot_map_retain(m, keep)**, withkeepa*proc(h: dojo_handle, value: u64) -> b32: remove every value it rejects, in one pass, keeping the rows packed. Which way should the loop walk? (Not checked by the harness; write your own test.)
When it fails
FAIL at operation 233 (seed 1): remove(33:1) returned 214, the model says 210
debug it: build/dojo/slot_map/harness --seed 1 --stop 233
with a breakpoint on dojo_stop_here, which runs just before that operation
The wrong value usually means a slot pointing at the wrong row: an earlier remove moved a row and didn't fix its slot. Run the harness under your debugger with those arguments and a breakpoint on dojo_stop_here, which runs just before that operation, then look at the slot of the handle being removed.
If the harness crashes instead, on a row past the end of values say, the runner runs it again with --trace 1 and prints the operation that was running, or says the crash came in the scenario, after the last step printed.
The playground. The kata's harness is dojo/3.slot_map/harness.rin, and the levels start from dojo/3.slot_map/level1.rin to level3.rin.
1// Dojo 3, the playground: a slot map.2//3// Dojos 1 and 2, put together. Values live packed in rows, so a loop over them4// has no holes (dojo 2). The rest of the program keeps handles, which survive5// rows moving and say when their value is gone (dojo 1). In between is a slot6// per index:7//8// handle 1:2 -> slots[1] { generation 2, row 0 } -> values[0]9//10// A lookup is two steps: the handle's generation must be the slot's, and the11// slot says the row. A remove is both dojos' work: bump the generation and12// free the index, then swap-remove the row and fix the slot of whoever moved.13// This is the ECS course's entity index (1.4), with one value per entity, and14// what game engines use for textures, meshes and sounds.15//16// This program inserts three values, removes one, reuses its index, and17// draws the slots and the rows after each step. To see it in a debugger,18// break on `slot_map_remove` and watch `slots`, `handles` and `values`.19//20// The switch: set check_generation to 0 and run it again. A removed handle21// still finds a slot, and reads whatever value is in that slot's row now. It's22// a global, read each time it's used, so a debugger can flip it mid-run too:23// stop on slot_map_remove, set it to 0 in the watch window, and continue.24//25// A value is a texture, named by its place in `textures`, since26// a struct holding C strings is not a value safe code may write to.27 28import "std/print.rin"29import "std/option.rin"30 31check_generation: b32 = 1;32 33// The values: a texture is its place here.34textures: [4]^const char = { "grass", "stone", "water", "lava" };35 36handle: struct = {37 index: u32;38 generation: u32;39}40 41slot: struct = {42 generation: u32;43 row: u32;44}45 46slot_map: struct = {47 slots: [8]slot;48 slot_count: u32;49 free: [8]u32;50 free_count: u32;51 values: [8]u32;52 handles: [8]handle;53 count: u32;54}55 56slot_map_insert: proc(m: *slot_map, value: u32) -> handle = {57 index: u32 = 0;58 if (m*.free_count > 0) {59 m*.free_count -= 1;60 index = m*.free[m*.free_count];61 } else {62 index = m*.slot_count;63 m*.slots[index].generation = 1;64 m*.slot_count += 1;65 }66 m*.slots[index].row = m*.count;67 h: handle = { .index = index, .generation = m*.slots[index].generation };68 m*.values[m*.count] = value;69 m*.handles[m*.count] = h;70 m*.count += 1;71 return h;72}73 74slot_map_contains: proc(m: *slot_map, h: handle) -> b32 = {75 if (h.index >= m*.slot_count) {76 return 0;77 }78 return !check_generation || m*.slots[h.index].generation == h.generation;79}80 81// The value through the handle's slot, or none. A row past the end reads82// nothing.83slot_map_get: proc(m: *slot_map, h: handle) -> option<u32> = {84 if (!slot_map_contains(m, h) || m*.slots[h.index].row >= m*.count) {85 return option<u32>none();86 }87 return option<u32>some(m*.values[m*.slots[h.index].row]);88}89 90slot_map_remove: proc(m: *slot_map, h: handle) -> void = {91 if (!slot_map_contains(m, h)) {92 return;93 }94 row: u32 = m*.slots[h.index].row;95 m*.slots[h.index].generation += 1;96 m*.free[m*.free_count] = h.index;97 m*.free_count += 1;98 last: u32 = m*.count - 1;99 m*.values[row] = m*.values[last];100 m*.handles[row] = m*.handles[last];101 m*.count = last;102 if (row < last) {103 m*.slots[m*.handles[row].index].row = row;104 }105}106 107is_free: proc(m: *slot_map, index: u32) -> b32 = {108 for (i: u32 = 0; i < m*.free_count; i += 1) {109 if (m*.free[i] == index) {110 return 1;111 }112 }113 return 0;114}115 116draw: proc(m: *slot_map) -> void = {117 printfmt(" slots ");118 for (i: u32 = 0; i < m*.slot_count; i += 1) {119 if (i > 0) {120 printfmt(" | ");121 }122 if (is_free(m, i)) {123 printfmt("{}: generation {}, free", i, m*.slots[i].generation);124 } else {125 printfmt("{}: generation {}, row {}", i, m*.slots[i].generation, m*.slots[i].row);126 }127 }128 printfmt("\n rows ");129 for (r: u32 = 0; r < m*.count; r += 1) {130 if (r > 0) {131 printfmt(" | ");132 }133 printfmt("{}:{} {}", m*.handles[r].index, m*.handles[r].generation, textures[m*.values[r]]);134 }135 printfmt("\n");136}137 138read_value: proc(m: *slot_map, name: ^const char, h: handle) -> void = {139 value: option<u32> = slot_map_get(m, h);140 if (value.has_value) {141 printfmt("{} {}:{}: {}", name, h.index, h.generation, textures[value.value]);142 } else {143 printfmt("{} {}:{}: nothing: it is gone", name, h.index, h.generation);144 }145}146 147main: proc() -> i32 = {148 m: slot_map = {};149 printfmt("insert three textures\n");150 grass: handle = slot_map_insert(m&, 0);151 stone: handle = slot_map_insert(m&, 1);152 water: handle = slot_map_insert(m&, 2);153 printfmt(" grass {}:{} stone {}:{} water {}:{}\n", grass.index, grass.generation,154 stone.index, stone.generation, water.index, water.generation);155 draw(m&);156 157 printfmt("remove stone: free index 1 in its next life, and water moves into row 1\n");158 slot_map_remove(m&, stone);159 draw(m&);160 printfmt(" ");161 read_value(m&, "water", water);162 printfmt("\n ");163 read_value(m&, "stone", stone);164 printfmt("\n");165 166 printfmt("insert lava: index 1 again, generation 2, in a new row at the end\n");167 lava: handle = slot_map_insert(m&, 3);168 printfmt(" lava {}:{}\n", lava.index, lava.generation);169 draw(m&);170 printfmt(" ");171 read_value(m&, "stone", stone);172 printfmt("\n ");173 read_value(m&, "lava", lava);174 printfmt("\n");175 176 printfmt("remove grass: lava, in the last row, moves into row 0\n");177 slot_map_remove(m&, grass);178 draw(m&);179 printfmt(" ");180 read_value(m&, "lava", lava);181 printfmt(" ");182 read_value(m&, "water", water);183 printfmt(" ");184 read_value(m&, "grass", grass);185 printfmt("\n");186 return 0;187}What play_slot_map prints. tools/check.py fails the playground if a single byte differs.
insert three textures
grass 0:1 stone 1:1 water 2:1
slots 0: generation 1, row 0 | 1: generation 1, row 1 | 2: generation 1, row 2
rows 0:1 grass | 1:1 stone | 2:1 water
remove stone: free index 1 in its next life, and water moves into row 1
slots 0: generation 1, row 0 | 1: generation 2, free | 2: generation 1, row 1
rows 0:1 grass | 2:1 water
water 2:1: water
stone 1:1: nothing: it is gone
insert lava: index 1 again, generation 2, in a new row at the end
lava 1:2
slots 0: generation 1, row 0 | 1: generation 2, row 2 | 2: generation 1, row 1
rows 0:1 grass | 2:1 water | 1:2 lava
stone 1:1: nothing: it is gone
lava 1:2: lava
remove grass: lava, in the last row, moves into row 0
slots 0: generation 2, free | 1: generation 2, row 0 | 2: generation 1, row 1
rows 1:2 lava | 2:1 water
lava 1:2: lava water 2:1: water grass 0:1: nothing: it is gone
The dojo · block 4
A type-erased column
dojo/4.column/play_column.rin
A column is one component's rows, kept as bytes. It doesn't know the type it holds, only how many bytes make a row and how they must be aligned. That is enough to grow, to zero a new row, and to move the last row into a hole, which is everything a table does to its columns. The type comes back only at the edge: a typed view turns the bytes into a []T, for a T of the column's size and alignment.
Two things make it work:
- Aligned storage. The bytes come from the region aligned to the column's alignment, so every row of the type lands where that type may live.
- Growing by a factor. A full column doubles, so the copies that growing costs add up to about one per push, however many rows there are.
The invariant: the column holds count rows of size bytes each, packed from the start of a block aligned to align, with room for capacity rows. A typed view is only handed out for a plain-data T of exactly size bytes, aligned to exactly align, and only when the rows fit in the bytes. Plain data means a struct of numbers with no pointers in it, since a column copies rows as bytes.
In the course: 2.2 builds the column, 2.3 puts columns in a table, and 2.5 is about what happens to a pointer into a column that grows.
Watch it: the playground
python tools/dojo.py play column
It fills a column of vec2s and draws each row's bytes next to the value they are:
push 3 zeroed rows, and write each through the typed view
capacity 4, 3 rows in use, 32 bytes of storage
row 0 00 00 80 3f 00 00 00 40 as a vec2 (1, 2)
row 1 00 00 40 40 00 00 80 40 as a vec2 (3, 4)
Then it grows the column, swap-removes from it, and asks for the rows as two wrong types. Step through it: break on column_grow and watch the rows move to the new block. The old block stays in the region, unused, so a slice taken before the grow still points somewhere valid, and anything written through it is lost without a word.
The switch at the top, grow_by_doubling, picks how a full column grows. Set it to 0, so the column grows by one row, and run it again. The last line is the point: the same 1000 pushes copy 1,020 rows when the column doubles, and 499,494 when it grows by one.
1000 pushes into a u32 column, growing by doubling
9 grows, 1020 rows copied to make room
Do reps: the kata
You write column. Its bytes are a slice, data: []u8 in level, so even type erasure is mostly checked code: slice<u8>copy and slice<u8>zero do the work, and a row past the end stops the program like any index. Three things need unsafe { }, a few lines each, and their signatures are the promise safe code relies on:
column_makekeeps the region's arena, since a region can't be stored: a fieldarena: ^memops_arena, andc*.arena = r.column_growtakes a block from that arena, becausenew_slice<u8>can't ask for an alignment, and makes it a slice.column_slicetests the address and casts the bytes toT.
Keep each block that small: put your checks outside it, where the checker can see them.
A full column's new bytes come from the arena, aligned to the column's alignment:
raw: ^u8 = cast(memops_arena_push_zero(c*.arena[0]&, c*.size * capacity, c*.align), ^u8);
bytes: []u8 in level = cast(raw[0..c*.size * capacity], []u8 in level);
and the rows go across with slice<u8>copy(bytes[0..kept], c*.data[0..kept]). Slicing checks both ends, so the copy can't run off either block. The typed view is a generic proc that checks everything it relies on before it casts:
column_slice: proc<T>(c: *const column) -> []T in level = {
plus column_make(r, size, align), column_len, column_capacity, column_push_zeroed (a zeroed row at the end, and its row number), column_swap_remove(c, row), and column_row(c, row), a row's bytes as a []u8 in level. The kata has no vec to lean on: it is the thing a vec would be built from.
**What an unsafe block may believe.** rin has no private fields. Any checked code holding a *column can write c*.count = 1000, so an unsafe block must not trust a field or an argument unless the value carries its own bound, as a slice does. That decides what column_slice tests:
- The type's size and alignment, against the column's. The column can't know the type, but it can refuse every type that can't be the one it was made for.
- That the type is plain data,
reflect_is_plain(T<>&). Any bytes are then a validT, so a wrong type of the right shape gives wrong numbers, never a bad pointer: avec3column read as a struct of threeu32s gives 1065353216, 1073741824 and 1077936128, the bits of 1.0, 2.0 and 3.0. ATwith a pointer in it would turn bytes into an address, so it stops the program.T<>needs a struct or an enum, so a row is always one: the four-byte row here isword: struct = { value: u32; }, andcolumn_slice<u32>doesn't compile. - That the rows fit:
c*.countagainstc*.data.length / size.countis a field, and can say anything;lengthbelongs to the slice, and can't be wrong about it. - The address itself:
cast(c*.data.data, usize) % align, the bytes as they are, not whatc*.alignsays about them.
The checked procs get this for nothing: a column_row with a lying count or size stops at the slice, range 6000..6012 out of bounds (length 48). The one field still believed is arena: a raw pointer carries no bound. rin seals a struct holding one to the file that declares it, so the harness and anything else importing this file cannot write its fields. Inside the file that is on you: only column_make writes arena, and every unsafe author owes that much.
| level | you write | target |
|---|---|---|
| 1 | column_slice, the typed view, in an otherwise finished file | 3:00 |
| 2 | every proc body, and a column_grow of your own; the fields and the other signatures are given | 15:00 |
| 3 | the whole file, from a blank page | 20:00 |
python tools/dojo.py start column 2 kata.rin from level 2, and the clock starts
python tools/dojo.py column your kata.rin against the harness
python tools/dojo.py column --sanitize the same with AddressSanitizer and UBSan
The harness first runs a short scenario: three vec3s written through column_slice and read back as bytes, a swap-remove, a column aligned to 16, and a column of tags, zero-byte rows. Then it runs 20,000 random pushes, writes and swap-removes, shared between four row types: 3 bytes aligned to 1, a word holding a u32, a 12-byte vec3, and a 16-byte vec4 aligned to 16. Each is checked against a vec of that type, through column_slice and through column_row, along with the alignment of every view, and that a full column grows by at least half. The harness has two trusted helpers of its own, bytes_of and slice_address, to look at a value's bytes and a slice's address. The seeds give the same runs as rust-dod's dojo.
--sanitize is worth the wait here. A slice made from a raw pointer, raw[0..n], has whatever length you say, so a block asked for too small often works on your machine. AddressSanitizer and UBSan don't let it happen by luck. (They need the C compiler's sanitizer runtime; with clang that may be missing, so try CC=gcc.)
What the harness can't check is a refusal: a column read as the wrong type should stop the program with a message, and so should a row past the rows in use, and a harness that stops can't go on to the next step. It never lies to your column either, so a column_slice that tests only the type passes it. Try them by hand in a scratch program: a copy of your kata with level: region; where its import "harness.rin" was, and a main that opens the region, level: region = { ... }, makes c: *column in level = column_make(level, sizeof(vec3), alignof(vec3)); in it, and pushes two rows. Then, one at a time:
- read it as a
word(the wrong size), then as a struct of twelveu8s (the right size, aligned to 1); - ask for
column_row(c, 2); - write
c*.count = 1000;, then read it as avec3; - write
c*.data = c*.data[1..25];, then read it as avec3; - make a column of a struct with an
option<*u32 in level>in it, by its ownsizeofandalignof, and read it as that struct.
Each should stop with a message that says what was wrong. Then read the vec3s you wrote as a struct of three u32s, which should not stop, and print what comes back. rust-dod's harness checks the first refusals, because Rust can catch a panic.
Level 4, variations. The same harness, different insides:
- Grow by 1.5 instead of 2. Count the grows and the rows copied for 1000 pushes. Why do some allocators prefer it?
- Allocate with **
memops_arena_realloc_**, onc*.arena[0]&and insidecolumn_grow'sunsafe { }, instead of a new block and aslice<u8>copy. Hand it the old block byc*.data.dataandc*.data.length, the slice's own bound, not byc*.capacity. When does it move the rows, and when can it extend the block in place? - Shrink: when a swap-remove leaves the column less than a quarter full, move the rows to a block of half the capacity. Why a quarter and not a half? (Push and remove one row, over and over, at the boundary.) In a region, what does shrinking actually give back?
When it fails
FAIL at operation 11 (seed 1): the column was full at 4 rows and grew to 5; it should grow by at least half
debug it: build/dojo/column/harness --seed 1 --stop 11
with a breakpoint on dojo_stop_here, which runs just before that operation
python tools/dojo.py column built that harness with your kata. Run it under your debugger with those arguments and a breakpoint on dojo_stop_here (gdb --args build/dojo/column/harness --seed 1 --stop 11, then break dojo_stop_here, run). It stops just before the operation that fails; step out and into your code.
If the harness crashes instead, on a slice range past the end say, the runner runs it again with --trace 1 and prints the operation that was running.
The playground. The kata's harness is dojo/4.column/harness.rin, and the levels start from dojo/4.column/level1.rin to level3.rin.
1// Dojo 4, the playground: a type-erased column.2//3// A column is one component's rows, kept as bytes. It doesn't know the type4// it holds, only how many bytes make a row and how they must be aligned, and5// that is enough to grow, to zero a row, and to move the last row into a hole.6// The type comes back only at the edge: a typed view turns the bytes into a7// []T, for a T of the column's size and alignment.8//9// Two things make it work:10//11// * the storage starts on a boundary of the column's alignment, so every12// row of the type lands where it may live;13// * a full column grows by a factor, not by a row, so the copies that14// growing costs add up to about one per push.15//16// This program fills a column, draws each row's bytes next to the value they17// are, grows it, swap-removes from it, and asks it for the wrong type. Break18// on `column_grow` in a debugger and watch the rows move to the new block.19//20// The switch: set grow_by_doubling to 0, so a full column grows by one row,21// and run it again. Look at the last line: the same 1000 pushes, and how many22// rows were copied to make room for them. It's a global, read each time a23// column grows, so a debugger can flip it mid-run too: stop on column_grow, set24// it to 0 in the watch window, and continue.25//26// As in the kata, most of it is checked: the bytes are a `[]u8`. Only keeping the27// arena, taking an aligned block and viewing bytes as T need unsafe { }. The arena28// is the region `level`.29 30import "std/print.rin"31import "std/memops.rin"32import "std/fmt.rin"33import "std/slice.rin"34import "std/reflect.rin"35 36grow_by_doubling: b32 = 1;37 38level: region;39 40vec2: struct = {41 x: f32;42 y: f32;43}44 45// A column's rows are structs, as components are: reflection answers whether a46// struct is plain data, and has no record for a bare number.47word: struct = {48 value: u32;49}50 51// 8 bytes, like a vec2, but aligned to 1.52bytes8: struct = {53 b: [8]u8;54}55 56column: struct = {57 data: []u8 in level;58 size: u64;59 align: u64;60 count: u64;61 capacity: u64;62 // What growing has cost so far.63 grows: u64;64 copied: u64;65 arena: ^memops_arena;66}67 68column_make: proc(r: region level, size: u64, align: u64) -> *column in level = {69 unsafe {70 c: *column in level = new<column>(r);71 c*.arena = r;72 c*.size = size;73 c*.align = align;74 return c;75 }76}77 78// A new block from the region, and the rows in use copied into it. The old79// block stays in the region, unused: an arena frees everything at once, never80// one allocation.81column_grow: proc(c: *column) -> void = {82 capacity: u64 = 4;83 if (c*.capacity > 0) {84 capacity = grow_by_doubling ? c*.capacity * 2 : c*.capacity + 1;85 }86 unsafe {87 raw: ^u8 = cast(memops_arena_push_zero(c*.arena[0]&, c*.size * capacity, c*.align), ^u8);88 bytes: []u8 in level = cast(raw[0..c*.size * capacity], []u8 in level);89 kept: u64 = c*.size * c*.count;90 slice<u8>copy(bytes[0..kept], c*.data[0..kept]);91 c*.data = bytes;92 }93 c*.grows += 1;94 c*.copied += c*.count;95 c*.capacity = capacity;96}97 98column_push_zeroed: proc(c: *column) -> u64 = {99 if (c*.count == c*.capacity) {100 column_grow(c);101 }102 row: u64 = c*.count;103 c*.count += 1;104 slice<u8>zero(column_row(c, row));105 return row;106}107 108column_swap_remove: proc(c: *column, row: u64) -> void = {109 last: u64 = c*.count - 1;110 if (row != last) {111 moved: []u8 = column_row(c, last);112 slice<u8>copy(column_row(c, row), moved);113 }114 c*.count = last;115}116 117column_row: proc(c: *column, row: u64) -> []u8 in level = {118 return c*.data[row * c*.size..(row + 1) * c*.size];119}120 121// Whether T can be the column's type: its size and its alignment. The kata122// stops the program for a T that can't; this program wants to keep going.123column_fits: proc<T>(c: *const column) -> b32 = {124 size: u64 = sizeof(T);125 align: u64 = alignof(T);126 return size == c*.size && align == c*.align;127}128 129column_slice: proc<T>(c: *const column) -> []T in level = {130 size: u64 = sizeof(T);131 align: u64 = alignof(T);132 if (size != c*.size || align != c*.align) {133 printfmt("column: a column of {}-byte rows aligned to {} read as a type of {} bytes aligned to {}\n",134 c*.size, c*.align, size, align);135 exit(1);136 }137 if (!reflect_is_plain(T<>&)) {138 printfmt("column: a column holds plain data, and this {}-byte type is not\n", size);139 exit(1);140 }141 if (c*.count > c*.data.length / size) {142 printfmt("column: {} rows in use and room for {}\n", c*.count, c*.data.length / size);143 exit(1);144 }145 unsafe {146 if (cast(c*.data.data, usize) % align != 0) {147 printfmt("column: the bytes are not aligned to {}\n", align);148 exit(1);149 }150 raw: ^T = cast(c*.data.data, ^T);151 return cast(raw[0..c*.count], []T in level);152 }153}154 155draw: proc(c: *column, f: *fmt) -> void = {156 printfmt(" capacity {}, {} rows in use, {} bytes of storage\n", c*.capacity, c*.count, c*.capacity * c*.size);157 view: []vec2 = column_slice<vec2>(c);158 for (row: u64 = 0; row < c*.count; row += 1) {159 printfmt(" row {} ", row);160 bytes: []u8 = column_row(c, row);161 fmt_reset(f);162 for (i: u64 = 0; i < bytes.length; i += 1) {163 fmt_char(f, ' ');164 fmt_hex(f, cast(bytes[i], u64), 2, 0);165 }166 v: vec2 = view[row];167 printfmt("{} as a vec2 ({:.0}, {:.0})\n", fmt_text(f), cast(v.x, f64), cast(v.y, f64));168 }169}170 171push: proc(c: *column, x: f32, y: f32) -> void = {172 row: u64 = column_push_zeroed(c);173 view: []vec2 = column_slice<vec2>(c);174 view[row] = { .x = x, .y = y };175}176 177refused: proc(fits: b32) -> ^const char = {178 return fits ? "a slice" : "refused";179}180 181main: proc() -> i32 = {182 level: region = {183 f: *fmt in level = fmt_make(level, 64);184 c: *column in level = column_make(level, sizeof(vec2), alignof(vec2));185 printfmt("a column of vec2: {}-byte rows aligned to {}, kept in blocks from an arena\n", c*.size, c*.align);186 187 printfmt("push 3 zeroed rows, and write each through the typed view\n");188 for (i: u64 = 0; i < 3; i += 1) {189 push(c, cast(2 * i + 1, f32), cast(2 * i + 2, f32));190 }191 draw(c, f);192 193 printfmt("push 2 more: the column is full at 4 rows, so it grows before the fifth\n");194 push(c, 7.0, 8.0);195 before: u64 = c*.grows;196 push(c, 9.0, 10.0);197 if (c*.grows > before) {198 printfmt(" grew to {} rows: a new block from the arena, {} rows copied into it, the old one left there\n",199 c*.capacity, c*.count - 1);200 printfmt(" a slice taken before the grow still points into the old block: writes through it are lost\n");201 }202 draw(c, f);203 204 printfmt("swap_remove row 0: the last row's 8 bytes move into row 0\n");205 column_swap_remove(c, 0);206 draw(c, f);207 208 printfmt("ask for the rows as other types\n");209 printfmt(" as u32: {}, a u32 is 4 bytes and these rows are 8\n", refused(column_fits<u32>(c)));210 printfmt(" as bytes8: {}, the size is right, but it is aligned to 1, not 4\n", refused(column_fits<bytes8>(c)));211 212 big: *column in level = column_make(level, sizeof(word), alignof(word));213 for (i: u32 = 0; i < 1000; i += 1) {214 row: u64 = column_push_zeroed(big);215 values: []word = column_slice<word>(big);216 values[row].value = i;217 }218 if (grow_by_doubling) {219 printfmt("1000 pushes into a u32 column, growing by doubling\n");220 } else {221 printfmt("1000 pushes into a u32 column, growing by one row at a time\n");222 }223 printfmt(" {} grows, {} rows copied to make room\n", big*.grows, big*.copied);224 sum: u32 = 0;225 values: []word = column_slice<word>(big);226 for (i: u64 = 0; i < values.length; i += 1) {227 sum += values[i].value;228 }229 printfmt(" and the rows are all there: they add up to {}\n", sum);230 }231 return 0;232}What play_column prints. tools/check.py fails the playground if a single byte differs.
a column of vec2: 8-byte rows aligned to 4, kept in blocks from an arena
push 3 zeroed rows, and write each through the typed view
capacity 4, 3 rows in use, 32 bytes of storage
row 0 00 00 80 3f 00 00 00 40 as a vec2 (1, 2)
row 1 00 00 40 40 00 00 80 40 as a vec2 (3, 4)
row 2 00 00 a0 40 00 00 c0 40 as a vec2 (5, 6)
push 2 more: the column is full at 4 rows, so it grows before the fifth
grew to 8 rows: a new block from the arena, 4 rows copied into it, the old one left there
a slice taken before the grow still points into the old block: writes through it are lost
capacity 8, 5 rows in use, 64 bytes of storage
row 0 00 00 80 3f 00 00 00 40 as a vec2 (1, 2)
row 1 00 00 40 40 00 00 80 40 as a vec2 (3, 4)
row 2 00 00 a0 40 00 00 c0 40 as a vec2 (5, 6)
row 3 00 00 e0 40 00 00 00 41 as a vec2 (7, 8)
row 4 00 00 10 41 00 00 20 41 as a vec2 (9, 10)
swap_remove row 0: the last row's 8 bytes move into row 0
capacity 8, 4 rows in use, 64 bytes of storage
row 0 00 00 10 41 00 00 20 41 as a vec2 (9, 10)
row 1 00 00 40 40 00 00 80 40 as a vec2 (3, 4)
row 2 00 00 a0 40 00 00 c0 40 as a vec2 (5, 6)
row 3 00 00 e0 40 00 00 00 41 as a vec2 (7, 8)
ask for the rows as other types
as u32: refused, a u32 is 4 bytes and these rows are 8
as bytes8: refused, the size is right, but it is aligned to 1, not 4
1000 pushes into a u32 column, growing by doubling
9 grows, 1020 rows copied to make room
and the rows are all there: they add up to 499500
The dojo · block 5
A command buffer
dojo/5.commands/play_commands.rin
Systems walk tables, and while they walk, they decide to change which things exist: a bullet's time is up, a turret fires, a drone catches fire. Doing it on the spot breaks the walk. A despawn swap-removes, so the last row moves into the hole, behind the loop, and is never looked at. So a system records the change instead, as a command, and the frame applies every command, in order, once nothing is walking anything.
Three details make it work:
- Pending spawns. A spawn has no handle until it is applied, so recording one hands back a pending number, and later commands in the same buffer can name the spawn by it.
- Skipping the dead. Two systems may both despawn the same thing, or set a value on something an earlier command removed. Applied in order, the second finds a dead handle and is skipped. Generations make that one comparison.
- Room that stays. After the buffer is applied its vecs are cleared, and keep their room, so once a game has run a few frames, recording a command allocates nothing.
The invariant: recording changes nothing but the buffer. Applying it has exactly the effect of running every command straight away, in the order it was recorded, with each pending number meaning the handle its spawn made.
In the course: 3.3 builds the buffer and shows the bug it fixes, and 3.4's game runs on it. The Advanced lesson 6 has one too.
Watch it: the playground
python tools/dojo.py play commands
A packed world of five things, and one frame of three systems. The expire system records a despawn for each thing whose life is up, the spawner records a replacement and sets its life through its pending number, and a second system despawns something the first already did. Then the buffer is applied:
apply the buffer, in the order it was recorded
despawn 0:1: ok, 4:1 moves into 0:1's row
despawn 2:1: ok, 3:1 moves into 2:1's row
despawn 3:1: ok, it was the last row
spawn with life 1: pending 0 is 3:2
set pending 0 life 5: ok, 3:2 has life 5
despawn 2:1: skipped, 2:1 is dead
Step through it: break on commands_apply and walk the list. Then flip the switch at the top, defer, to 0, so every system changes the world on the spot, and run it again. The expire system despawns 0:1, 4:1 moves into its row behind the loop, and when the loop ends, 3:1 is still there with no life:
things left with no life: none
becomes things left with no life: 3:1, moved behind the loop and never looked at. That's 3.3's bug.
Do reps: the kata
You write commands. A command names its target either way, a handle or a spawn of this buffer:
target: struct = {
is_pending: b32;
handle: dojo_handle;
spawn: pending;
}
made by target_entity(h) or target_pending(p), so commands_set and commands_despawn take either:
commands_set: proc(c: *commands, t: target, value: u32) -> void = {
commands_apply: proc(c: *commands, w: *world) -> applied = {
plus commands_make(r), which makes the buffer and its two vecs in the region level, commands_spawn(c, value), which returns a pending, commands_despawn(c, t), and commands_len, the commands waiting. The world is given, in the harness: a generational store of numbers with world_spawn, world_despawn and world_set, where the last two return dojo_status.dead for a dead handle. commands_apply returns applied, the count of commands that changed something and of those that were skipped.
| level | you write | target |
|---|---|---|
| 1 | commands_apply, in an otherwise finished file | 3:00 |
| 2 | every proc body; the types and signatures are given | 15:00 |
| 3 | the whole file, from a blank page | 20:00 |
python tools/dojo.py start commands 2 kata.rin from level 2, and the clock starts
python tools/dojo.py commands your kata.rin against the harness
The harness first runs a short scenario: a set, a despawn, a spawn that reuses the despawned index, a set through the spawn's pending number, and two commands that find their target dead. Then a fuzz run, frame after frame: random commands recorded against live handles, dead handles and the frame's own pending spawns, then applied. A model runs the same commands straight away on a copy of the world from before the frame, and the harness checks the two worlds match, the applied and skipped counts match, and the buffer is empty afterwards. Each frame of the run opens the region frame for its lists and its model, gone when the frame ends. The two copies of the world hold pointers typed in level, so they are made in level and stay until the run ends. The seeds give the same runs as rust-dod's dojo.
Level 4, variations. The same harness, different insides:
- Keep the spawned handles in the command list itself, as 3.3 does: a spawn command holds the handle
commands_applygives it, and a pending number finds its spawn through it. Nospawnedvec. - Record the whole buffer as **one
vec<u8>**: a kind byte, then the command's fields as bytes. One allocation for any mix of commands, read back in order. Safe code can't see a struct as bytes, so au32goes in a byte at a time, with shifts andcast(..., u8), and comes back the same way. - Keep the buffer in **the region
frame**, opened once per game frame, as 3.3 and lesson 6 do:commands_make(r: region frame) -> *commands in frame, its vecsin frame, and nothing to clear, since the end of the region lets them all go. The harness makes one buffer inlevelfor the whole run, so this one needs a test of your own. What must not live in that region? Keep a pointer into it in a world that lives inlevel, and the checker tells you:this points into region 'frame', which may end before the memory in region 'level' that holds it.
When it fails
FAIL at operation 28 (seed 1): commands_apply returned applied 7, skipped 4, the model says applied 9, skipped 2
debug it: build/dojo/commands/harness --seed 1 --stop 28
with a breakpoint on dojo_stop_here, which runs just before that operation
python tools/dojo.py commands built that harness with your kata. Run it under your debugger with those arguments and a breakpoint on dojo_stop_here (gdb --args build/dojo/commands/harness --seed 1 --stop 28, then break dojo_stop_here, run). It stops just before the operation that fails; step out and into your code.
If the harness crashes instead, on a pending number past the spawns applied so far say, the runner runs it again with --trace 1 and prints the operation that was running.
The playground. The kata's harness is dojo/5.commands/harness.rin, and the levels start from dojo/5.commands/level1.rin to level3.rin.
1// Dojo 5, the playground: a command buffer.2//3// Systems walk tables, and while they walk they decide to change which things4// exist: a bullet's time is up, a turret fires. Doing it on the spot breaks the5// walk. A despawn swap-removes, so the last row moves into the hole, behind the6// loop, and is never looked at. So a system records the change instead, a7// command, and the frame applies every command, in order, once no one is8// walking anything:9//10// * a spawn has no handle until it is applied, so the buffer hands back a11// pending number, and later commands can name the spawn by it;12// * a command whose target died earlier in the same buffer is skipped, which13// generations make one comparison;14// * the buffer is cleared after it is applied, and keeps its memory for the15// next frame: a frame arena.16//17// This program builds a small packed world, runs one frame of three systems,18// and prints every command, then the buffer being applied. Break on19// `commands_apply` in a debugger and step through the list.20//21// The switch: set defer to 0, so the systems change the world on the spot, and22// run it again. Look at which things are left with no life. It's a global, so23// a debugger can flip it too: the frame reads it once, at its start, so it's24// all one way or the other; stop on that line, set it to 0 in the watch window,25// and continue.26//27// The world and the buffer live in the region `level`, opened in28// main.29 30import "std/print.rin"31import "std/memops.rin"32import "std/vec.rin"33 34level: region;35 36defer: b32 = 1;37 38handle: struct = {39 index: u32;40 generation: u32;41}42 43slot: struct = {44 generation: u32;45 row: u64;46}47 48// A packed world, like the course's tables: things in rows 0 to count - 1,49// each with a life in frames, and a slot per index that says which row it is50// in.51world: struct = {52 slots: *vec<slot> in level;53 free: *vec<u32> in level;54 owners: *vec<handle> in level;55 lives: *vec<u32> in level;56}57 58world_make: proc(r: region level) -> *world in level = {59 return new<world>(r, {60 .slots = vec<slot>make(r, 0), .free = vec<u32>make(r, 0),61 .owners = vec<handle>make(r, 0), .lives = vec<u32>make(r, 0) });62}63 64world_alive: proc(w: *world, h: handle) -> b32 = {65 return cast(h.index, u64) < w*.slots*.length66 && vec<slot>at(w*.slots, cast(h.index, u64))*.generation == h.generation;67}68 69world_spawn: proc(w: *world, life: u32) -> handle = {70 index: u32 = 0;71 if (!vec<u32>is_empty(w*.free)) {72 index = vec<u32>pop(w*.free);73 } else {74 index = cast(w*.slots*.length, u32);75 _ = vec<slot>push(w*.slots, { .generation = 1 });76 }77 s: *slot = vec<slot>at(w*.slots, cast(index, u64));78 s*.row = w*.owners*.length;79 h: handle = { .index = index, .generation = s*.generation };80 _ = vec<handle>push(w*.owners, h);81 _ = vec<u32>push(w*.lives, life);82 return h;83}84 85// What a despawn did: 0 refused, the target was dead; 1 the last row moved86// into the hole, into `moved`; 2 the target was the last row.87despawn_result: struct = {88 what: u32;89 moved: handle;90}91 92world_despawn: proc(w: *world, h: handle) -> despawn_result = {93 result: despawn_result = {};94 if (!world_alive(w, h)) {95 return result;96 }97 s: *slot = vec<slot>at(w*.slots, cast(h.index, u64));98 row: u64 = s*.row;99 s*.generation += 1;100 _ = vec<u32>push(w*.free, h.index);101 last_owner: handle = vec<handle>pop(w*.owners);102 last_life: u32 = vec<u32>pop(w*.lives);103 if (row == w*.owners*.length) {104 result.what = 2;105 return result;106 }107 vec<handle>at(w*.owners, row)* = last_owner;108 vec<u32>at(w*.lives, row)* = last_life;109 vec<slot>at(w*.slots, cast(last_owner.index, u64))*.row = row;110 result.what = 1;111 result.moved = last_owner;112 return result;113}114 115world_set_life: proc(w: *world, h: handle, life: u32) -> b32 = {116 if (!world_alive(w, h)) {117 return 0;118 }119 row: u64 = vec<slot>at(w*.slots, cast(h.index, u64))*.row;120 vec<u32>at(w*.lives, row)* = life;121 return 1;122}123 124draw: proc(w: *world) -> void = {125 for (row: u64 = 0; row < w*.owners*.length; row += 1) {126 h: handle = vec<handle>at(w*.owners, row)*;127 printfmt(" row {} {}:{} life {}\n", row, h.index, h.generation, vec<u32>at(w*.lives, row)*);128 }129}130 131print_despawned: proc(r: despawn_result, h: handle) -> void = {132 if (r.what == 1) {133 printfmt("ok, {}:{} moves into {}:{}'s row\n", r.moved.index, r.moved.generation, h.index, h.generation);134 } else if (r.what == 2) {135 printfmt("ok, it was the last row\n");136 } else {137 printfmt("skipped, {}:{} is dead\n", h.index, h.generation);138 }139}140 141// ---------------------------------------------------------------------------142// The buffer143// ---------------------------------------------------------------------------144 145command_kind: enum = {146 spawn,147 set_life,148 despawn,149}150 151// A command, and who it is for: a handle, or the `spawn`-th spawn of this152// buffer if `to_pending` is set.153command: struct = {154 kind: command_kind;155 to_pending: b32;156 entity: handle;157 spawn: u64;158 life: u32;159}160 161commands: struct = {162 list: *vec<command> in level;163 spawns: u64;164 spawned: *vec<handle> in level;165}166 167commands_make: proc(r: region level) -> *commands in level = {168 return new<commands>(r, { .list = vec<command>make(r, 0), .spawned = vec<handle>make(r, 0) });169}170 171print_target: proc(c: command) -> void = {172 if (c.to_pending) {173 printfmt("pending {}", c.spawn);174 } else {175 printfmt("{}:{}", c.entity.index, c.entity.generation);176 }177}178 179print_command: proc(c: command) -> void = {180 if (c.kind == command_kind.spawn) {181 printfmt("spawn with life {}", c.life);182 } else if (c.kind == command_kind.set_life) {183 printfmt("set ");184 print_target(c);185 printfmt(" life {}", c.life);186 } else {187 printfmt("despawn ");188 print_target(c);189 }190}191 192commands_record: proc(b: *commands, c: command) -> void = {193 printfmt(" record: ");194 print_command(c);195 printfmt("\n");196 _ = vec<command>push(b*.list, c);197}198 199commands_spawn: proc(b: *commands, life: u32) -> u64 = {200 commands_record(b, { .kind = command_kind.spawn, .life = life });201 b*.spawns += 1;202 return b*.spawns - 1;203}204 205commands_resolve: proc(b: *commands, c: command) -> handle = {206 if (c.to_pending) {207 return vec<handle>at(b*.spawned, c.spawn)*;208 }209 return c.entity;210}211 212// Every command in order, then an empty buffer that keeps its memory.213commands_apply: proc(b: *commands, w: *world) -> void = {214 for (i: u64 = 0; i < b*.list*.length; i += 1) {215 c: command = vec<command>at(b*.list, i)*;216 printfmt(" ");217 print_command(c);218 printfmt(": ");219 if (c.kind == command_kind.spawn) {220 h: handle = world_spawn(w, c.life);221 _ = vec<handle>push(b*.spawned, h);222 printfmt("pending {} is {}:{}\n", b*.spawned*.length - 1, h.index, h.generation);223 } else if (c.kind == command_kind.set_life) {224 h: handle = commands_resolve(b, c);225 if (world_set_life(w, h, c.life)) {226 printfmt("ok, {}:{} has life {}\n", h.index, h.generation, c.life);227 } else {228 printfmt("skipped, {}:{} is dead\n", h.index, h.generation);229 }230 } else {231 h: handle = commands_resolve(b, c);232 print_despawned(world_despawn(w, h), h);233 }234 }235 vec<command>clear(b*.list);236 vec<handle>clear(b*.spawned);237 b*.spawns = 0;238}239 240main: proc() -> i32 = {241 level: region = {242 w: *world in level = world_make(level);243 lives: [5]u32 = { 0, 3, 0, 0, 2 };244 for (i: u64 = 0; i < 5; i += 1) {245 _ = world_spawn(w, lives[i]);246 }247 cmds: *commands in level = commands_make(level);248 printfmt("a world of 5 things, packed in rows; a life of 0 means its time is up\n");249 draw(w);250 251 deferring: b32 = defer;252 if (deferring) {253 printfmt("the expire system walks the rows, and records a despawn for each life of 0\n");254 } else {255 printfmt("the expire system walks the rows, and despawns each life of 0 on the spot\n");256 }257 row: u64 = 0;258 while (row < w*.owners*.length) {259 h: handle = vec<handle>at(w*.owners, row)*;260 if (vec<u32>at(w*.lives, row)* == 0) {261 if (deferring) {262 commands_record(cmds, { .kind = command_kind.despawn, .entity = h });263 } else {264 printfmt(" row {}: despawn {}:{}: ", row, h.index, h.generation);265 print_despawned(world_despawn(w, h), h);266 }267 }268 row += 1;269 }270 271 printfmt("the spawner makes a replacement, and gives it a life through its pending number\n");272 if (deferring) {273 p: u64 = commands_spawn(cmds, 1);274 commands_record(cmds, { .kind = command_kind.set_life, .to_pending = 1, .spawn = p, .life = 5 });275 } else {276 h: handle = world_spawn(w, 1);277 printfmt(" spawn: {}:{}\n", h.index, h.generation);278 if (world_set_life(w, h, 5)) {279 printfmt(" set {}:{} life 5: ok\n", h.index, h.generation);280 } else {281 printfmt(" set {}:{} life 5: dead\n", h.index, h.generation);282 }283 }284 285 doomed: handle = { .index = 2, .generation = 1 };286 printfmt("a second system despawns {}:{} too: it doesn't know the first one did\n", doomed.index, doomed.generation);287 if (deferring) {288 commands_record(cmds, { .kind = command_kind.despawn, .entity = doomed });289 printfmt("the buffer holds {} commands, and the world has not changed:\n", cmds*.list*.length);290 draw(w);291 printfmt("apply the buffer, in the order it was recorded\n");292 commands_apply(cmds, w);293 } else {294 printfmt(" despawn {}:{}: ", doomed.index, doomed.generation);295 print_despawned(world_despawn(w, doomed), doomed);296 }297 298 printfmt("the world after the frame\n");299 draw(w);300 printfmt("things left with no life:");301 left: u64 = 0;302 for (r: u64 = 0; r < w*.owners*.length; r += 1) {303 if (vec<u32>at(w*.lives, r)* == 0) {304 h: handle = vec<handle>at(w*.owners, r)*;305 printfmt(" {}:{}", h.index, h.generation);306 left += 1;307 }308 }309 if (left == 0) {310 printfmt(" none\n");311 } else {312 printfmt(", moved behind the loop and never looked at\n");313 }314 printfmt("commands left in the buffer: {}\n", cmds*.list*.length);315 }316 return 0;317}What play_commands prints. tools/check.py fails the playground if a single byte differs.
a world of 5 things, packed in rows; a life of 0 means its time is up
row 0 0:1 life 0
row 1 1:1 life 3
row 2 2:1 life 0
row 3 3:1 life 0
row 4 4:1 life 2
the expire system walks the rows, and records a despawn for each life of 0
record: despawn 0:1
record: despawn 2:1
record: despawn 3:1
the spawner makes a replacement, and gives it a life through its pending number
record: spawn with life 1
record: set pending 0 life 5
a second system despawns 2:1 too: it doesn't know the first one did
record: despawn 2:1
the buffer holds 6 commands, and the world has not changed:
row 0 0:1 life 0
row 1 1:1 life 3
row 2 2:1 life 0
row 3 3:1 life 0
row 4 4:1 life 2
apply the buffer, in the order it was recorded
despawn 0:1: ok, 4:1 moves into 0:1's row
despawn 2:1: ok, 3:1 moves into 2:1's row
despawn 3:1: ok, it was the last row
spawn with life 1: pending 0 is 3:2
set pending 0 life 5: ok, 3:2 has life 5
despawn 2:1: skipped, 2:1 is dead
the world after the frame
row 0 4:1 life 2
row 1 1:1 life 3
row 2 3:2 life 5
things left with no life: none
commands left in the buffer: 0
The dojo · block 6
Bitsets
dojo/6.bitset/play_bitset.rin
A set of small numbers, 0 to 63, fits in one u64: bit id is set when id is in the set. An ECS keeps what each kind of entity is made of this way, a signature with a bit per component, and a query as two sets: the ids a table must have, and the ids it must not. The questions an ECS asks all day are then one or two instructions each, however many components there are:
with(set, id) set | (1 shl id)
without(set, id) set & ~(1 shl id)
has(set, id) (set & (1 shl id)) != 0
contains_all(set, all) (set & all) == all
intersects(set, none) (set & none) != 0
The same trick is a file's permissions, which of 64 workers are idle, and which chunks of a map are loaded.
The invariant: bit id is set exactly when id is in the set. Everything else follows from it: one set contains all of another when masking it with the other leaves the other whole, and two sets share an id when their AND isn't zero.
In the course: 2.1 gives each component a bit and each kind of entity a signature, 2.4 finds a table by its signature, 2.6 moves an entity by setting or clearing one bit, and 3.1 matches queries against tables with contains_all and intersects.
Watch it: the playground
python tools/dojo.py play bitset
It gives five kinds of thing a signature each, draws them as rows of bits, highest id first, and runs queries over them:
query move: all {position velocity}, none {frozen}
all 00000011
none 01000000
player 00011111 matches
drone 00001111 matches
turret 00101101 no
rock 00001001 no
ghost 01001011 no
Read it by columns: a kind matches when it has a 1 under every 1 of all, and a 0 under every 1 of none. Then the drone is frozen and the ghost thaws, one bit each, and the same query gives a different answer. Each kind keeps its ids twice, as a signature and as a b32 per id, and verify checks the fast answer against the slow one after every step. A kind is named by its place in kind_names, as a component is by its place in names, because a struct holding a C string is not a value safe code may write to.
Step through it: break on matches, and watch set & wanted in the watch window, in binary. Then flip the switch at the top, all_not_any, to 0 and run it again. contains_all now asks whether the signature has any of the ids. The turret and the rock have a position, so both match the movement query, and the turret and the ghost each have half of what the think query wants:
verify: turret matches, but it has no velocity
It's the most common bitset bug, and it compiles: the compiler has no idea which question you meant.
Do reps: the kata
You write procs over a u64. Bitsets are plain values, so there's no region and no _make in this kata, and nothing for the checker to object to. The one with the idea in it:
bitset_contains_all: proc(set: u64, wanted: u64) -> b32 = {
return (set & wanted) == wanted;
}
and a query, built from it:
bitset_matches: proc(set: u64, all: u64, none: u64) -> b32 = {
return bitset_contains_all(set, all) && !bitset_intersects(set, none);
}
plus bitset_bit (the set of id alone), bitset_with, bitset_without, bitset_has, bitset_intersects, bitset_count and bitset_ids, which writes the ids, smallest first, into a slice with room for 64 and returns how many. Ids are u32s, 0 to 63.
| level | you write | target |
|---|---|---|
| 1 | bitset_contains_all, in an otherwise finished file | 1:00 |
| 2 | every proc body; the signatures are given | 5:00 |
| 3 | the whole file, from a blank page | 6:00 |
python tools/dojo.py start bitset 2
python tools/dojo.py bitset
The harness runs the playground's kinds and queries as a scenario, then 20,000 random operations on a set: adding and taking away ids, and asking has, contains_all, intersects, matches, count and ids, against a model that keeps a b32 per id and answers one id at a time. The sets it asks about are mostly made from the set itself: part of it, part of it plus one more id, or ids it doesn't have, so the answers aren't nearly always false. Ids go up to 63, so a shift done in 32 bits shows. The seeds give the same runs as rust-dod's dojo.
rin notes. rin binds & tighter than ==, as Rust does, so set & wanted == wanted means what it says; in C it would be set & (wanted == wanted). The reference keeps the parentheses anyway, for anyone reading it with C eyes. And a bare 1 is an i32, which rin won't turn into a u64 for you: return 1 shl id; is refused, implicit conversion from 'i32' to 'u64' changes sign, so the reference writes 1'u64 shl cast(id, u64). The 32-bit bug needs a u32 on the left, one: u32 = 1; cast(one shl id, u64). rin masks a shift's amount to the width of what it shifts, so nothing there is undefined and --sanitize has nothing to report: id 63 lands on bit 31, and the harness is what catches it.
Level 4, variations. The same harness, different insides:
- Count without a loop over 64:
set & (set - 1)clears the lowest set bit. Loop until the set is empty, one step per id rather than per bit. - **
bitset_idsthe same way**:low = set & (0 - set)is the lowest set bit alone, its id is the number of bits below it,bitset_count(low - 1), and clearing it moves on. - More than 64 components: a struct of two
u64s, with idiat biti % 64of wordi / 64, and the same procs. What doescontains_allbecome? (Not checked by the harness; write your own test.)
When it fails
FAIL at operation 2 (seed 1): without({}, 57) gave {57}, it should give {}
debug it: build/dojo/bitset/harness --seed 1 --stop 2
with a breakpoint on dojo_stop_here, which runs just before that operation
That one is set ^ bitset_bit(id): an XOR flips the bit, so taking away an id the set doesn't have puts it in. Sets print as their ids, smallest first. Run the harness under your debugger with those arguments and a breakpoint on dojo_stop_here, which runs just before that operation.
The playground. The kata's harness is dojo/6.bitset/harness.rin, and the levels start from dojo/6.bitset/level1.rin to level3.rin.
1// Dojo 6, the playground: bitsets.2//3// A set of component ids fits in a u64: bit `id` is set when `id` is in the4// set. In an ECS that set is a signature, what a kind of entity is made of,5// and a query is two sets: the ids an entity must have, and the ids it must6// not. A query against a signature is then two ANDs and two compares,7// however many components there are:8//9// (signature & all) == all && (signature & none) == 010//11// This program gives five kinds of thing a signature each, draws them as rows12// of bits, runs queries over them, and changes two of them. Each kind keeps13// its ids twice: the fast way, a signature, and the slow way, a b32 per id.14// After every step, verify checks the fast way against the slow way and says15// what disagrees; break inside its `if`s to stop on the first.16//17// The switch: set all_not_any to 0 and run it again. contains_all then asks18// whether the signature has any of the ids, not all of them, and the turret,19// which can't move, matches the movement query. It's a global, read each time20// it's used, so a debugger can flip it mid-run too: stop on the first `query`21// in main, set it to 0 in the watch window, and continue.22//23// A kind is named by its place in `kind_names`, as a component24// is by its place in `names`, because a struct holding a C string is not a25// value safe code may write to.26 27import "std/print.rin"28 29all_not_any: b32 = 1;30 31names: [8]^const char = { "position", "velocity", "health", "sprite", "player", "turret", "frozen", "loot" };32kind_names: [8]^const char = { "player", "drone", "turret", "rock", "ghost", "", "", "" };33position: u32 = 0;34velocity: u32 = 1;35health: u32 = 2;36sprite: u32 = 3;37player: u32 = 4;38turret: u32 = 5;39frozen: u32 = 6;40 41bit: proc(id: u32) -> u64 = {42 return 1'u64 shl cast(id, u64);43}44 45with: proc(set: u64, id: u32) -> u64 = {46 return set | bit(id);47}48 49without: proc(set: u64, id: u32) -> u64 = {50 return set & ~bit(id);51}52 53has: proc(set: u64, id: u32) -> b32 = {54 return (set & bit(id)) != 0;55}56 57contains_all: proc(set: u64, wanted: u64) -> b32 = {58 if (all_not_any) {59 return (set & wanted) == wanted;60 }61 return (set & wanted) != 0;62}63 64matches: proc(set: u64, all: u64, none: u64) -> b32 = {65 return contains_all(set, all) && (set & none) == 0;66}67 68// Eight bits, highest id first, the way a number is written.69print_bits: proc(set: u64) -> void = {70 for (id: u32 = 8; id > 0; id -= 1) {71 if (has(set, id - 1)) {72 printfmt("1");73 } else {74 printfmt("0");75 }76 }77}78 79print_names: proc(set: u64) -> void = {80 printfmt("{");81 first: b32 = 1;82 for (id: u32 = 0; id < 8; id += 1) {83 if (has(set, id)) {84 if (!first) {85 printfmt(" ");86 }87 printfmt("{}", names[id]);88 first = 0;89 }90 }91 printfmt("}");92}93 94// A kind of thing: its signature and, the slow way, a b32 per id.95kind: struct = {96 signature: u64;97 parts: [8]b32;98}99 100// Up to eight kinds.101kinds: struct = {102 all: [8]kind;103 count: u32;104}105 106// What a system asks for: all of these ids, and none of those.107query: struct = {108 all: u64;109 none: u64;110}111 112add_kind: proc(k: *kinds, ids: []u32) -> u32 = {113 at: u32 = k*.count;114 for (i: u64 = 0; i < ids.length; i += 1) {115 k*.all[at].signature = with(k*.all[at].signature, ids[i]);116 k*.all[at].parts[ids[i]] = 1;117 }118 k*.count += 1;119 return at;120}121 122draw_kind: proc(it: *kind, name: ^const char) -> void = {123 printfmt(" {:-8}", name);124 print_bits(it*.signature);125 printfmt(" ");126 print_names(it*.signature);127 printfmt("\n");128}129 130set_part: proc(k: *kinds, at: u32, id: u32, on: b32) -> void = {131 if (on) {132 k*.all[at].signature = with(k*.all[at].signature, id);133 } else {134 k*.all[at].signature = without(k*.all[at].signature, id);135 }136 k*.all[at].parts[id] = on;137 draw_kind(k*.all[at]&, kind_names[at]);138}139 140run_query: proc(k: *kinds, q: query) -> void = {141 printfmt(" all ");142 print_bits(q.all);143 printfmt("\n none ");144 print_bits(q.none);145 printfmt("\n");146 for (i: u32 = 0; i < k*.count; i += 1) {147 printfmt(" {:-8}", kind_names[i]);148 print_bits(k*.all[i].signature);149 if (matches(k*.all[i].signature, q.all, q.none)) {150 printfmt(" matches\n");151 } else {152 printfmt(" no\n");153 }154 }155}156 157// The fast way against the slow way, for every kind: the signature's bits158// against its b32s, and the query's answer, if there is one, against one id159// at a time.160verify: proc(k: *kinds, q: option<*query>) -> void = {161 for (i: u32 = 0; i < k*.count; i += 1) {162 it: *kind = k*.all[i]&;163 for (id: u32 = 0; id < 8; id += 1) {164 if (has(it*.signature, id) != it*.parts[id]) {165 printfmt(" verify: {}'s signature and its parts disagree about {}\n", kind_names[i], names[id]);166 }167 }168 if (!q) {169 continue;170 }171 missing: i32 = -1;172 barred: i32 = -1;173 for (id: u32 = 0; id < 8; id += 1) {174 if (missing < 0 && has(q*.all, id) && !it*.parts[id]) {175 missing = cast(id, i32);176 }177 if (barred < 0 && has(q*.none, id) && it*.parts[id]) {178 barred = cast(id, i32);179 }180 }181 slow: b32 = missing < 0 && barred < 0;182 if (matches(it*.signature, q*.all, q*.none) != slow) {183 if (missing >= 0) {184 printfmt(" verify: {} matches, but it has no {}\n", kind_names[i], names[missing]);185 } else if (barred >= 0) {186 printfmt(" verify: {} matches, but it has {}\n", kind_names[i], names[barred]);187 } else {188 printfmt(" verify: {} should match, and doesn't\n", kind_names[i]);189 }190 }191 }192}193 194main: proc() -> i32 = {195 k: kinds = {};196 move_query: query = { .all = bit(position) | bit(velocity), .none = bit(frozen) };197 draw_query: query = { .all = bit(position) | bit(sprite), .none = 0 };198 think_query: query = { .all = bit(velocity) | bit(health), .none = bit(player) };199 player_ids: [5]u32 = { position, velocity, health, sprite, player };200 drone_ids: [4]u32 = { position, velocity, health, sprite };201 turret_ids: [4]u32 = { position, health, sprite, turret };202 rock_ids: [2]u32 = { position, sprite };203 ghost_ids: [4]u32 = { position, velocity, sprite, frozen };204 205 printfmt("five kinds of thing, one bit per component, highest id first\n");206 add_kind(k&, player_ids[0..5]);207 drone: u32 = add_kind(k&, drone_ids[0..4]);208 add_kind(k&, turret_ids[0..4]);209 add_kind(k&, rock_ids[0..2]);210 ghost: u32 = add_kind(k&, ghost_ids[0..4]);211 for (i: u32 = 0; i < k.count; i += 1) {212 draw_kind(k.all[i]&, kind_names[i]);213 }214 verify(k&, null);215 216 printfmt("query move: all {position velocity}, none {frozen}\n");217 run_query(k&, move_query);218 verify(k&, move_query&);219 220 printfmt("query draw: all {position sprite}, none {}\n", "{}");221 run_query(k&, draw_query);222 verify(k&, draw_query&);223 224 printfmt("query think: all {velocity health}, none {player}\n");225 run_query(k&, think_query);226 verify(k&, think_query&);227 228 printfmt("the drone is frozen: with(drone, frozen)\n");229 set_part(k&, drone, frozen, 1);230 verify(k&, null);231 232 printfmt("the ghost thaws: without(ghost, frozen)\n");233 set_part(k&, ghost, frozen, 0);234 verify(k&, null);235 236 printfmt("query move again: the drone stops, the ghost starts\n");237 run_query(k&, move_query);238 verify(k&, move_query&);239 240 hero: u64 = k.all[0].signature;241 count: u32 = 0;242 for (id: u32 = 0; id < 64; id += 1) {243 if (has(hero, id)) {244 count += 1;245 }246 }247 printfmt("the player's signature is the number {}: {} ids,", hero, count);248 for (id: u32 = 0; id < 64; id += 1) {249 if (has(hero, id)) {250 printfmt(" {}", id);251 }252 }253 printfmt("\n");254 return 0;255}What play_bitset prints. tools/check.py fails the playground if a single byte differs.
five kinds of thing, one bit per component, highest id first
player 00011111 {position velocity health sprite player}
drone 00001111 {position velocity health sprite}
turret 00101101 {position health sprite turret}
rock 00001001 {position sprite}
ghost 01001011 {position velocity sprite frozen}
query move: all {position velocity}, none {frozen}
all 00000011
none 01000000
player 00011111 matches
drone 00001111 matches
turret 00101101 no
rock 00001001 no
ghost 01001011 no
query draw: all {position sprite}, none {}
all 00001001
none 00000000
player 00011111 matches
drone 00001111 matches
turret 00101101 matches
rock 00001001 matches
ghost 01001011 matches
query think: all {velocity health}, none {player}
all 00000110
none 00010000
player 00011111 no
drone 00001111 matches
turret 00101101 no
rock 00001001 no
ghost 01001011 no
the drone is frozen: with(drone, frozen)
drone 01001111 {position velocity health sprite frozen}
the ghost thaws: without(ghost, frozen)
ghost 00001011 {position velocity sprite}
query move again: the drone stops, the ghost starts
all 00000011
none 01000000
player 00011111 matches
drone 01001111 no
turret 00101101 no
rock 00001001 no
ghost 00001011 matches
the player's signature is the number 31: 5 ids, 0 1 2 3 4
The dojo · block 7
An arena
dojo/7.arena/play_arena.rin
Most of what a frame allocates dies with the frame: commands, scratch lists, text for the screen. An arena hands all of it out of one buffer and keeps one number, how much of the buffer is used. A push rounds that number up to the alignment asked for, hands out the offset there, and moves the number past the new bytes. Nothing is freed on its own. A mark remembers the number, a rewind puts it back, and a clear sets it to 0: everything pushed since goes at once, in one store, with no list of what to free.
push(4, 4) with 1 used: round 1 up to 4, hand out offset 4, used is now 8
For an align that is a power of two, rounding up is (used + align - 1) & ~(align - 1): adding align - 1 reaches the next multiple unless used is one already, and the mask drops what's past it.
The invariant: every push not yet let go of starts at a multiple of its alignment, ends at or before used, and stops before the next push starts; used never passes the end of the buffer.
In the course: everywhere. A region is a memops_arena the checker can see. Every lesson from 2.2's columns on allocates from the region level, and never frees anything on its own. 3.3 adds a second, frame: region in level;, for one frame's commands, let go when it ends, and kayan does the same in 6. This block is the arena inside a region, small enough to write in a few minutes.
Watch it: the playground
python tools/dojo.py play arena
It pushes things of different sizes and alignments into a 64-byte buffer, rewinds, fills it to the last byte, and clears it. After each step it draws the buffer, 16 bytes to a group: a letter for each byte of a push, - for padding, . for free.
push b: a u32 count, 4 bytes, align 4
offset 4, used 8 of 64: 3 bytes of padding first
a---bbbb........|................|................|................
...
rewind to the mark: e and f are gone at once
a---bbbbcccccccc|dddddddddddddddd|................|................
Each push is filled with its letter, written through b*.bytes[offset], and verify checks after every step that each push is aligned, inside used, clear of the next, and still holds its letter. Step through it: break on bump_push and watch used. The playground keeps its 64 bytes inside the struct.
Then flip the switch at the top, align_offsets, to 0 and run it again. Pushes are handed out at used as it is, packed with no padding, and the u32, the u64 and the 16-byte vector land at offsets 1, 5 and 13:
verify: b is at offset 1, not a multiple of 4: a 4-byte thing can't be read from there
The playground only ever writes bytes, so nothing breaks yet, and safe code can't do anything else: reading a u32 out of bytes is a cast to a pointer, which takes unsafe { }. (Block 11 copies the bytes into a local with memcpy instead, which has no alignment to get wrong.) Do the cast from there, though, and it's undefined behaviour in C, so in rin: on x86 it usually works, slower when it straddles two cache lines; an aligned SIMD load faults; and the optimiser is allowed to assume it never happens. --sanitize reports it, as UBSan's "misaligned address".
Do reps: the kata
You write bump, an arena over bytes it gets from the region level. It isn't called arena, since the one every lesson uses already has that name: memops_arena.
bump: struct = {
bytes: []u8 in level;
// Bytes before `used` are handed out; bytes after it are free.
used: u64;
}
Its one proc with an idea in it:
bump_push: proc(b: *bump, size: u64, align: u64) -> option<u64> = {
the offset of size bytes, a multiple of align, or none, changing nothing, when they would end past the last byte. It can't be an address: a ^u8 has no region, so safe code can't hand one out. The caller writes and reads through bump_bytes(b)[offset], a slice the checker bounds-checks. Plus bump_make(r, capacity), which gets its bytes with new_slice<u8>(r, capacity), bump_bytes, which returns a []u8 in level, bump_used, bump_capacity, bump_mark, bump_rewind(b, mark) and bump_clear.
| level | you write | target |
|---|---|---|
| 1 | bump_push, in an otherwise finished file | 2:00 |
| 2 | every proc body; the fields and signatures are given | 6:00 |
| 3 | the whole file, from a blank page | 8:00 |
python tools/dojo.py start arena 2
python tools/dojo.py arena
python tools/dojo.py arena --sanitize the same with AddressSanitizer and UBSan
The harness runs the playground's steps as a scenario, then 20,000 random pushes, marks, rewinds and clears on a 256-byte buffer, against a model that keeps the used count and every push not yet let go of. Pushes are 1 to 24 bytes at alignments 1 to 16, with some sized to end exactly at the last byte and some one byte past it. Each push is filled with a pattern at the offset it gave, and read back later, so a push that lands on another shows. The seeds give the same runs as rust-dod's dojo.
rin notes. memops_arena_push hands out an address, and safe code never calls it: it is what a region is built on, behind new<T> and new_slice<T>. rust-dod's kata hands out an offset too, for its own reason: a slice from push would borrow the whole arena for as long as it was kept, and the arena couldn't push again until it was gone. An offset can still be kept past a rewind, which is the arena's one way to go wrong: an offset from before a rewind or a clear names bytes the next push will hand out again. In safe code that gets you the wrong bytes, and never memory outside the slice.
Level 4, variations. The same harness, different insides:
- Align the address, not the offset, which fails on purpose: round the address of byte
usedup, and subtract the address of byte 0 again. That takes a smallunsafe { },cast(b*.bytes[0]&, usize), since safe code can't turn a pointer into a number.memops_arenadoes it this way.new_slice<u8>aligns its bytes only as au8needs, so unless they happen to start on a multiple of 16, the harness soon finds an offset that isn't a multiple of its alignment. Which one is right, and what wouldbump_makehave to do for both to agree? - **
bump_push_zero**: the same, with the bytes zeroed. Rewind, push again, and read the bytes before zeroing: what's there? (Not checked by the harness; write your own test.) - Temporary scopes:
bump_temp_begin(b)returns a record of the mark andbump_temp_end(t)rewinds to it, likememops_arena_temp_beginand_end. The record keepsb, sobump_temp_begintakes a*bump in level, and the field is one too. What goes wrong if two scopes end out of order? (Not checked by the harness; write your own test.)
When it fails
FAIL in the scenario: push 4 bytes, align 4: 3 bytes of padding first gave 1, it should give 4
No rounding up at all: the offset is used as it is. The fuzz run is what finds the rest:
FAIL at operation 29 (seed 1): push(12, 2) gave offset 59, which isn't a multiple of 2
debug it: build/dojo/arena/harness --seed 1 --stop 29
with a breakpoint on dojo_stop_here, which runs just before that operation
That one rounds every alignment but 2, which the scenario never asks for. Run the harness under your debugger with those arguments and a breakpoint on dojo_stop_here, which runs just before that operation, and step into bump_push.
If the harness crashes instead, on an index past the end of the bytes say, index 257 out of bounds (length 256), the runner runs it again with --trace 1 and prints the operation that was running.
The playground. The kata's harness is dojo/7.arena/harness.rin, and the levels start from dojo/7.arena/level1.rin to level3.rin.
1// Dojo 7, the playground: an arena.2//3// An arena is one buffer and one number: how much of the buffer is used. A4// push rounds that number up to the alignment asked for, hands out the5// offset there, and moves the number past the new bytes. Nothing is freed on6// its own. A mark remembers the number, a rewind goes back to it, and a clear7// goes back to 0, and everything pushed since is gone at once, for the cost8// of one store. That is all the memory a frame of a game needs.9//10// This program pushes things of different sizes and alignments into a 64-byte11// buffer, rewinds, fills it to the last byte, and clears it. After each step it12// draws the buffer, 16 bytes to a group: a letter for each byte of a push, -13// for padding, . for free. Each push is filled with its letter, and verify14// checks the arena's invariant and that every push still holds its letter;15// break inside its `if`s to stop on the first thing wrong.16//17// The switch: set align_offsets to 0 and run it again. A push then hands out18// the used count as it is, without rounding it up, and the u32, the u64 and19// the 16-byte vector land at offsets they can't be read from as what they20// are. It's a global, read each time it's used, so a debugger can flip it21// mid-run too: stop on bump_push, set it to 0 in the watch window, and22// continue.23//24// A push hands out an offset into the arena's bytes, not an25// address, and the program writes its letters through `b*.bytes[offset]`,26// which is bounds-checked. The original handed out an address and wrote27// through it with memset; see README.md.28 29import "std/print.rin"30import "std/option.rin"31 32align_offsets: b32 = 1;33 34bump: struct = {35 // 64 bytes. An offset that is a multiple of 16, or of anything less, is36 // where a thing of that alignment can go.37 bytes: [64]u8;38 // Bytes before `used` are handed out; bytes after it are free.39 used: u64;40}41 42// The offset of `size` bytes at a multiple of `align`, a power of two, or43// none if they don't fit before the end.44bump_push: proc(b: *bump, size: u64, align: u64) -> option<u64> = {45 offset: u64 = b*.used;46 if (align_offsets) {47 offset = (b*.used + align - 1) & ~(align - 1);48 }49 if (offset + size > 64) {50 return option<u64>none();51 }52 b*.used = offset + size;53 return option<u64>some(offset);54}55 56bump_mark: proc(b: *bump) -> u64 = {57 return b*.used;58}59 60bump_rewind: proc(b: *bump, mark: u64) -> void = {61 b*.used = mark;62}63 64bump_clear: proc(b: *bump) -> void = {65 b*.used = 0;66}67 68// What the program has pushed and not yet let go of, to draw and to check:69// up to eight pushes, each a letter, an offset, a size and an alignment.70push_record: struct = {71 letter: u8;72 offset: u64;73 size: u64;74 align: u64;75}76 77pushes: struct = {78 live: [8]push_record;79 count: u32;80}81 82push: proc(b: *bump, ps: *pushes, letter: u8, size: u64, align: u64) -> void = {83 before: u64 = b*.used;84 p: option<u64> = bump_push(b, size, align);85 if (option<u64>is_none(p)) {86 if (size == 1) {87 printfmt(" no room for 1 byte after {} of 64\n", before);88 } else {89 printfmt(" no room for {} bytes after {} of 64\n", size, before);90 }91 return;92 }93 offset: u64 = option<u64>unwrap(p);94 for (i: u64 = 0; i < size; i += 1) {95 b*.bytes[offset + i] = letter;96 }97 ps*.live[ps*.count] = { .letter = letter, .offset = offset, .size = size, .align = align };98 ps*.count += 1;99 padding: u64 = offset - before;100 printfmt(" offset {}, used {} of 64", offset, b*.used);101 if (padding == 0 && b*.used == 64) {102 printfmt(": the last byte");103 } else if (padding == 1) {104 printfmt(": 1 byte of padding first");105 } else if (padding > 1) {106 printfmt(": {} bytes of padding first", padding);107 }108 printfmt("\n");109}110 111// Rewinding or clearing lets go of every push from after the mark.112let_go: proc(b: *bump, ps: *pushes) -> void = {113 while (ps*.count > 0 && ps*.live[ps*.count - 1].offset >= b*.used) {114 ps*.count -= 1;115 }116}117 118draw: proc(b: *bump, ps: *pushes) -> void = {119 cells: [64]u8 = {};120 for (i: u64 = 0; i < 64; i += 1) {121 cells[i] = i < b*.used ? '-' : '.';122 }123 for (k: u32 = 0; k < ps*.count; k += 1) {124 for (i: u64 = 0; i < ps*.live[k].size; i += 1) {125 cells[ps*.live[k].offset + i] = ps*.live[k].letter;126 }127 }128 printfmt(" ");129 for (i: u64 = 0; i < 64; i += 1) {130 if (i > 0 && i % 16 == 0) {131 printfmt("|");132 }133 printfmt("{}", cast(cells[i], c8));134 }135 printfmt("\n");136}137 138// The invariant: every push is aligned, inside the used bytes, clear of the139// one after it, and still holds its letter.140verify: proc(b: *bump, ps: *pushes) -> void = {141 if (b*.used > 64) {142 printfmt(" verify: used is {}, past the end of the buffer\n", b*.used);143 }144 for (k: u32 = 0; k < ps*.count; k += 1) {145 p: push_record = ps*.live[k];146 name: c8 = cast(p.letter, c8);147 if (p.offset % p.align != 0) {148 printfmt(" verify: {} is at offset {}, not a multiple of {}: a {}-byte thing can't be read from there\n", name,149 p.offset, p.align, p.align);150 }151 if (p.offset + p.size > b*.used) {152 printfmt(" verify: {} ends at {}, past the used count, {}\n", name, p.offset + p.size, b*.used);153 }154 if (k + 1 < ps*.count && p.offset + p.size > ps*.live[k + 1].offset) {155 printfmt(" verify: {} runs into {}\n", name, cast(ps*.live[k + 1].letter, c8));156 }157 for (i: u64 = 0; i < p.size; i += 1) {158 if (b*.bytes[p.offset + i] != p.letter) {159 printfmt(" verify: {}'s bytes have been written over\n", name);160 break;161 }162 }163 }164}165 166main: proc() -> i32 = {167 b: bump = {};168 ps: pushes = {};169 printfmt("a 64-byte buffer, 16 bytes to a group: a letter per byte pushed, - for padding, . for free\n");170 171 printfmt("push a: a u8 flag, 1 byte, align 1\n");172 push(b&, ps&, 'a', 1, 1);173 draw(b&, ps&);174 verify(b&, ps&);175 176 printfmt("push b: a u32 count, 4 bytes, align 4\n");177 push(b&, ps&, 'b', 4, 4);178 draw(b&, ps&);179 verify(b&, ps&);180 181 printfmt("push c: a u64 id, 8 bytes, align 8\n");182 push(b&, ps&, 'c', 8, 8);183 draw(b&, ps&);184 verify(b&, ps&);185 186 printfmt("push d: four f32s for SIMD, 16 bytes, align 16\n");187 push(b&, ps&, 'd', 16, 16);188 draw(b&, ps&);189 verify(b&, ps&);190 191 mark: u64 = bump_mark(b&);192 printfmt("mark: remember used, {}\n", mark);193 194 printfmt("push e: a name, 3 bytes, align 1\n");195 push(b&, ps&, 'e', 3, 1);196 draw(b&, ps&);197 verify(b&, ps&);198 199 printfmt("push f: a u64 time, 8 bytes, align 8\n");200 push(b&, ps&, 'f', 8, 8);201 draw(b&, ps&);202 verify(b&, ps&);203 204 printfmt("rewind to the mark: e and f are gone at once\n");205 bump_rewind(b&, mark);206 let_go(b&, ps&);207 draw(b&, ps&);208 verify(b&, ps&);209 210 printfmt("push g: a path, 40 bytes, align 8\n");211 push(b&, ps&, 'g', 40, 8);212 draw(b&, ps&);213 verify(b&, ps&);214 215 printfmt("push h: a 2x4 matrix of f32s, 32 bytes, align 16\n");216 push(b&, ps&, 'h', 32, 16);217 draw(b&, ps&);218 verify(b&, ps&);219 220 printfmt("push i: one more byte\n");221 push(b&, ps&, 'i', 1, 1);222 draw(b&, ps&);223 verify(b&, ps&);224 225 printfmt("clear: the frame is over, and everything goes at once\n");226 bump_clear(b&);227 let_go(b&, ps&);228 draw(b&, ps&);229 verify(b&, ps&);230 231 printfmt("push j: the next frame's first u64, over a's old bytes\n");232 push(b&, ps&, 'j', 8, 8);233 draw(b&, ps&);234 verify(b&, ps&);235 return 0;236}What play_arena prints. tools/check.py fails the playground if a single byte differs.
a 64-byte buffer, 16 bytes to a group: a letter per byte pushed, - for padding, . for free
push a: a u8 flag, 1 byte, align 1
offset 0, used 1 of 64
a...............|................|................|................
push b: a u32 count, 4 bytes, align 4
offset 4, used 8 of 64: 3 bytes of padding first
a---bbbb........|................|................|................
push c: a u64 id, 8 bytes, align 8
offset 8, used 16 of 64
a---bbbbcccccccc|................|................|................
push d: four f32s for SIMD, 16 bytes, align 16
offset 16, used 32 of 64
a---bbbbcccccccc|dddddddddddddddd|................|................
mark: remember used, 32
push e: a name, 3 bytes, align 1
offset 32, used 35 of 64
a---bbbbcccccccc|dddddddddddddddd|eee.............|................
push f: a u64 time, 8 bytes, align 8
offset 40, used 48 of 64: 5 bytes of padding first
a---bbbbcccccccc|dddddddddddddddd|eee-----ffffffff|................
rewind to the mark: e and f are gone at once
a---bbbbcccccccc|dddddddddddddddd|................|................
push g: a path, 40 bytes, align 8
no room for 40 bytes after 32 of 64
a---bbbbcccccccc|dddddddddddddddd|................|................
push h: a 2x4 matrix of f32s, 32 bytes, align 16
offset 32, used 64 of 64: the last byte
a---bbbbcccccccc|dddddddddddddddd|hhhhhhhhhhhhhhhh|hhhhhhhhhhhhhhhh
push i: one more byte
no room for 1 byte after 64 of 64
a---bbbbcccccccc|dddddddddddddddd|hhhhhhhhhhhhhhhh|hhhhhhhhhhhhhhhh
clear: the frame is over, and everything goes at once
................|................|................|................
push j: the next frame's first u64, over a's old bytes
offset 0, used 8 of 64
jjjjjjjj........|................|................|................
The dojo · block 8
Index, not pointer
dojo/8.index/play_index.rin
A linked list whose nodes live side by side in one vec, and link to each other by index: node 1's next is 0, a position in the vec, not an address. When the vec is full and grows, it moves: the nodes are copied to a bigger place, and the old one is let go. Every index is still right after the move, because an index says where a node is in the vec, wherever the vec is. A pointer says where the node is in memory, so it still points at the old place, and nothing tells you.
That's the whole idea, and it's everywhere in data-oriented code: an ECS keeps an entity's row by number because a growing column moves its rows, a mesh keeps triangles as three vertex indices, a scene graph keeps a parent index. A freed node goes on a free list and is used again before the vec grows, as in dojo 1.
The invariant: walking next from the head visits every live node once and ends at the tail; each node's prev is the node before it; and a node's index names it from the moment it's made until it's removed, however many times the vec moves in between.
In the course: 2.4 keeps an entity's table and row as numbers in its slot. It's why a pointer into 2.2's columns goes stale when a column grows, and why 2.5 says to keep handles and borrow pointers only to use them. The tables themselves are level pointers in a vec, so the vec can grow without moving a table: the other way out, for a thing that never has to move.
Watch it: the playground
python tools/dojo.py play index
Memory in the playground is an array of 16 cells, small enough to see whole in a debugger, and the list takes 2 cells of it, then 4, then 8. It keeps node b two ways: by its index, and by a *node to its cell. Then the list fills up and moves, and the program writes to b:
push_back 30 as c: the list is full
the list moves from cells 0..2 to cells 2..6
c is node 2
...
write 21 into b through its index, 1: cell 3 now
nodes 0: 10, prev -, next 1 | 1: 21, prev 0, next 2 | 2: 30, prev 1, next -
list 10 21 30
memory 0..2 old copy: 10 20 | 2..6 the list: 10 21 30 .
After every step verify walks the links and checks that every node holds what the program last wrote to it. Put a breakpoint on broken, which it calls when something is wrong.
Then break on the line that writes 21 into b, set use_index to 0 in the watch window, and continue. The write goes through the kept pointer, into the old copy, and the list never sees it:
write 21 into b through what was kept, cell 1: where b was before the move
...
memory 0..2 old copy: 10 21 | 2..6 the list: 10 20 30 .
verify: node 1 holds 20, but the program last wrote 21 to it
Nothing crashes, and AddressSanitizer stays quiet too: the old copy is still memory the program owns, as a region's old blocks are until the region ends. The write is simply lost. The checker lets the playground keep that pointer because its cells are a fixed array inside the list, so to it nothing has moved. In the kata the nodes are in a vec, and it won't.
Do reps: the kata
You write list, made in the region level, with nodes and free each a *vec<...> in level. The idea is in list_remove: the two neighbours link to each other, past the node, and the node's index goes on the free list. The node's fields are read first, and it is marked free, before the calls that touch other nodes, since each of those ends what n may be used for:
list_remove: proc(l: *list, node: u32) -> option<u64> = {
n: option<*list_node> = list_live(l, node);
if (!n) {
return option<u64>none();
}
prev: u32 = n*.prev;
next: u32 = n*.next;
value: u64 = n*.value;
n*.alive = 0;
if (prev != list_none) {
list_at(l, prev)*.next = next;
} else {
l*.head = next;
}
if (next != list_none) {
list_at(l, next)*.prev = prev;
} else {
l*.tail = prev;
}
and the playground's lesson is one the checker holds you to, in list_insert_after:
// A new node between `node` and the one after it; list_none if `node` isn't
// in the list. Every pointer into the nodes is taken after list_take, which
// can move them.
list_insert_after: proc(l: *list, node: u32, value: u64) -> u32 = {
plus list_make(r), which sets head and tail to list_none, list_push_back, list_push_front, list_get, list_len, and the links to walk the list by: list_head, list_tail, list_next and list_prev. A link past either end is list_none, the largest u32, which no node ever gets.
| level | you write | target |
|---|---|---|
| 1 | list_remove, in an otherwise finished file | 3:00 |
| 2 | every proc body; the fields and signatures are given | 12:00 |
| 3 | the whole file, from a blank page | 15:00 |
python tools/dojo.py start index 2
python tools/dojo.py index
The harness runs the playground's steps as a scenario, then 20,000 random pushes, inserts, removes, reads and writes against a model that keeps the live nodes in list order. After every operation it walks your list both ways, list_head to the end by list_next, and list_tail back by list_prev, so a link left wrong is caught by the operation that left it. The list grows to about a thousand nodes, moving many times, and every node is checked by the index it was given. A freed index must be handed out again before a new one is made. The seeds give the same runs as rust-dod's dojo.
rin notes. Rust's kata has Option<u32> links; here a link is a u32, and list_none stands for none, as UINT32_MAX would in C. list_get returns an option<*u64 in level>, a pointer that is null for a node not in the list and tested before use, to read or to change the value; Rust's kata has get and get_mut. Helpers that hand out a node say the same, list_at a *list_node in level and list_live an option<*list_node in level>. The checker refuses a result that comes out of a vec in level and doesn't say so.
Level 4, variations. The same harness, different insides:
- The free list through the nodes: a freed node's
nextis the next free node, and the list keepsfirst_free: u32. Nofreevec. - **A pointer kept across
list_take**, which fails on purpose, twice: inlist_insert_after, takelist_live(l, node)once, beforelist_take, and write through it after. The checker stops it first:'n' may be stale: 'l', which it was taken from, has been changed since; take it again. Now declare itn: option<*list_node in level>, a pointer good for as long as the region, which is true: the old block is still there. It builds. Which operation does the harness stop at, and where did the write go? Then put it back. - A list walker:
list_each(l, visit), withvisita*proc(node: u32, value: u64) -> void, called head to tail. Read the next link before you call it: the checker counts a call through a proc pointer as one that may change the list. (Not checked by the harness; write your own test.)
When it fails
FAIL at operation 17 (seed 1): walking head to tail, list_next(3) gives 2, but 3 is the last of 3 nodes
debug it: build/dojo/index/harness --seed 1 --stop 17
with a breakpoint on dojo_stop_here, which runs just before that operation
A link pointing somewhere it shouldn't is usually a node from the free list that kept its old links, or a neighbour that wasn't told about a change: an insert at the tail that didn't move the tail, a remove that fixed next but not prev, or a write through a pointer kept in level past a move. Run the harness under your debugger with those arguments and a breakpoint on dojo_stop_here, which runs just before that operation, and look at the nodes the message names.
If the harness crashes instead, on a node past the end of the vec say, the runner runs it again with --trace 1 and prints the operation that was running, or says the crash came in the scenario, after the last step printed.
An index alone can't tell a removed node from the new node that took its place: a is node 0, and so is e after it. That's dojo 1's problem, and its answer, a generation beside the index, works here unchanged.
The playground. The kata's harness is dojo/8.index/harness.rin, and the levels start from dojo/8.index/level1.rin to level3.rin.
1// Dojo 8, the playground: index, not pointer.2//3// A linked list whose nodes live side by side in one array and link to each4// other by index: a node's next is "node 2", not an address. When the array is5// full it moves: the nodes are copied to a place twice the size, and the old6// place is let go. An index is a position in the array, so it finds its node7// after the move. A pointer is a place in memory, so it still points where8// the node was.9//10// Memory here is an array of 16 cells, small enough to see whole in a11// debugger's watch window. The list takes 2 cells, then 4, then 8: cells 0..2,12// then 2..6, then 6..14. Each move leaves the old copy where it was, the way13// freed memory keeps its bytes until something else is put there.14//15// The program keeps node b two ways: its index, and a pointer to it. Then the16// list moves, and the program writes to b. verify() runs after every step: it17// checks the links, and that every node holds what the program last wrote to18// it. Put a breakpoint on `broken` to stop where it finds something wrong.19//20// The switch: use_index. Break on the line that writes 21 into b, set it to 021// in the watch window, and continue. The write goes through the kept pointer22// to the old copy, and verify reports it lost.23 24import "std/print.rin"25import "std/option.rin"26 27use_index: b32 = 1;28 29// No node: the link past either end.30no_node: u32 = 0xffffffff'u32;31 32node: struct = {33 value: u64;34 prev: u32;35 next: u32;36 alive: b32;37}38 39list: struct = {40 // All of memory. Node i is in cell base + i, and the list owns cells41 // base..base + capacity.42 memory: [16]node;43 base: u32;44 capacity: u32;45 // Nodes made so far are 0..made; the freed ones wait in free.46 made: u32;47 free: [8]u32;48 free_count: u32;49 head: u32;50 tail: u32;51}52 53// Where node `index` is now.54list_node: proc(l: *list, index: u32) -> *node = {55 return l*.memory[l*.base + index]&;56}57 58// A node for `value`: a freed one, or a new one, moving the list to twice the59// room first when it's full.60list_take: proc(l: *list, value: u64) -> u32 = {61 index: u32 = 0;62 if (l*.free_count > 0) {63 l*.free_count -= 1;64 index = l*.free[l*.free_count];65 } else {66 if (l*.made == l*.capacity) {67 to: u32 = l*.base + l*.capacity;68 printfmt(" the list moves from cells {}..{} to cells {}..{}\n", l*.base, to, to, to + 2 * l*.capacity);69 for (i: u32 = 0; i < l*.made; i += 1) {70 l*.memory[to + i] = l*.memory[l*.base + i];71 }72 l*.base = to;73 l*.capacity *= 2;74 }75 l*.made += 1;76 index = l*.made - 1;77 }78 list_node(l, index)* = { .value = value, .prev = no_node, .next = no_node, .alive = 1 };79 return index;80}81 82list_push_back: proc(l: *list, value: u64) -> u32 = {83 index: u32 = list_take(l, value);84 list_node(l, index)*.prev = l*.tail;85 if (l*.tail != no_node) {86 list_node(l, l*.tail)*.next = index;87 } else {88 l*.head = index;89 }90 l*.tail = index;91 return index;92}93 94list_push_front: proc(l: *list, value: u64) -> u32 = {95 index: u32 = list_take(l, value);96 list_node(l, index)*.next = l*.head;97 if (l*.head != no_node) {98 list_node(l, l*.head)*.prev = index;99 } else {100 l*.tail = index;101 }102 l*.head = index;103 return index;104}105 106list_insert_after: proc(l: *list, at: u32, value: u64) -> u32 = {107 index: u32 = list_take(l, value);108 next: u32 = list_node(l, at)*.next;109 list_node(l, index)* = { .value = value, .prev = at, .next = next, .alive = 1 };110 list_node(l, at)*.next = index;111 if (next != no_node) {112 list_node(l, next)*.prev = index;113 } else {114 l*.tail = index;115 }116 return index;117}118 119list_remove: proc(l: *list, index: u32) -> void = {120 n: *node = list_node(l, index);121 if (n*.prev != no_node) {122 list_node(l, n*.prev)*.next = n*.next;123 } else {124 l*.head = n*.next;125 }126 if (n*.next != no_node) {127 list_node(l, n*.next)*.prev = n*.prev;128 } else {129 l*.tail = n*.prev;130 }131 n*.alive = 0;132 l*.free[l*.free_count] = index;133 l*.free_count += 1;134}135 136// A link, or - for none.137print_link: proc(link: u32) -> void = {138 if (link == no_node) {139 printfmt("-");140 } else {141 printfmt("{}", link);142 }143}144 145// A value, or - for none.146print_value: proc(value: option<u64>) -> void = {147 if (value.has_value) {148 printfmt("{}", value.value);149 } else {150 printfmt("-");151 }152}153 154draw: proc(l: *list) -> void = {155 printfmt(" nodes ");156 for (i: u32 = 0; i < l*.made; i += 1) {157 if (i > 0) {158 printfmt(" | ");159 }160 n: *node = l*.memory[l*.base + i]&;161 if (n*.alive) {162 printfmt("{}: {}, prev ", i, n*.value);163 print_link(n*.prev);164 printfmt(", next ");165 print_link(n*.next);166 } else {167 printfmt("{}: free", i);168 }169 }170 printfmt("\n list ");171 count: u32 = 0;172 for (at: u32 = l*.head; at != no_node && count < l*.made; at = l*.memory[l*.base + at].next) {173 if (count > 0) {174 printfmt(" ");175 }176 printfmt("{}", l*.memory[l*.base + at].value);177 count += 1;178 }179 printfmt("\n");180}181 182// Every place the list has lived, oldest first, and what's in each cell.183draw_memory: proc(l: *list) -> void = {184 printfmt(" memory ");185 from: u32 = 0;186 size: u32 = 2;187 while (from <= l*.base) {188 if (from > 0) {189 printfmt(" | ");190 }191 if (from == l*.base) {192 printfmt("{}..{} the list:", from, from + size);193 } else {194 printfmt("{}..{} old copy:", from, from + size);195 }196 for (c: u32 = from; c < from + size; c += 1) {197 if (l*.memory[c].alive) {198 printfmt(" {}", l*.memory[c].value);199 } else {200 printfmt(" .");201 }202 }203 from += size;204 size *= 2;205 }206 printfmt("\n");207}208 209// verify calls this when it finds something wrong, then says what.210broken: proc() -> void = {211 printfmt(" verify: ");212}213 214// The list's invariant, and the program's: walking from the head reaches215// every live node once, each node's prev is the one before it, the walk ends216// at the tail, and every node holds what the program last wrote to it.217verify: proc(l: *list, wrote: []option<u64>) -> void = {218 before: u32 = no_node;219 linked: u32 = 0;220 for (at: u32 = l*.head; at != no_node; at = l*.memory[l*.base + at].next) {221 n: *node = l*.memory[l*.base + at]&;222 if (!n*.alive) {223 broken();224 printfmt("node {} is linked, but it's free\n", at);225 }226 if (n*.prev != before) {227 broken();228 printfmt("node {} comes after ", at);229 print_link(before);230 printfmt(", but its prev is ");231 print_link(n*.prev);232 printfmt("\n");233 }234 linked += 1;235 if (linked > l*.made) {236 broken();237 printfmt("the links go round in a loop\n");238 break;239 }240 before = at;241 }242 if (l*.tail != before) {243 broken();244 printfmt("the list ends at ");245 print_link(before);246 printfmt(", but the tail is ");247 print_link(l*.tail);248 printfmt("\n");249 }250 alive: u32 = 0;251 for (i: u32 = 0; i < l*.made; i += 1) {252 n: *node = l*.memory[l*.base + i]&;253 holds: option<u64> = option<u64>none();254 if (n*.alive) {255 holds = option<u64>some(n*.value);256 alive += 1;257 }258 want: option<u64> = wrote[i];259 if (holds.has_value != want.has_value || holds.value != want.value) {260 broken();261 printfmt("node {} holds ", i);262 print_value(holds);263 printfmt(", but the program last wrote ");264 print_value(want);265 printfmt(" to it\n");266 }267 }268 if (linked != alive) {269 broken();270 printfmt("{} nodes are linked, but {} are alive\n", linked, alive);271 }272}273 274main: proc() -> i32 = {275 l: list = {};276 l.capacity = 2;277 l.head = no_node;278 l.tail = no_node;279 // What the program last wrote to each node, for verify.280 wrote: [8]option<u64> = {};281 printfmt("a list with room for 2 nodes, in cells 0..2 of memory\n");282 283 printfmt("push_back 10 as a\n");284 a: u32 = list_push_back(l&, 10);285 wrote[a] = option<u64>some(10);286 printfmt(" a is node {}\n", a);287 draw(l&);288 verify(l&, wrote);289 290 printfmt("push_back 20 as b\n");291 b: u32 = list_push_back(l&, 20);292 wrote[b] = option<u64>some(20);293 printfmt(" b is node {}\n", b);294 draw(l&);295 verify(l&, wrote);296 297 kept: *node = list_node(l&, b);298 kept_cell: u32 = l.base + b;299 printfmt("keep b's index, {}, and where it is, cell {}, which is what a pointer holds: the same, until the list moves\n",300 b, kept_cell);301 302 printfmt("push_back 30 as c: the list is full\n");303 c: u32 = list_push_back(l&, 30);304 wrote[c] = option<u64>some(30);305 printfmt(" c is node {}\n", c);306 draw(l&);307 verify(l&, wrote);308 309 if (use_index) {310 printfmt("write 21 into b through its index, {}: cell {} now\n", b, l.base + b);311 list_node(l&, b)*.value = 21;312 } else {313 printfmt("write 21 into b through what was kept, cell {}: where b was before the move\n", kept_cell);314 kept*.value = 21;315 }316 wrote[b] = option<u64>some(21);317 draw(l&);318 draw_memory(l&);319 verify(l&, wrote);320 321 printfmt("push_front 5 as d\n");322 d: u32 = list_push_front(l&, 5);323 wrote[d] = option<u64>some(5);324 printfmt(" d is node {}\n", d);325 draw(l&);326 verify(l&, wrote);327 328 printfmt("remove a: node {} goes on the free list\n", a);329 list_remove(l&, a);330 wrote[a] = option<u64>none();331 draw(l&);332 verify(l&, wrote);333 334 printfmt("insert 25 after b as e\n");335 e: u32 = list_insert_after(l&, b, 25);336 wrote[e] = option<u64>some(25);337 printfmt(" e is node {}: a's node, from the free list\n", e);338 draw(l&);339 verify(l&, wrote);340 341 printfmt("push_back 40 as f: the list is full again\n");342 f: u32 = list_push_back(l&, 40);343 wrote[f] = option<u64>some(40);344 printfmt(" f is node {}\n", f);345 draw(l&);346 draw_memory(l&);347 verify(l&, wrote);348 349 printfmt("read b through its index, {}: {}\n", b, list_node(l&, b)*.value);350 return 0;351}What play_index prints. tools/check.py fails the playground if a single byte differs.
a list with room for 2 nodes, in cells 0..2 of memory
push_back 10 as a
a is node 0
nodes 0: 10, prev -, next -
list 10
push_back 20 as b
b is node 1
nodes 0: 10, prev -, next 1 | 1: 20, prev 0, next -
list 10 20
keep b's index, 1, and where it is, cell 1, which is what a pointer holds: the same, until the list moves
push_back 30 as c: the list is full
the list moves from cells 0..2 to cells 2..6
c is node 2
nodes 0: 10, prev -, next 1 | 1: 20, prev 0, next 2 | 2: 30, prev 1, next -
list 10 20 30
write 21 into b through its index, 1: cell 3 now
nodes 0: 10, prev -, next 1 | 1: 21, prev 0, next 2 | 2: 30, prev 1, next -
list 10 21 30
memory 0..2 old copy: 10 20 | 2..6 the list: 10 21 30 .
push_front 5 as d
d is node 3
nodes 0: 10, prev 3, next 1 | 1: 21, prev 0, next 2 | 2: 30, prev 1, next - | 3: 5, prev -, next 0
list 5 10 21 30
remove a: node 0 goes on the free list
nodes 0: free | 1: 21, prev 3, next 2 | 2: 30, prev 1, next - | 3: 5, prev -, next 1
list 5 21 30
insert 25 after b as e
e is node 0: a's node, from the free list
nodes 0: 25, prev 1, next 2 | 1: 21, prev 3, next 0 | 2: 30, prev 0, next - | 3: 5, prev -, next 1
list 5 21 25 30
push_back 40 as f: the list is full again
the list moves from cells 2..6 to cells 6..14
f is node 4
nodes 0: 25, prev 1, next 2 | 1: 21, prev 3, next 0 | 2: 30, prev 0, next 4 | 3: 5, prev -, next 1 | 4: 40, prev 2, next -
list 5 21 25 30 40
memory 0..2 old copy: 10 20 | 2..6 old copy: 25 21 30 5 | 6..14 the list: 25 21 30 5 40 . . .
read b through its index, 1: 21
The dojo · block 9
Interning
dojo/9.interning/play_interning.rin
An interner gives each distinct key a small number, an id: the next one, 0, 1, 2, the first time it sees the key, and the same one every time after. Find it, or make it. The ids are dense, so they index plain vecs: the key of id 3 is keys[3], and whatever else belongs to that key can live at index 3 of its own vec.
Behind it is a hash table, written here by hand: a row of slots, each empty or holding an id. A key's hash says which slot to look in first. If that one holds another key, look in the next one, and the next, wrapping round at the end, until the key or an empty slot turns up. This is linear probing. A full table would make those runs long, and an absent key's run endless, so the table doubles before it's more than 3/4 full, and every id is put back where its key's hash says in the bigger table: a different slot, since the hash now picks from twice as many.
The invariant: every id is in exactly one slot, and a walk from its key's hash reaches that slot before it reaches an empty one; the table is never more than 3/4 full; and ids are 0 to len - 1, in the order their keys first came.
In the course: 2.4's kayan_table_for is an interner. Its keys are signatures and its ids are table indices, with std's hashmap doing the lookup, and std's hashmap is this same design: linear probing, a power of two, grown before 3/4. Here you write it. 2.6 then makes the lookup rare, by remembering where each add and remove leads, and dojo 10 practises that.
Watch it: the playground
python tools/dojo.py play interning
It interns the signatures of a few kinds of thing in a game, in a table of 8 slots that grows to 16, and prints every slot it looks at:
intern rock, signature 7
slots looked at: 4 5 6; new: id 3, in slot 6
table 1:turret 4:drone 5:player 6:rock, 4 of 8 slots used
...
intern bird, signature 15
slots looked at: 1 2; not there
7 ids would fill more than 3/4 of 8 slots: the table grows to 16, and every id
goes where its key's hash says now
slots looked at: 9 10; new: id 6, in slot 10
table 4:rock 5:player 7:chest 8:tank 9:turret 10:bird 12:drone, 7 of 16 slots used
After every step verify checks the invariant; put a breakpoint on broken, which it calls when something is wrong.
Then break on the first interner_intern, set mix_hash to 0 in the watch window, and continue. The hash is now the key itself, so the first slot is the key's lowest bits. Those are the components nearly every kind has, position, sprite and health, since they were made first, so every kind starts at the same slot and the runs pile up:
intern chest, signature 135
slots looked at: 7 0 1 2 3; new: id 4, in slot 3
...
slots looked at, in all: 41
against 17 with the hash. Nothing is wrong, only slow, so verify stays quiet. Flip it later instead, with ids already in the table, and verify complains straight away: the ids are where the old hash put them, and the new hash looks somewhere else. A table's hash can't change under it.
Do reps: the kata
You write interner, its table included, made in the region level, with slots and keys each a *vec<...> in level. The idea is in interner_intern:
interner_intern: proc(t: *interner, key: u64) -> u32 = {
at: u64 = interner_slot_of(t, key);
if (vec<u32>at(t*.slots, at)* != 0) {
return vec<u32>at(t*.slots, at)* - 1;
}
if ((t*.keys*.length + 1) * 4 > t*.slots*.length * 3) {
interner_grow(t);
at = interner_slot_of(t, key);
}
and the API around it:
interner_find: proc(t: *interner, key: u64) -> option<u32> = {
interner_key: proc(t: *interner, id: u32) -> u64 = {
plus interner_make(r) and interner_len. A slot holds id + 1, so that 0 can mean empty and key 0 is a key like any other. interner_make has 8 empty slots to push, which level 2's version leaves to you. Growing doesn't need a new vec, or a region: interner_grow clears slots, pushes twice as many zeros, and puts every id back.
| level | you write | target |
|---|---|---|
| 1 | interner_intern, in an otherwise finished file | 3:00 |
| 2 | every proc body; the fields and signatures are given | 10:00 |
| 3 | the whole file, from a blank page | 12:00 |
python tools/dojo.py start interning 2
python tools/dojo.py interning
The harness runs the playground's steps as a scenario, then 20,000 random interns and finds against a model, std's hashmap and a vec of the keys in order. The keys are the kinds an ECS's tables meet: small numbers, 0 among them; signatures with a few bits set; keys that differ only in their high bits; and any u64 at all. Ids must be dense, in the order keys were first interned, and never change; interner_find must agree with interner_intern; and interner_key must give the key back. It checks what the interner does, not how fast: an interner with no mixing in its hash passes, and the playground shows what it costs. The seeds give the same runs as rust-dod's dojo.
Level 4, variations. The same harness, different insides:
- The hash beside the id: two u32s a slot, the id and the low bits of its key's hash, so a run compares hashes and reads
keysonly when they match, andinterner_grownever hashes a key again. - Robin Hood: on the way along a run, an id that is closer to its home slot than the new one gives its slot up and moves on instead. The longest run gets much shorter. How does
interner_findstop early on an absent key? - Grow at 1/2: what happens to the slots looked at in the playground, and to the memory?
When it fails
FAIL in the scenario: find turret gave none, it should give 2
A key that was interned and can't be found usually means interner_grow copied the slots across instead of putting each id back by its key's hash: in a table twice the size, a key's run starts somewhere else. A run that steps off the end without wrapping stops the program with rin runtime: vec index out of bounds, and the runner runs it again with --trace 1 to show the operation that was running. If the run stops and never ends, the table filled up: a lookup for a key that isn't there walks round and round, never meeting an empty slot to stop at. Break into it in your debugger and look at slots.
In the fuzz run, a failure says which operation and seed, as in the other blocks; run the harness under your debugger with the --seed and --stop it gives, and a breakpoint on dojo_stop_here.
The playground. The kata's harness is dojo/9.interning/harness.rin, and the levels start from dojo/9.interning/level1.rin to level3.rin.
1// Dojo 9, the playground: interning.2//3// An interner gives each distinct key a small id, the first time it sees the4// key, and the same id every time after: find it, or make it. The ECS does5// this with signatures, to find the table for a set of components. Behind it6// is a hash table: a row of slots, each empty or holding an id. A key's hash7// says which slot to look in first; if that one holds another key, look in the8// next, and the next, until the key or an empty slot turns up. A table more9// than 3/4 full makes those runs long, so it doubles before that, and every id10// is put back where its key's hash says in the bigger table.11//12// Here the table has 8 slots, then 16, small enough to see whole in a13// debugger's watch window. The keys are the signatures of a few kinds of14// thing in a game: a bit per component, with the common components first.15//16// verify() runs after every step: every id must be reachable from its key's17// hash without crossing an empty slot, and the table at most 3/4 full. Put a18// breakpoint on `broken` to stop where it finds something wrong.19//20// The switch: mix_hash. Break on the first intern, set it to 0 in the watch21// window, and continue. The hash is then the key itself, and the slot its22// lowest bits: the components every kind of thing has. Watch the runs get23// long, and the count of slots looked at at the end. Flip it later instead,24// with ids already in the table, and verify says why a table's hash can't25// change under it.26 27import "std/print.rin"28 29mix_hash: b32 = 1;30 31// The kinds of thing, and their signatures, which main fills in.32kind_names: [7]^const char = { "player", "drone", "turret", "rock", "chest", "tank", "bird" };33kind_keys: [7]u64 = {};34 35name: proc(key: u64) -> ^const char = {36 for (i: u32 = 0; i < 7; i += 1) {37 if (kind_keys[i] == key) {38 return kind_names[i];39 }40 }41 return "?";42}43 44// splitmix64's finisher, or, with the switch off, the key as it is.45hash: proc(key: u64) -> u64 = {46 if (!mix_hash) {47 return key;48 }49 x: u64 = key;50 x ^= x shr 30;51 x *= 0xbf58476d1ce4e5b9'u64;52 x ^= x shr 27;53 x *= 0x94d049bb133111eb'u64;54 return x ^ (x shr 31);55}56 57interner: struct = {58 // slots[0..capacity] is the table: 0 is empty, anything else an id + 1.59 slots: [16]u32;60 capacity: u32;61 keys: [12]u64;62 length: u32;63 // Slots looked at, in all.64 probes: u32;65}66 67// Where `key` is, or the empty slot where it would go, printing the slots it68// looks at on the way.69interner_slot_of: proc(t: *interner, key: u64) -> u32 = {70 mask: u32 = t*.capacity - 1;71 at: u32 = cast(hash(key), u32) & mask;72 printfmt(" slots looked at:");73 while (1) {74 printfmt(" {}", at);75 t*.probes += 1;76 s: u32 = t*.slots[at];77 if (s == 0 || t*.keys[s - 1] == key) {78 return at;79 }80 at = (at + 1) & mask;81 }82 return at;83}84 85// The key's id + 1, or 0 if it isn't there.86interner_find: proc(t: *interner, key: u64) -> u32 = {87 return t*.slots[interner_slot_of(t, key)];88}89 90interner_intern: proc(t: *interner, key: u64) -> u32 = {91 at: u32 = interner_slot_of(t, key);92 if (t*.slots[at] != 0) {93 printfmt("; found: id {}\n", t*.slots[at] - 1);94 return t*.slots[at] - 1;95 }96 if ((t*.length + 1) * 4 > t*.capacity * 3) {97 printfmt("; not there\n");98 printfmt(" {} ids would fill more than 3/4 of {} slots: the table grows to {}, and every id\n",99 t*.length + 1, t*.capacity, t*.capacity * 2);100 printfmt(" goes where its key's hash says now\n");101 t*.capacity *= 2;102 for (i: u32 = 0; i < 16; i += 1) {103 t*.slots[i] = 0;104 }105 mask: u32 = t*.capacity - 1;106 for (id: u32 = 0; id < t*.length; id += 1) {107 i: u32 = cast(hash(t*.keys[id]), u32) & mask;108 while (t*.slots[i] != 0) {109 i = (i + 1) & mask;110 }111 t*.slots[i] = id + 1;112 }113 at = interner_slot_of(t, key);114 }115 id: u32 = t*.length;116 t*.keys[t*.length] = key;117 t*.length += 1;118 t*.slots[at] = id + 1;119 printfmt("; new: id {}, in slot {}\n", id, at);120 return id;121}122 123draw: proc(t: *interner) -> void = {124 printfmt(" table ");125 first: b32 = 1;126 for (i: u32 = 0; i < t*.capacity; i += 1) {127 if (t*.slots[i] != 0) {128 if (!first) {129 printfmt(" ");130 }131 printfmt("{}:{}", i, name(t*.keys[t*.slots[i] - 1]));132 first = 0;133 }134 }135 printfmt(", {} of {} slots used\n", t*.length, t*.capacity);136}137 138// verify calls this when it finds something wrong, then says what.139broken: proc() -> void = {140 printfmt(" verify: ");141}142 143// The table's invariant: each id is in one slot, reachable from its key's144// hash without crossing an empty slot, and the table is at most 3/4 full.145verify: proc(t: *interner) -> void = {146 mask: u32 = t*.capacity - 1;147 used: u32 = 0;148 for (i: u32 = 0; i < t*.capacity; i += 1) {149 if (t*.slots[i] != 0) {150 used += 1;151 }152 }153 if (used != t*.length) {154 broken();155 printfmt("{} slots are used, but there are {} ids\n", used, t*.length);156 }157 if (t*.length * 4 > t*.capacity * 3) {158 broken();159 printfmt("{} ids in {} slots is more than 3/4 full\n", t*.length, t*.capacity);160 }161 for (id: u32 = 0; id < t*.length; id += 1) {162 key: u64 = t*.keys[id];163 start: u32 = cast(hash(key), u32) & mask;164 at: u32 = start;165 found: b32 = 0;166 for (step: u32 = 0; step < t*.capacity; step += 1) {167 if (t*.slots[at] == id + 1) {168 found = 1;169 break;170 }171 if (t*.slots[at] == 0) {172 break;173 }174 at = (at + 1) & mask;175 }176 if (!found) {177 broken();178 printfmt("id {}, {}, can't be found: its run from slot {} meets the empty slot {} first\n",179 id, name(key), start, at);180 }181 }182}183 184main: proc() -> i32 = {185 // Components, a bit each. The ones nearly everything has were made first,186 // so they have the lowest bits.187 position: u64 = 1'u64 shl 0;188 sprite: u64 = 1'u64 shl 1;189 health: u64 = 1'u64 shl 2;190 velocity: u64 = 1'u64 shl 3;191 player_tag: u64 = 1'u64 shl 4;192 ai: u64 = 1'u64 shl 5;193 damage: u64 = 1'u64 shl 6;194 loot: u64 = 1'u64 shl 7;195 player: u64 = position | sprite | health | velocity | player_tag;196 drone: u64 = position | sprite | health | velocity | ai;197 turret: u64 = position | sprite | health | ai | damage;198 rock: u64 = position | sprite | health;199 chest: u64 = position | sprite | health | loot;200 tank: u64 = position | sprite | health | velocity | ai | damage;201 bird: u64 = position | sprite | health | velocity;202 kind_keys[0] = player;203 kind_keys[1] = drone;204 kind_keys[2] = turret;205 kind_keys[3] = rock;206 kind_keys[4] = chest;207 kind_keys[5] = tank;208 kind_keys[6] = bird;209 210 t: interner = {};211 t.capacity = 8;212 printfmt("an interner with 8 slots; a slot holds an id + 1, or 0 when it's empty\n");213 214 printfmt("intern player, signature {}\n", player);215 _ = interner_intern(t&, player);216 draw(t&);217 verify(t&);218 219 printfmt("intern drone, signature {}\n", drone);220 _ = interner_intern(t&, drone);221 draw(t&);222 verify(t&);223 224 printfmt("intern turret, signature {}\n", turret);225 _ = interner_intern(t&, turret);226 draw(t&);227 verify(t&);228 229 printfmt("intern drone again\n");230 _ = interner_intern(t&, drone);231 verify(t&);232 233 printfmt("intern rock, signature {}\n", rock);234 _ = interner_intern(t&, rock);235 draw(t&);236 verify(t&);237 238 printfmt("intern chest, signature {}\n", chest);239 _ = interner_intern(t&, chest);240 draw(t&);241 verify(t&);242 243 printfmt("find tank, signature {}\n", tank);244 found: u32 = interner_find(t&, tank);245 if (found != 0) {246 printfmt("; found: id {}\n", found - 1);247 } else {248 printfmt("; not there\n");249 }250 verify(t&);251 252 printfmt("intern tank\n");253 _ = interner_intern(t&, tank);254 draw(t&);255 verify(t&);256 257 printfmt("intern bird, signature {}\n", bird);258 _ = interner_intern(t&, bird);259 draw(t&);260 verify(t&);261 262 printfmt("find turret\n");263 found = interner_find(t&, turret);264 if (found != 0) {265 printfmt("; found: id {}, and id {}'s key is {}\n", found - 1, found - 1, t.keys[found - 1]);266 } else {267 printfmt("; not there\n");268 }269 verify(t&);270 271 printfmt("slots looked at, in all: {}\n", t.probes);272 return 0;273}What play_interning prints. tools/check.py fails the playground if a single byte differs.
an interner with 8 slots; a slot holds an id + 1, or 0 when it's empty
intern player, signature 31
slots looked at: 5; new: id 0, in slot 5
table 5:player, 1 of 8 slots used
intern drone, signature 47
slots looked at: 4; new: id 1, in slot 4
table 4:drone 5:player, 2 of 8 slots used
intern turret, signature 103
slots looked at: 1; new: id 2, in slot 1
table 1:turret 4:drone 5:player, 3 of 8 slots used
intern drone again
slots looked at: 4; found: id 1
intern rock, signature 7
slots looked at: 4 5 6; new: id 3, in slot 6
table 1:turret 4:drone 5:player 6:rock, 4 of 8 slots used
intern chest, signature 135
slots looked at: 7; new: id 4, in slot 7
table 1:turret 4:drone 5:player 6:rock 7:chest, 5 of 8 slots used
find tank, signature 111
slots looked at: 7 0; not there
intern tank
slots looked at: 7 0; new: id 5, in slot 0
table 0:tank 1:turret 4:drone 5:player 6:rock 7:chest, 6 of 8 slots used
intern bird, signature 15
slots looked at: 1 2; not there
7 ids would fill more than 3/4 of 8 slots: the table grows to 16, and every id
goes where its key's hash says now
slots looked at: 9 10; new: id 6, in slot 10
table 4:rock 5:player 7:chest 8:tank 9:turret 10:bird 12:drone, 7 of 16 slots used
find turret
slots looked at: 9; found: id 2, and id 2's key is 103
slots looked at, in all: 17
The dojo · block 10
Memoized edges
dojo/10.edges/play_edges.rin
In an archetype ECS, an entity that gains a component moves to the table whose signature has one more bit. Finding that table by its signature is a search. But a game moves entities between the same few tables over and over: a drone catches fire, the fire goes out, it catches fire again. So each table remembers where each component leads, once a move has found it:
node 3 {position velocity health} --+burning--> node 4
node 4 {position velocity health burning} --burning--> node 3
The first move from 3 with burning searches, and writes the answer on both nodes: an add edge one way and a remove edge back. Every move after that, in either direction, follows an edge. Remembering the answer to a slow question is memoizing, and the nodes and their edges are the archetype graph that flecs is built around.
The invariant: every known edge leads to the node one component away, and that node has the edge back. Each move that changes a signature searches at most once; after that, both directions are edges.
In the course: 2.6 adds add_edges and remove_edges to each table, and moves entities along them.
Watch it: the playground
python tools/dojo.py play edges
A drone gains and loses components, eight nodes at most. Each move says where it went and how, and the graph is drawn twice:
catches fire: node 3 -> 4 {position velocity health burning}, by a lookup
the fire goes out: node 4 -> 3 {position velocity health}, by an edge
catches fire again: node 3 -> 4 {position velocity health burning}, by an edge
...
node 3 {position velocity health}: +burning 4, -velocity 5, -health 2
...
moves that looked up a node by signature: 7; that followed an edge: 4
Step through it: break on with and watch the two edges get written after find_or_make. verify runs after every move and prints any edge that leads to the wrong node or has no edge back. Then flip the switch at the top, memo, to 0 and run it again. The drone goes to the same nodes, but nothing is remembered: eleven moves, eleven lookups, and every node drawn with no edges known. With a few hundred tables and a game moving thousands of entities a frame, that's the difference.
Do reps: the kata
You write graph, made in the region level, with nodes a *vec<graph_node> in level. The move that adds a component:
graph_with: proc(g: *graph, node: u32, id: u32) -> u32 = {
signature: u64 = graph_signature(g, node);
if ((signature & graph_bit(id)) != 0) {
return node;
}
edge: u32 = vec<graph_node>at(g*.nodes, cast(node, u64))*.add[id];
if (edge != 0) {
g*.hits += 1;
return edge - 1;
}
to: u32 = graph_find_or_make(g, signature | graph_bit(id));
// graph_find_or_make can make a node, and the vec can move them all to
// grow, so both pointers are taken after it.
vec<graph_node>at(g*.nodes, cast(node, u64))*.add[id] = to + 1;
vec<graph_node>at(g*.nodes, cast(to, u64))*.remove[id] = node + 1;
return to;
}
and its mirror, graph_without, plus graph_make(r) (node 0 is the empty signature, and level 2's version leaves that node to you), graph_signature, graph_len, graph_lookups (moves that searched) and graph_hits (moves that followed an edge). The edges are stored as one more than the node, so a zeroed node knows none, 64 each way:
graph_node: struct = {
signature: u64;
// add[id] is one more than the node with component id added, once a move
// has looked it up, so 0 means not known yet; remove[id] is the same, with
// it removed.
add: [64]u32;
remove: [64]u32;
}
| level | you write | target |
|---|---|---|
| 1 | graph_with, in an otherwise finished file | 3:00 |
| 2 | every proc body; the fields and signatures are given | 10:00 |
| 3 | the whole file, from a blank page | 12:00 |
python tools/dojo.py start edges 2
python tools/dojo.py edges
The harness runs the playground's moves as a scenario, then 20,000 random moves from random nodes, with ids up to 63, against a model that knows each node's signature and which edges a graph that writes both directions knows by now. After every move it checks the node, its signature, and both counters: a known edge is a hit, any other move that changes the signature is one lookup, and a move that changes nothing (adding what the node has) is neither. The seeds give the same runs as rust-dod's dojo.
rin notes. The comment in graph_with is the one to remember: a pointer into nodes taken before graph_find_or_make can point at memory the vec has moved out of, so the edges are written through fresh pointers after it. Here the checker holds you to it: keep one across that call, and it says 'n' may be stale: 'g.nodes', which it was taken from, has been changed since; take it again. That's dojo 8's lesson, and 2.6 has the same comment over its slots.
Level 4, variations. The same harness, different insides:
- One hash map of edges, a
*hashmap<u64, u32> in levelfromhashmap<u64, u32>make(r), keyed bynode * 128 + id * 2 + adding, instead of two 64-wide arrays per node. Which takes less memory with 300 tables, and which is faster to follow? - A hash map for the slow path:
graph_find_or_makesearches every node. Keep ahashmap<u64, u32>from signature to node, as 2.4 does, or build one in dojo 9. - Removing a node. An ECS that deletes empty tables must forget every edge into them. Which nodes have edges into node
n? (Not checked by the harness; write your own test.)
When it fails
FAIL in the scenario: lookups and hits gave 5 1, it should give 4 2
Five lookups where the model counts four: the fire going out searched, instead of following the edge back that catching fire should have written. The edge back is the one people forget, or write on the wrong node: the move from 3 to 4 sets node 3's add edge and node 4's remove edge.
The scenario's ids stop at 3. The fuzz run's go up to 63, and a failure there says which operation and seed:
FAIL at operation 9 (seed 1): with(3, 63) gave node 6, which is {2 3 31}; from {2 3} it should be {2 3 63}
debug it: build/dojo/edges/harness --seed 1 --stop 9
with a breakpoint on dojo_stop_here, which runs just before that operation
That's a bit made in 32 bits, 1 shl id on a u32: bit 63 lands on bit 31. rin masks a shift's amount to the width of what it shifts, so nothing is undefined and --sanitize stays quiet. Run the harness under your debugger with those arguments and a breakpoint on dojo_stop_here, and step into the move.
The playground. The kata's harness is dojo/10.edges/harness.rin, and the levels start from dojo/10.edges/level1.rin to level3.rin.
1// Dojo 10, the playground: memoized edges.2//3// In an archetype ECS, an entity that gains a component moves to the table4// whose signature has one more bit. Finding that table by its signature is a5// search. But a game moves entities between the same few tables over and6// over, so each table remembers where each component leads: an add edge per7// component, and a remove edge back. The tables and their edges form a graph,8// the archetype graph of the course's 2.6.9//10// Here the nodes are bare signatures, eight at most, over four components. A11// drone gains and loses components, and each move prints the node it went to12// and whether it took a lookup or followed an edge. verify runs after every13// move: every known edge must lead to the node with that one component added14// (or removed) and have its edge back, and the drone's node must have the15// components the drone has. Put a breakpoint inside its failure branch to stop16// where the graph goes wrong.17//18// The switch: set memo to 0 and run it again. Nothing is remembered: the19// drone goes to the same nodes, but every move is a lookup. It's a global,20// read on every move, so a debugger can flip it mid-run too: stop on the21// first `lose` in main, set it to 0 in the watch window, and continue.22 23import "std/print.rin"24 25memo: b32 = 1;26 27names: [4]^const char = { "position", "velocity", "health", "burning" };28position: const u32 = 0;29velocity: const u32 = 1;30health: const u32 = 2;31burning: const u32 = 3;32 33bit: proc(id: u32) -> u64 = {34 return 1'u64 shl cast(id, u64);35}36 37node: struct = {38 signature: u64;39 // add[id] is one more than the node with component id added, once a move40 // has found it, so 0 means not known yet; remove[id] the same, with it41 // removed.42 add: [4]u32;43 remove: [4]u32;44}45 46graph: struct = {47 nodes: [8]node;48 count: u32;49 lookups: u32;50 edges: u32;51}52 53// The slow path: search every node for the signature, and make one if none54// has it.55find_or_make: proc(g: *graph, signature: u64) -> u32 = {56 g*.lookups += 1;57 for (i: u32 = 0; i < g*.count; i += 1) {58 if (g*.nodes[i].signature == signature) {59 return i;60 }61 }62 g*.nodes[g*.count] = { .signature = signature };63 g*.count += 1;64 return g*.count - 1;65}66 67with: proc(g: *graph, from: u32, id: u32) -> u32 = {68 if (memo && g*.nodes[from].add[id] != 0) {69 g*.edges += 1;70 return g*.nodes[from].add[id] - 1;71 }72 to: u32 = find_or_make(g, g*.nodes[from].signature | bit(id));73 if (memo) {74 g*.nodes[from].add[id] = to + 1;75 g*.nodes[to].remove[id] = from + 1;76 }77 return to;78}79 80without: proc(g: *graph, from: u32, id: u32) -> u32 = {81 if (memo && g*.nodes[from].remove[id] != 0) {82 g*.edges += 1;83 return g*.nodes[from].remove[id] - 1;84 }85 to: u32 = find_or_make(g, g*.nodes[from].signature & ~bit(id));86 if (memo) {87 g*.nodes[from].remove[id] = to + 1;88 g*.nodes[to].add[id] = from + 1;89 }90 return to;91}92 93print_names: proc(signature: u64) -> void = {94 printfmt("{");95 first: b32 = 1;96 for (id: u32 = 0; id < 4; id += 1) {97 if ((signature & bit(id)) != 0) {98 if (!first) {99 printfmt(" ");100 }101 printfmt("{}", names[id]);102 first = 0;103 }104 }105 printfmt("}");106}107 108drone: struct = {109 node: u32;110 // What the drone has, kept apart from the graph: verify checks the two111 // agree.112 has: u64;113}114 115// Every edge leads to the node one component away, and comes back; and the116// drone's node is the one with its components.117verify: proc(g: *graph, d: *drone) -> void = {118 for (n: u32 = 0; n < g*.count; n += 1) {119 here: *node = g*.nodes[n]&;120 for (id: u32 = 0; id < 4; id += 1) {121 if (here*.add[id] != 0) {122 there: *node = g*.nodes[here*.add[id] - 1]&;123 if (there*.signature != (here*.signature | bit(id)) || there*.remove[id] != n + 1) {124 printfmt("verify: node {}'s +{} edge leads to node {} ", n, names[id], here*.add[id] - 1);125 print_names(there*.signature);126 printfmt("\n");127 }128 }129 if (here*.remove[id] != 0) {130 there: *node = g*.nodes[here*.remove[id] - 1]&;131 if (there*.signature != (here*.signature & ~bit(id)) || there*.add[id] != n + 1) {132 printfmt("verify: node {}'s -{} edge leads to node {} ", n, names[id], here*.remove[id] - 1);133 print_names(there*.signature);134 printfmt("\n");135 }136 }137 }138 }139 if (g*.nodes[d*.node].signature != d*.has) {140 printfmt("verify: the drone has ");141 print_names(d*.has);142 printfmt(", but its node {} is ", d*.node);143 print_names(g*.nodes[d*.node].signature);144 printfmt("\n");145 }146}147 148report: proc(g: *graph, d: *drone, what: ^const char, from: u32, lookups: u32) -> void = {149 printfmt("{}: node {} -> {} ", what, from, d*.node);150 print_names(g*.nodes[d*.node].signature);151 if (g*.lookups > lookups) {152 printfmt(", by a lookup\n");153 } else {154 printfmt(", by an edge\n");155 }156 verify(g, d);157}158 159gain: proc(g: *graph, d: *drone, id: u32, what: ^const char) -> void = {160 from: u32 = d*.node;161 lookups: u32 = g*.lookups;162 d*.node = with(g, d*.node, id);163 d*.has = d*.has | bit(id);164 report(g, d, what, from, lookups);165}166 167lose: proc(g: *graph, d: *drone, id: u32, what: ^const char) -> void = {168 from: u32 = d*.node;169 lookups: u32 = g*.lookups;170 d*.node = without(g, d*.node, id);171 d*.has = d*.has & ~bit(id);172 report(g, d, what, from, lookups);173}174 175draw: proc(g: *graph) -> void = {176 for (n: u32 = 0; n < g*.count; n += 1) {177 here: *node = g*.nodes[n]&;178 printfmt(" node {} ", n);179 print_names(here*.signature);180 printfmt(":");181 known: u32 = 0;182 for (id: u32 = 0; id < 4; id += 1) {183 if (here*.add[id] != 0) {184 printfmt("{} +{} {}", known > 0 ? "," : "", names[id], here*.add[id] - 1);185 known += 1;186 }187 }188 for (id: u32 = 0; id < 4; id += 1) {189 if (here*.remove[id] != 0) {190 printfmt("{} -{} {}", known > 0 ? "," : "", names[id], here*.remove[id] - 1);191 known += 1;192 }193 }194 if (known == 0) {195 printfmt(" no edges known");196 }197 printfmt("\n");198 }199}200 201main: proc() -> i32 = {202 g: graph = {};203 g.count = 1;204 d: drone = {};205 printfmt("the drone starts at node 0 {}\n", "{}");206 gain(g&, d&, position, "gains position");207 gain(g&, d&, velocity, "gains velocity");208 gain(g&, d&, health, "gains health");209 gain(g&, d&, burning, "catches fire");210 lose(g&, d&, burning, "the fire goes out");211 gain(g&, d&, burning, "catches fire again");212 lose(g&, d&, burning, "the fire goes out");213 printfmt("the graph so far:\n");214 draw(g&);215 lose(g&, d&, velocity, "stops");216 gain(g&, d&, burning, "catches fire");217 gain(g&, d&, velocity, "moves again");218 lose(g&, d&, burning, "the fire goes out");219 printfmt("the graph:\n");220 draw(g&);221 printfmt("moves that looked up a node by signature: {}; that followed an edge: {}\n", g.lookups, g.edges);222 return 0;223}What play_edges prints. tools/check.py fails the playground if a single byte differs.
the drone starts at node 0 {}
gains position: node 0 -> 1 {position}, by a lookup
gains velocity: node 1 -> 2 {position velocity}, by a lookup
gains health: node 2 -> 3 {position velocity health}, by a lookup
catches fire: node 3 -> 4 {position velocity health burning}, by a lookup
the fire goes out: node 4 -> 3 {position velocity health}, by an edge
catches fire again: node 3 -> 4 {position velocity health burning}, by an edge
the fire goes out: node 4 -> 3 {position velocity health}, by an edge
the graph so far:
node 0 {}: +position 1
node 1 {position}: +velocity 2, -position 0
node 2 {position velocity}: +health 3, -velocity 1
node 3 {position velocity health}: +burning 4, -health 2
node 4 {position velocity health burning}: -burning 3
stops: node 3 -> 5 {position health}, by a lookup
catches fire: node 5 -> 6 {position health burning}, by a lookup
moves again: node 6 -> 4 {position velocity health burning}, by a lookup
the fire goes out: node 4 -> 3 {position velocity health}, by an edge
the graph:
node 0 {}: +position 1
node 1 {position}: +velocity 2, -position 0
node 2 {position velocity}: +health 3, -velocity 1
node 3 {position velocity health}: +burning 4, -velocity 5, -health 2
node 4 {position velocity health burning}: -velocity 6, -burning 3
node 5 {position health}: +velocity 3, +burning 6
node 6 {position health burning}: +velocity 4, -burning 5
moves that looked up a node by signature: 7; that followed an edge: 4
The dojo · block 11
A reflection walk
dojo/11.reflect/play_reflect.rin
A reflect record lists a struct's fields: each one's name, where its bytes start in the struct, how many there are, and its type, with the type's own record when it's a struct too. rin's compiler writes one for every struct: lamp<> is lamp's. With records, one piece of code can list, read, write and compare the fields of any type it's handed, by name: an inspector, an editor, a save file, a network diff. It never names the type. It walks the bytes by the record:
"at.y" in a lamp: lamp's field at, offset 8 -> vec2's field y, offset 4 -> bytes 12..16
Each offset counts from the start of the field's own struct, so a path adds them up as it goes down. And an offset is the record's to say, not something to work out from the sizes before it: a u8 followed by an f32 leaves 3 bytes of padding, and the f32 starts at 4.
The invariant: the walk reads and writes exactly the bytes the compiler gave each field. Read by path, every field is what the struct holds.
In the course: 2.7 prints any entity field by field from records. The Advanced lesson 5 builds an inspector, and an editor's write by name.
Watch it: the playground
python tools/dojo.py play reflect
It prints a lamp's record, walks the lamp by it, changes fields by path, and diffs the lamp with how it was:
a lamp's record: 20 bytes
on: offset 0, size 1, u8
power: offset 4, size 4, f32
at: offset 8, size 8, a struct
flicker: offset 16, size 4, f32, hidden
...
set at.x to 5.00: true
diff with the lamp before: power at.x
set flicker to 0.50: true
diff with the lamp before: power at.x
The walk reads the lamp through its bytes, with a few trusted procs at the top of the file for what safe code can't do, read four bytes as an f32, index a C string, see a struct as bytes. Step through it: break on find and watch offset add up. verify runs after every step and checks every f32 the walk reads against the field read directly, l.power and so on. Then flip the switch at the top, use_record_offsets, to 0 and run it again. The walk adds up the sizes before each field instead, as if nothing had padding: power is read from offset 1, three padding bytes and one byte of the real power, and writing at.x lands on power. The record printed above it still says 4.
Do reps: the kata
You write the walk, over the compiler's records, reflect and reflect_field from std/reflect.rin. Safe code cannot index a C string or read four bytes as an f32, so the kata itself stays checked and gets those from outside. The harness gives it four trusted helpers, each a few lines: walk_cstr_at(s, i), walk_cstr_append(dst, at, src), walk_read_f32(bytes, at) and walk_write_f32(bytes, at, value). And it asks the record for field i with std's reflect_field_at(type, i), which says when it is past the end, instead of indexing reflect_fields(type), and compares type names with reflect_cstr_equal.
First, every field a tool shows, by path, walking into the ones that are structs. The path so far is a slice of the leaf's own text:
walk_leaves: proc(type: *const reflect, prefix: []const c8, base: u64, out: *vec<walk_leaf>) -> void = {
for (i: u64 = 0; i < reflect_field_count(type); i += 1) {
f: option<*const reflect_field> = reflect_field_at(type, i);
if (!f) {
continue;
}
if (reflect_field_has_attr(f, "hidden")) {
continue;
}
leaf: walk_leaf = { .offset = base + f*.offset, .size = f*.size };
for (k: u64 = 0; k < prefix.length; k += 1) {
leaf.path.text[k] = prefix[k];
}
end: u64 = walk_cstr_append(leaf.path.text[0..48], prefix.length, f*.name);
if (reflect_is_struct(f*.info)) {
leaf.path.text[end] = '.';
walk_leaves(unwrap(f*.info), leaf.path.text[0..end + 1], leaf.offset, out);
} else {
_ = vec<walk_leaf>push(out, leaf);
}
}
}
and on it and on a walk down one path, walk_find, the procs the harness calls. The two that make a list make it in a region the caller names, r: region R:
walk_paths: proc(r: region R, type: *const reflect) -> *vec<walk_path> in R = {
walk_diff: proc(r: region R, type: *const reflect, a: []const u8, b: []const u8) -> *vec<walk_path> in R = {
walk_get_f32: proc(type: *const reflect, bytes: []const u8, path: ^const char) -> option<f32> = {
walk_set_f32: proc(type: *const reflect, bytes: []u8, path: ^const char, value: f32) -> b32 = {
A path is a walk_path, 48 characters at most, ending in a zero. A hidden field is left out of walk_paths and walk_diff, with everything inside it, but a path still reaches it: hiding is for tools that list, not for code that asks by name. walk_get_f32 and walk_set_f32 refuse a path that names a struct, a u8, or nothing.
| level | you write | target |
|---|---|---|
| 1 | walk_leaves, the walk, in an otherwise finished file | 4:00 |
| 2 | every proc body; the types and signatures are given | 15:00 |
| 3 | the whole file, from a blank page | 18:00 |
python tools/dojo.py start reflect 2
python tools/dojo.py reflect
The harness's test types have the traps in them: lamp has padding after on: u8, room has a lamp inside it (paths three deep, offsets added twice), and a hidden draft: vec2. It runs the playground's steps as a scenario, then 20,000 random reads, writes and diffs of a room by path, each checked against the same fields reached directly, r.light.at.x and so on. The paths include ones that name a struct, a u8, or nothing: light.atx.x, corner.x.y, light.. The seeds give the same runs as rust-dod's dojo.
rin notes. rin's compiler writes the records; Rust's kata gets them from a macro in its harness, the way 2.7 makes them, and has paths return Vec<String> where this one fills a *vec<walk_path> in R in the caller's region. The harness opens the region frame once per fuzz operation for it, so what a walk allocates is gone after it. walk_read_f32 and walk_write_f32 cut a field's bytes out with a slice, bytes[at..at + 4], so a wrong offset past the end stops the program instead of reading past the value, and copy them through a local f32 with memcpy, so a field at any offset can be read without casting to a pointer that may be misaligned.
Level 4, variations. The same harness, different insides:
- **
walk_findby recursion**: find the field the first name names, and recurse into its record with the rest of the path. Safe code can't move a^const charalong, so the rest of the path is an index, astartpassed to a helper. - Paths without a vec of leaves: one walk that does both jobs, with
out, andaandbfor a diff, passed down to each leaf:walk_pathskeeps every leaf, andwalk_diffthe ones whose bytes differ. A callback looks like the way to do it, and it costs the harness its build. The checker can't see through a proc pointer, so it says of what the harness handswalk_diff:this was taken from 'before', and 'walk_diff' can move it through a global: take it after the call. - A tiny text format: write a value as
path = valuelines and read it back. That's a save file and a network message. (Not checked by the harness; write your own test.)
When it fails
FAIL at operation 4 (seed 1): set_f32("light.", -50.25) returns true, it should return false
debug it: build/dojo/reflect/harness --seed 1 --stop 4
with a breakpoint on dojo_stop_here, which runs just before that operation
The name after the dot is empty, and it matched a field anyway: a name compare that stops at the end of the shorter of the two finds range for nothing, and at for atx. Run the harness under your debugger with those arguments and a breakpoint on dojo_stop_here, then step through walk_find, watching each name and offset. If it crashes instead, the runner runs it again with --trace 1 and prints the operation that was running.
The playground. The kata's harness is dojo/11.reflect/harness.rin, and the levels start from dojo/11.reflect/level1.rin to level3.rin.
1// Dojo 11, the playground: a reflection walk.2//3// A reflect record lists a struct's fields: each one's name, where its bytes4// start in the struct, how many there are, and its type, with the type's own5// record if it's a struct too. With records, one piece of code can list,6// read, write and compare the fields of any type it's handed, by name, the way7// an inspector, an editor, a save file or a network diff does. It never names8// the type: it walks the bytes by the record.9//10// rin's compiler writes a record for every struct: lamp<> is lamp's. The walk11// below reads the lamp through its bytes, and verify runs after every step:12// every f32 the walk reads by path must be what the lamp holds, read as13// l.power and so on. Put a breakpoint inside its failure branch to stop where14// they differ.15//16// The switch: set use_record_offsets to 0 and run it again. The walk adds up17// the sizes of the fields before each one instead, as if structs had no18// padding. That's right for `x` and `y`, and wrong from `power` on: it starts19// at 4, after 3 bytes of padding, not at 1. It's a global, read on every step,20// so a debugger can flip it mid-run too: stop on the first `set` in main, set21// it to 0 in the watch window, and continue.22//23// The few things safe code can't do, read four bytes as an f32,24// index a C string, see a struct as bytes, are the trusted procs at the top,25// each a line or two. The walk itself is safe: it asks the record for field i26// with reflect_field_at, which says when it is past the end.27 28import "std/print.rin"29import "std/reflect.rin"30 31use_record_offsets: b32 = 1;32 33vec2: struct = { x: f32; y: f32; }34lamp: struct = { on: u8; power: f32; at: vec2; flicker: f32 @ "hidden"; }35 36// ---------------------------------------------------------------------------37// Bytes and C strings, which safe code cannot touch by itself38// ---------------------------------------------------------------------------39 40read_f32: proc(row: []const u8, at: u64) -> f32 = {41 unsafe {42 field: []const u8 = row[at..at + 4];43 value: f32 = 0.0;44 memcpy(value&, field.data, 4);45 return value;46 }47}48 49write_f32: proc(row: []u8, at: u64, value: f32) -> void = {50 unsafe {51 field: []u8 = row[at..at + 4];52 memcpy(field.data, value&, 4);53 }54}55 56bytes_of: proc(l: *lamp) -> []u8 = {57 unsafe {58 return cast(l, ^u8)[0..lamp<>.size];59 }60}61 62// Character `i` of a C string: 0 at its end, and past it.63cstr_at: proc(s: ^const char, i: u64) -> c8 = {64 unsafe {65 for (k: u64 = 0; k < i; k += 1) {66 if (s[k] == 0) {67 return 0;68 }69 }70 return s[i];71 }72}73 74// Copy `src` into `dst` from `at`, with a zero after it; the index of the75// zero.76cstr_append: proc(dst: []c8, at: u64, src: ^const char) -> u64 = {77 unsafe {78 end: u64 = at;79 for (i: u64 = 0; src[i] != 0 && end + 1 < dst.length; i += 1) {80 dst[end] = src[i];81 end += 1;82 }83 dst[end] = 0;84 return end;85 }86}87 88// Zero-ended text as a C string, to print.89cstr: proc(text: []const c8) -> ^const char = {90 unsafe {91 // Only with its terminator: whoever reads this reads up to it.92 for (i: u64 = 0; i < text.length; i += 1) {93 if (text[i] == 0) {94 return cast(text[0]&, ^const char);95 }96 }97 printfmt("cstr: {} bytes and no terminator\n", text.length);98 exit(1);99 return "";100 }101}102 103// ---------------------------------------------------------------------------104// The walk105// ---------------------------------------------------------------------------106 107// Where field i starts in its struct: what the record says or, with the108// switch off, the sizes of the fields before it added up.109offset_in: proc(type: *const reflect, i: u64) -> u64 = {110 f: option<*const reflect_field> = reflect_field_at(type, i);111 if (!f) {112 return 0;113 }114 if (use_record_offsets) {115 return f*.offset;116 }117 sum: u64 = 0;118 for (k: u64 = 0; k < i; k += 1) {119 before: option<*const reflect_field> = reflect_field_at(type, k);120 if (before) {121 sum += before*.size;122 }123 }124 return sum;125}126 127// Every field a tool shows, depth first, printed with its value. Struct128// fields are walked into, and hidden ones left out. `prefix` is the path so129// far, "at." inside the lamp's at.130show: proc(type: *const reflect, row: []const u8, prefix: []const c8, base: u64) -> void = {131 for (i: u64 = 0; i < type*.count; i += 1) {132 f: option<*const reflect_field> = reflect_field_at(type, i);133 if (!f) {134 continue;135 }136 if (reflect_field_has_attr(f, "hidden")) {137 continue;138 }139 text: [48]c8 = {};140 for (k: u64 = 0; k < prefix.length; k += 1) {141 text[k] = prefix[k];142 }143 end: u64 = cstr_append(text[0..48], prefix.length, f*.name);144 at: u64 = base + offset_in(type, i);145 if (reflect_is_struct(f*.info)) {146 text[end] = '.';147 show(unwrap(f*.info), row, text[0..end + 1], at);148 } else if (reflect_cstr_equal(f*.type, "f32")) {149 printfmt(" {}: {:.2}\n", cstr(text[0..48]), cast(read_f32(row, at), f64));150 } else {151 printfmt(" {}: {}\n", cstr(text[0..48]), row[at]);152 }153 }154}155 156// The paths of the fields a tool shows whose bytes differ, each after a space.157diff: proc(type: *const reflect, a: []const u8, b: []const u8, prefix: []const c8, base: u64) -> void = {158 for (i: u64 = 0; i < type*.count; i += 1) {159 f: option<*const reflect_field> = reflect_field_at(type, i);160 if (!f) {161 continue;162 }163 if (reflect_field_has_attr(f, "hidden")) {164 continue;165 }166 text: [48]c8 = {};167 for (k: u64 = 0; k < prefix.length; k += 1) {168 text[k] = prefix[k];169 }170 end: u64 = cstr_append(text[0..48], prefix.length, f*.name);171 at: u64 = base + offset_in(type, i);172 if (reflect_is_struct(f*.info)) {173 text[end] = '.';174 diff(unwrap(f*.info), a, b, text[0..end + 1], at);175 } else {176 for (k: u64 = at; k < at + f*.size; k += 1) {177 if (a[k] != b[k]) {178 printfmt(" {}", cstr(text[0..48]));179 break;180 }181 }182 }183 }184}185 186// Is `name` exactly path[start..end]?187name_is: proc(name: ^const char, path: ^const char, start: u64, end: u64) -> b32 = {188 i: u64 = 0;189 while (start + i < end) {190 if (cstr_at(name, i) != cstr_at(path, start + i)) {191 return 0;192 }193 i += 1;194 }195 return cstr_at(name, i) == 0;196}197 198// The field a path names, and where its bytes start; null if none has it.199find: proc(type: *const reflect, path: ^const char, offset: *u64) -> option<*const reflect_field> = {200 found: option<*const reflect_field> = null;201 record: *const reflect = type;202 start: u64 = 0;203 more: b32 = 1;204 offset* = 0;205 while (more) {206 end: u64 = start;207 while (cstr_at(path, end) != 0 && cstr_at(path, end) != '.') {208 end += 1;209 }210 found = null;211 for (i: u64 = 0; i < reflect_field_count(record); i += 1) {212 f: option<*const reflect_field> = reflect_field_at(record, i);213 if (!found && f && name_is(f*.name, path, start, end)) {214 found = f;215 offset* += offset_in(record, i);216 }217 }218 if (!found) {219 return null;220 }221 more = cstr_at(path, end) != 0;222 // A path that goes on through a field with no record of its own names nothing.223 if (more) {224 next_record: option<*const reflect> = found*.info;225 if (!next_record) {226 return null;227 }228 record = next_record;229 }230 start = end + 1;231 }232 return found;233}234 235get_f32: proc(type: *const reflect, row: []const u8, path: ^const char, value: *f32) -> b32 = {236 at: u64 = 0;237 f: option<*const reflect_field> = find(type, path, at&);238 if (!f) {239 return 0;240 }241 if (!reflect_cstr_equal(f*.type, "f32")) {242 return 0;243 }244 value* = read_f32(row, at);245 return 1;246}247 248set_f32: proc(type: *const reflect, row: []u8, path: ^const char, value: f32) -> b32 = {249 at: u64 = 0;250 f: option<*const reflect_field> = find(type, path, at&);251 if (!f) {252 return 0;253 }254 if (!reflect_cstr_equal(f*.type, "f32")) {255 return 0;256 }257 write_f32(row, at, value);258 return 1;259}260 261// Every f32 of the lamp read by path, against the same field read directly.262// The first that differs is printed, and how many do.263verify: proc(l: *lamp) -> void = {264 paths: [4]^const char = { "power", "at.x", "at.y", "flicker" };265 held: [4]f32 = { l*.power, l*.at.x, l*.at.y, l*.flicker };266 wrong: u32 = 0;267 first: u64 = 0;268 for (i: u64 = 0; i < 4; i += 1) {269 walked: f32 = 0.0;270 if (!get_f32(lamp<>&, bytes_of(l), paths[i], walked&) || walked != held[i]) {271 if (wrong == 0) {272 first = i;273 }274 wrong += 1;275 }276 }277 if (wrong > 0) {278 walked: f32 = 0.0;279 printfmt("verify: the walk reads {} as ", paths[first]);280 if (get_f32(lamp<>&, bytes_of(l), paths[first], walked&)) {281 printfmt("{:.2}", cast(walked, f64));282 } else {283 printfmt("nothing");284 }285 printfmt(", the lamp holds {:.2}; {} of 4 f32s are wrong\n", cast(held[first], f64), wrong);286 }287}288 289set: proc(l: *lamp, path: ^const char, value: f32) -> void = {290 done: b32 = set_f32(lamp<>&, bytes_of(l), path, value);291 printfmt("set {} to {:.2}: {}\n", path, cast(value, f64), done ? "true" : "false");292 verify(l);293}294 295main: proc() -> i32 = {296 info: *const reflect = lamp<>&;297 printfmt("a lamp's record: {} bytes\n", info*.size);298 for (i: u64 = 0; i < info*.count; i += 1) {299 f: option<*const reflect_field> = reflect_field_at(info, i);300 if (!f) {301 continue;302 }303 printfmt(" {}: offset {}, size {}, ", f*.name, f*.offset, f*.size);304 if (reflect_is_struct(f*.info)) {305 printfmt("a struct");306 } else {307 printfmt("{}", f*.type);308 }309 if (reflect_field_has_attr(f, "hidden")) {310 printfmt(", hidden");311 }312 printfmt("\n");313 }314 l: lamp = {};315 l.on = 1;316 l.power = 60.0;317 l.at = { .x = 3.0, .y = 4.0 };318 l.flicker = 0.25;319 none: [1]c8 = {};320 printfmt("the lamp, walked by its record:\n");321 show(lamp<>&, bytes_of(l&), none[0..0], 0);322 verify(l&);323 before: lamp = l;324 set(l&, "power", 75.0);325 set(l&, "at.x", 5.0);326 printfmt("diff with the lamp before:");327 diff(lamp<>&, bytes_of(before&), bytes_of(l&), none[0..0], 0);328 printfmt("\n");329 set(l&, "flicker", 0.5);330 printfmt("diff with the lamp before:");331 diff(lamp<>&, bytes_of(before&), bytes_of(l&), none[0..0], 0);332 printfmt("\n");333 set(l&, "on", 0.0);334 set(l&, "at", 1.0);335 set(l&, "at.z", 1.0);336 printfmt("the lamp, walked by its record:\n");337 show(lamp<>&, bytes_of(l&), none[0..0], 0);338 return 0;339}What play_reflect prints. tools/check.py fails the playground if a single byte differs.
a lamp's record: 20 bytes
on: offset 0, size 1, u8
power: offset 4, size 4, f32
at: offset 8, size 8, a struct
flicker: offset 16, size 4, f32, hidden
the lamp, walked by its record:
on: 1
power: 60.00
at.x: 3.00
at.y: 4.00
set power to 75.00: true
set at.x to 5.00: true
diff with the lamp before: power at.x
set flicker to 0.50: true
diff with the lamp before: power at.x
set on to 0.00: false
set at to 1.00: false
set at.z to 1.00: false
the lamp, walked by its record:
on: 1
power: 75.00
at.x: 5.00
at.y: 4.00
The dojo · block 12
A watermark
dojo/12.watermark/play_watermark.rin
A query keeps the list of tables it matches, so a system doesn't search for them every frame. Tables are only ever added, at the end, and never removed. So the query remembers how many tables it has looked at, its watermark, and an update looks only at the tables past it:
tables 0 P 1 PV 2 PVH | 3 PVHB 4 H
^ movers has checked 3: an update looks at 3 and 4
A frame with no new tables costs one comparison, and each query looks at each table once, ever. It works for anything that only grows at the end: a log read by several readers, a list of events, the lines of a file being written.
The invariant: a query lists exactly the tables below its watermark that match, each once, in order, and it has looked at each of them once.
In the course: 3.1 builds the query and its checked count, and 3.2 updates every system's query at the start of every frame, which is what makes the watermark matter.
Watch it: the playground
python tools/dojo.py play watermark
It makes tables a few at a time, and updates two queries: movers, all {Position Velocity}, and healthy, all {Health}, none {Burning}. After each step it draws the tables, and under them what each query has looked at, with a | at its watermark:
update movers: it looks at tables 3 and 4 only
table 0 1 2 3 4
has P PV PVH PVHB H
movers - yes yes yes - | checked 5, scanned 5, tables 1 2 3
healthy | checked 0, scanned 0, tables none
scanned counts every table an update has looked at. With the watermark it always equals checked. Then verify checks both queries against the tables and prints what's wrong; break inside its if to stop when it fires.
Step through it: break on query_update and watch from, checked and found. Then flip the switch at the top, use_watermark, to 0, and run it again. Every update starts from table 0, and the second update of movers, with nothing new, finds tables 1 and 2 again:
movers - yes yes | checked 3, scanned 6, tables 1 2 1 2
verify: movers lists tables 1 2 1 2; it should list 1 2
verify: movers has looked at 6 tables to check 3; each should be looked at once
A system walking that list would run twice over the same entities, and the work grows with every frame. The switch is a global, read on every update, so a debugger can flip it mid-run too: stop on query_update, set it to 0 in the watch window, and continue.
Do reps: the kata
You write query:
query: struct = {
all: u64;
none: u64;
// The tables that match, by index, in the order they were found.
tables: *vec<u32> in level;
// The watermark: tables 0 to checked - 1 have been looked at.
checked: u64;
// How many tables query_update has looked at, ever. With the watermark,
// each table once.
scanned: u64;
}
and its update, which is given the world's tables as their signatures, a []const u64, since it only reads them:
query_update: proc(q: *query, signatures: []const u64) -> void = {
plus query_make(r, all, none), which makes the query and its list of tables in the region level, query_tables (the matching tables, in increasing order, a []u32 in level), query_checked and query_scanned. A table matches when its signature has every bit of all and no bit of none.
| level | you write | target |
|---|---|---|
| 1 | query_update, in an otherwise finished file | 2:00 |
| 2 | every proc body; the fields and signatures are given | 5:00 |
| 3 | the whole file, from a blank page | 7:00 |
python tools/dojo.py start watermark 2
python tools/dojo.py watermark
The harness runs the playground's steps as a scenario, then 20,000 random operations: tables made with random signatures, four queries with random masks updated at random, and now and then one replaced by a new query, which must start from nothing. The new one is made in the region, and the old one stays in level until the run ends, which is what an arena does. A model keeps, for each query, the tables that match among the ones it has been updated over. After every update the query must have checked the whole list, list exactly the model's tables, and have scanned each table once. Some none masks have two bits, so "none of these" and "not all of these" give different answers. The seeds give the same runs as rust-dod's dojo.
Level 4, variations. The same harness, different insides:
- **A while loop, and
scannedadded once**:scannedgrows by the number of new tables, which is known before the loop starts. - Push, don't poll: flecs does the reverse. When the world makes a table, it tells every query, and
query_updatehas nothing left to do. What does the world have to keep, and what happens to a query made after the table? (Not checked by the harness; write your own test.) - A table that can go away: the list keeps a tombstone where a removed table was, a signature no query matches. What must a query do with a table it already lists? (Not checked by the harness.)
When it fails
The scenario catches the common mistakes first:
FAIL in the scenario: make tables 3 and 4, update movers: tables gave 1 2 0, it should give 1 2 3
That's a count of the new tables pushed as if it were the table's index: a loop that counts from 0 over the tables past the watermark has to add checked back. A failure in the fuzz run says where:
FAIL at operation 27 (seed 1): after query_update, query 1 (all {0 5}, none {0 6}): it lists 1, the model 0; the first difference is at position 0: table 1, where the model has nothing
debug it: build/dojo/watermark/harness --seed 1 --stop 27
with a breakpoint on dojo_stop_here, which runs just before that operation
That one tested none the way it tested all, (signature & none) != none, which is "not every bit of none" rather than "no bit of it". It only differs when none has more than one bit. Run the harness under your debugger with those arguments and a breakpoint on dojo_stop_here, then step into query_update and look at the table it lists.
The playground. The kata's harness is dojo/12.watermark/harness.rin, and the levels start from dojo/12.watermark/level1.rin to level3.rin.
1// Dojo 12, the playground: a watermark.2//3// A query keeps the list of tables it matches, so a system doesn't search for4// them every frame. Tables are only ever added, at the end, and never removed.5// So the query remembers how many it has looked at, its watermark, and an6// update looks only at the tables past it:7//8// tables 0 P 1 PV 2 PVH | 3 PVHB 4 H9// ^ movers has checked 3: an update looks at 3 and 410//11// A frame with no new tables costs one comparison, and each query looks at12// each table once, ever. This is the course's query (3.1), with the world's13// tables reduced to their signatures.14//15// This program makes tables a few at a time and updates two queries. After16// each step it draws the tables, and under them what each query has looked17// at, with a | at its watermark. Then verify checks each query and says what18// is wrong, if anything; break inside its `if` to stop there. To see an19// update in a debugger, break on `query_update` and watch `from`, `checked`20// and `found`.21//22// The switch: set use_watermark to 0 and run it again. Every update looks at23// every table, from 0, so a query lists its tables again each time. It's a24// global, read on every update, so a debugger can flip it mid-run too: stop25// on `query_update`, set it to 0 in the watch window, and continue.26//27// A query is named by its place in `query_names`, since a struct28// holding a C string is not a value safe code may write to, and the two29// queries are shown one after the other instead of through an array of30// pointers.31 32import "std/print.rin"33 34use_watermark: b32 = 1;35 36// Four components, a bit each.37position: u64 = 1;38velocity: u64 = 2;39health: u64 = 4;40burning: u64 = 8;41 42// The two queries' names; a query holds its place here.43query_names: [2]^const char = { "movers", "healthy" };44 45// The world's tables, as far as a query cares: a signature each, added at the46// end and never removed.47tables: struct = {48 signatures: [8]u64;49 count: u64;50}51 52tables_make: proc(t: *tables, signature: u64) -> void = {53 t*.signatures[t*.count] = signature;54 t*.count += 1;55}56 57query: struct = {58 name: u32;59 all: u64;60 none: u64;61 // The matching tables, by index.62 found: [8]u32;63 found_count: u64;64 // The watermark: tables 0 to checked - 1 have been looked at.65 checked: u64;66 // How many times an update has looked at a table.67 scanned: u64;68}69 70matches: proc(signature: u64, all: u64, none: u64) -> b32 = {71 return (signature & all) == all && (signature & none) == 0;72}73 74query_make: proc(name: u32, all: u64, none: u64) -> query = {75 q: query = {};76 q.name = name;77 q.all = all;78 q.none = none;79 return q;80}81 82query_update: proc(q: *query, t: *tables) -> void = {83 from: u64 = 0;84 if (use_watermark) {85 from = q*.checked;86 }87 for (ti: u64 = from; ti < t*.count; ti += 1) {88 q*.scanned += 1;89 if (matches(t*.signatures[ti], q*.all, q*.none)) {90 q*.found[q*.found_count] = cast(ti, u32);91 q*.found_count += 1;92 }93 }94 q*.checked = t*.count;95}96 97query_found: proc(q: *query) -> []u32 = {98 return q*.found[0..q*.found_count];99}100 101print_list: proc(items: []u32) -> void = {102 if (items.length == 0) {103 printfmt("none");104 }105 for (i: u64 = 0; i < items.length; i += 1) {106 if (i > 0) {107 printfmt(" ");108 }109 printfmt("{}", items[i]);110 }111}112 113// The invariant: a query lists the tables below its watermark that match,114// each once, in order, and it has looked at each of those tables once.115verify: proc(t: *tables, q: *query) -> void = {116 want: [8]u32 = {};117 want_count: u64 = 0;118 for (ti: u64 = 0; ti < q*.checked; ti += 1) {119 if (matches(t*.signatures[ti], q*.all, q*.none)) {120 want[want_count] = cast(ti, u32);121 want_count += 1;122 }123 }124 same: b32 = want_count == q*.found_count;125 for (i: u64 = 0; same && i < want_count; i += 1) {126 same = want[i] == q*.found[i];127 }128 if (!same) {129 printfmt(" verify: {} lists tables ", query_names[q*.name]);130 print_list(query_found(q));131 printfmt("; it should list ");132 print_list(want[0..want_count]);133 printfmt("\n");134 }135 if (q*.scanned != q*.checked) {136 printfmt(" verify: {} has looked at {} tables to check {}; each should be looked at once\n",137 query_names[q*.name], q*.scanned, q*.checked);138 }139}140 141// The components in a signature, a letter each, padded to `width` columns.142print_letters: proc(signature: u64, width: u64) -> void = {143 names: [4]c8 = { 'P', 'V', 'H', 'B' };144 n: u64 = 0;145 for (i: u64 = 0; i < 4; i += 1) {146 if (((signature shr i) & 1) == 1) {147 printfmt("{}", names[i]);148 n += 1;149 }150 }151 for (k: u64 = n; k < width; k += 1) {152 printfmt(" ");153 }154}155 156has_found: proc(q: *query, table: u64) -> b32 = {157 for (i: u64 = 0; i < q*.found_count; i += 1) {158 if (cast(q*.found[i], u64) == table) {159 return 1;160 }161 }162 return 0;163}164 165// The tables, then a row per query: yes or - for each table it has looked at,166// and a | at its watermark.167// A query's row: yes or - for each table it has looked at, and a | at its168// watermark.169show_query: proc(t: *tables, q: *query) -> void = {170 printfmt(" {:-10}", query_names[q*.name]);171 for (ti: u64 = 0; ti < q*.checked; ti += 1) {172 printfmt("{}", has_found(q, ti) ? "yes " : "- ");173 }174 printfmt("| ");175 for (ti: u64 = q*.checked; ti < t*.count; ti += 1) {176 printfmt(" ");177 }178 printfmt("checked {}, scanned {}, tables ", q*.checked, q*.scanned);179 print_list(query_found(q));180 printfmt("\n");181}182 183// The tables, then a row per query.184show: proc(t: *tables, a: *query, b: *query) -> void = {185 printfmt(" table");186 for (ti: u64 = 0; ti < t*.count; ti += 1) {187 if (ti == 0) {188 printfmt(" {}", ti);189 } else {190 printfmt(" {}", ti);191 }192 }193 printfmt("\n has ");194 for (ti: u64 = 0; ti < t*.count; ti += 1) {195 print_letters(t*.signatures[ti], ti + 1 < t*.count ? 5 : 0);196 }197 printfmt("\n");198 show_query(t, a);199 show_query(t, b);200 verify(t, a);201 verify(t, b);202}203 204main: proc() -> i32 = {205 t: tables = {};206 movers: query = query_make(0, position | velocity, 0);207 healthy: query = query_make(1, health, burning);208 printfmt("components: P position, V velocity, H health, B burning\n");209 printfmt("movers: all {}, none {}; healthy: all {}, none {}\n", "{P V}", "{}", "{H}", "{B}");210 211 printfmt("make tables 0 to 2\n");212 tables_make(t&, position);213 tables_make(t&, position | velocity);214 tables_make(t&, position | velocity | health);215 show(t&, movers&, healthy&);216 217 printfmt("update movers: it looks at tables 0 to 2\n");218 query_update(movers&, t&);219 show(t&, movers&, healthy&);220 221 printfmt("update movers again: no new tables, so it looks at nothing\n");222 query_update(movers&, t&);223 show(t&, movers&, healthy&);224 225 printfmt("make tables 3 and 4\n");226 tables_make(t&, position | velocity | health | burning);227 tables_make(t&, health);228 show(t&, movers&, healthy&);229 230 printfmt("update movers: it looks at tables 3 and 4 only\n");231 query_update(movers&, t&);232 show(t&, movers&, healthy&);233 234 printfmt("update healthy, for the first time: it looks at all five\n");235 query_update(healthy&, t&);236 show(t&, movers&, healthy&);237 238 printfmt("make table 5\n");239 tables_make(t&, velocity | health);240 show(t&, movers&, healthy&);241 242 printfmt("update healthy: it looks at table 5 only\n");243 query_update(healthy&, t&);244 show(t&, movers&, healthy&);245 return 0;246}What play_watermark prints. tools/check.py fails the playground if a single byte differs.
components: P position, V velocity, H health, B burning
movers: all {P V}, none {}; healthy: all {H}, none {B}
make tables 0 to 2
table 0 1 2
has P PV PVH
movers | checked 0, scanned 0, tables none
healthy | checked 0, scanned 0, tables none
update movers: it looks at tables 0 to 2
table 0 1 2
has P PV PVH
movers - yes yes | checked 3, scanned 3, tables 1 2
healthy | checked 0, scanned 0, tables none
update movers again: no new tables, so it looks at nothing
table 0 1 2
has P PV PVH
movers - yes yes | checked 3, scanned 3, tables 1 2
healthy | checked 0, scanned 0, tables none
make tables 3 and 4
table 0 1 2 3 4
has P PV PVH PVHB H
movers - yes yes | checked 3, scanned 3, tables 1 2
healthy | checked 0, scanned 0, tables none
update movers: it looks at tables 3 and 4 only
table 0 1 2 3 4
has P PV PVH PVHB H
movers - yes yes yes - | checked 5, scanned 5, tables 1 2 3
healthy | checked 0, scanned 0, tables none
update healthy, for the first time: it looks at all five
table 0 1 2 3 4
has P PV PVH PVHB H
movers - yes yes yes - | checked 5, scanned 5, tables 1 2 3
healthy - - yes - yes | checked 5, scanned 5, tables 2 4
make table 5
table 0 1 2 3 4 5
has P PV PVH PVHB H VH
movers - yes yes yes - | checked 5, scanned 5, tables 1 2 3
healthy - - yes - yes | checked 5, scanned 5, tables 2 4
update healthy: it looks at table 5 only
table 0 1 2 3 4 5
has P PV PVH PVHB H VH
movers - yes yes yes - | checked 5, scanned 5, tables 1 2 3
healthy - - yes - yes yes | checked 6, scanned 6, tables 2 4 5
The dojo · block 13
A dispatch table
dojo/13.dispatch/play_dispatch.rin
A frame is a list of systems run in order, and the list is data: a name and a pointer to a proc for each, in an array the program changes as it runs. Adding, removing, reordering or switching off a system is an edit to the array, not to the code that calls them:
schedule 0 move 1 shoot 2 damage 3 die
a frame for each enabled system, in order: entry.run(world)
The order is a statement about data: a system that reads what another writes runs after it. damage writes health and die reads it, so damage goes before die, and a drone dies in the frame its health runs out. The same table drives an interpreter's opcodes, a menu's actions, a network protocol's message handlers: a number or a name picks a proc.
The invariant: a run calls every enabled system once, in the list's order, and nothing else. Edits keep the order of everything they don't touch.
In the course: 3.2 makes a system a name, a query and a proc pointer, and runs a schedule of them each frame, and 3.4's game has nine, each placed after what it reads.
Watch it: the playground
python tools/dojo.py play dispatch
A drone flies at a turret. Four systems, move, shoot, damage and die, run frame by frame, and between frames the program edits the schedule: shoot is switched off while the turret reloads, back on, and removed at the end. Each system is made in the region level, and the schedule's array holds pointers to them, for the reason the kata explains below. After each step it prints the schedule or the world:
frame 2
ran move shoot damage die
drone 4 from the turret, health 60, alive
switch shoot off: the turret is reloading
schedule move (shoot off) damage die
frame 3
ran move damage die
Then verify checks that the drone is dead exactly when its health has run out; break inside its if to stop when it fires. Step through it: break on schedule_run and step into each call, through the pointer, into the system.
Then flip the switch at the top, in_order, to 0, and run it again. damage is added at the end of the schedule instead of before die, so die reads the health from before this frame's damage:
frame 5
ran move shoot die damage
drone 0 from the turret, health -20, alive
verify: the drone has health -20 and is still alive
and it dies in frame 6, a frame late. A frame late is a drone that fires once more after it should be dead, or a death sound one frame after the explosion. The switch is a global, read when damage is added, so a debugger can flip it too: stop on the first line of main, set it to 0 in the watch window, and continue.
Do reps: the kata
You write schedule. A system is a plain pointer to a proc, which takes the harness's ctx. Here the safe subset changes the design: a C string and a proc pointer are both pointers without a region, so a struct holding them is one safe code cannot write, hold by value or make with new. So the name and the proc you are given go behind a pointer into the region level, made by the kata's one trusted proc, and never change:
system_proc: alias = *proc(c: *ctx) -> void;
// A system as it was given: its name and its proc. Made in the region by
// schedule_system_make, the kata's one trusted proc.
schedule_system: struct = {
name: ^const char;
run: system_proc;
}
schedule_system_make: proc(r: region level, name: ^const char, run: system_proc) -> *schedule_system in level = {
unsafe {
s: *schedule_system in level = cast(memops_arena_push_zero(r, sizeof(schedule_system), alignof(schedule_system)),
*schedule_system in level);
s*.name = name;
s*.run = run;
return s;
}
}
What does change, enabled and calls, sits beside that pointer in an entry, and the schedule is a vec of entries, in the order they run:
// A system in the schedule: which, whether it runs, and how many times
// schedule_run has called it.
schedule_entry: struct = {
system: *schedule_system in level;
enabled: b32;
calls: u32;
}
schedule: struct = {
// In the order they run.
systems: *vec<schedule_entry> in level;
}
A proc that makes a system takes the region, since a struct can't keep it:
schedule_insert_before: proc(s: *schedule, r: region level, before: ^const char, name: ^const char,
run: system_proc) -> b32 = {
schedule_run: proc(s: *schedule, c: *ctx) -> void = {
plus schedule_make(r), schedule_add(s, r, name, run) (at the end), schedule_remove(s, name), schedule_set_enabled(s, name, on), schedule_len and schedule_name(s, i) (every name, enabled or not, in order), and schedule_calls(s, name). schedule_insert_before, schedule_remove and schedule_set_enabled return 0, and change nothing, when there's no system by that name. Names are C strings, compared with dojo_cstr_equal, since strcmp is C's and safe code can't call it.
| level | you write | target |
|---|---|---|
| 1 | schedule_run, in an otherwise finished file | 2:00 |
| 2 | every proc body; the fields and signatures are given | 10:00 |
| 3 | the whole file, from a blank page | 15:00 |
python tools/dojo.py start dispatch 2
python tools/dojo.py dispatch
The harness has twelve systems, input to save, and each one writes its number in the ctx's log when it's called, so after a run the log says what ran, in what order, and through which proc. It runs a short scenario, then 20,000 random adds, inserts, removes, enables, disables and runs against a model that keeps the schedule as a plain list. After every operation the order and every count must match the model, and every run must log exactly the enabled systems, in order. The seeds give the same runs as rust-dod's dojo.
rin notes. Rust's kata has order(), which returns the names as a Vec; here schedule_len and schedule_name give them one at a time, so nothing is allocated to read them.
Level 4, variations. The same harness, different insides:
- A fixed array:
[16]schedule_entryand a count instead of a vec, as the playground has. An array starts as zeros, so the entry'ssystemhas to be anoption<*schedule_system in level>, tested before use. What happens on the seventeenth add? - Struct of arrays: the system pointers, the flags and the counts in three vecs, one place per system. The names and procs stay together behind their pointer, since a
vec<^const char>is refused:a raw pointer lasts only for a call.schedule_insert_beforeandschedule_removemust move all three the same way. - Reads and writes: each system declares bitmasks of what it reads and writes, and
schedule_insert_beforerefuses a place where a system would read something a later one writes. Then: which neighbours could run at the same time, on two threads? (Not checked by the harness; write your own test.)
A system with data of its own is one more pointer beside the proc pointer, handed back on every call: 3.4's kayan_system has a ctx: option<*game in level> for that, typed so the checker knows what it points at, which is what a closure is, written out.
When it fails
The scenario catches the common mistakes first:
FAIL in the scenario: run: what ran gave move, it should give move damage die
That run stopped at the first disabled system, collide: a break where a skip was meant. A failure in the fuzz run says where:
FAIL at operation 110 (seed 1): after schedule_remove(die), schedule_calls(damage) is 0; the model counts 2 since it was added
debug it: build/dojo/dispatch/harness --seed 1 --stop 110
with a breakpoint on dojo_stop_here, which runs just before that operation
That kata kept the counts in a vec of their own, the struct of arrays above, and schedule_remove popped the last count rather than the removed system's, so the counts after it shifted one place. Run the harness under your debugger with those arguments and a breakpoint on dojo_stop_here, then step into schedule_remove and watch both vecs.
If the harness crashes instead, on a proc pointer that was never set, say, the runner runs it again with --trace 1 and prints the operation that was running, or says the crash came in the scenario, after the last step printed.
The playground. The kata's harness is dojo/13.dispatch/harness.rin, and the levels start from dojo/13.dispatch/level1.rin to level3.rin.
1// Dojo 13, the playground: a dispatch table.2//3// A frame is a list of systems run in order, and the list is data: a name and4// a pointer to a proc for each, in an array the program changes as it runs.5// Adding, removing, reordering or switching off a system is an edit to the6// array, not to the code that calls them:7//8// schedule 0 move 1 shoot 2 damage 3 die9// a frame for each enabled system, in order: entry.run(world)10//11// The order is a statement about data: a system that reads what another12// writes runs after it. damage writes health and die reads it, so damage13// goes before die, and a drone dies in the frame its health runs out. This is14// the course's schedule (3.2).15//16// This program builds a schedule of four systems for a drone flying at a17// turret, runs it frame by frame, and edits it between frames: shoot is18// switched off while the turret reloads, and removed at the end. After each19// step it prints the schedule or the world, and verify checks that the drone20// is dead exactly when its health has run out; break inside its `if` to stop21// there. To see the dispatch in a debugger, break on `schedule_run` and step22// into each call.23//24// The switch: set in_order to 0 and run it again. damage is added at the end25// of the schedule, after die, instead of before it, and the drone dies a26// frame late. It's a global, read when damage is added, so a debugger can27// flip it too: stop on the first line of main, set it to 0 in the watch28// window, and continue.29//30// A name and a proc pointer are both pointers without a region,31// so safe code cannot keep them in a struct by value. Each system is made in32// the region `level` by system_make, trusted, and the schedule's array holds33// pointers to them, with the enabled flag beside each. name_is, also trusted,34// is strcmp, which safe code may not call itself.35 36import "std/print.rin"37import "std/memops.rin"38import "std/cstd.rin"39 40level: region;41 42in_order: b32 = 1;43 44// A drone flying at a turret.45world: struct = {46 distance: i32;47 health: i32;48 alive: b32;49 // Hits this frame, waiting for damage.50 hits: i32;51}52 53// The systems. Each reads and writes a little of the world.54move_system: proc(w: *world) -> void = {55 if (w*.alive) {56 w*.distance -= 2;57 if (w*.distance < 0) {58 w*.distance = 0;59 }60 }61}62 63shoot_system: proc(w: *world) -> void = {64 if (w*.alive && w*.distance <= 4) {65 w*.hits += 1;66 }67}68 69damage_system: proc(w: *world) -> void = {70 if (w*.alive) {71 w*.health -= 40 * w*.hits;72 }73 w*.hits = 0;74}75 76die_system: proc(w: *world) -> void = {77 if (w*.alive && w*.health <= 0) {78 w*.alive = 0;79 }80}81 82// A system is a plain pointer to a proc.83system_proc: alias = *proc(w: *world) -> void;84 85// A system: its name and its proc, kept in the region. (Not called `system`:86// C has a proc of that name.)87sys: struct = {88 name: ^const char;89 run: system_proc;90}91 92system_make: proc(r: region level, name: ^const char, run: system_proc) -> *sys in level = {93 unsafe {94 s: *sys in level = cast(memops_arena_push_zero(r, sizeof(sys), alignof(sys)), *sys in level);95 s*.name = name;96 s*.run = run;97 return s;98 }99}100 101// Are two names the same text? strcmp is C's, so safe code calls it through102// this.103name_is: proc(a: ^const char, b: ^const char) -> b32 = {104 unsafe {105 return strcmp(a, b) == 0;106 }107}108 109schedule_entry: struct = {110 system: option<*sys in level>;111 enabled: b32;112}113 114schedule: struct = {115 entries: [8]schedule_entry;116 count: u64;117}118 119// Where the system called `name` is, or count if it isn't there.120schedule_find: proc(s: *schedule, name: ^const char) -> u64 = {121 for (i: u64 = 0; i < s*.count; i += 1) {122 sys_entry: option<*sys in level> = s*.entries[i].system;123 if (sys_entry && name_is(sys_entry*.name, name)) {124 return i;125 }126 }127 return s*.count;128}129 130schedule_add: proc(s: *schedule, r: region level, name: ^const char, run: system_proc) -> void = {131 s*.entries[s*.count] = { .system = system_make(r, name, run), .enabled = 1 };132 s*.count += 1;133}134 135schedule_insert_before: proc(s: *schedule, r: region level, before: ^const char, name: ^const char,136 run: system_proc) -> b32 = {137 at: u64 = schedule_find(s, before);138 if (at == s*.count) {139 return 0;140 }141 for (i: u64 = s*.count; i > at; i -= 1) {142 s*.entries[i] = s*.entries[i - 1];143 }144 s*.entries[at] = { .system = system_make(r, name, run), .enabled = 1 };145 s*.count += 1;146 return 1;147}148 149schedule_remove: proc(s: *schedule, name: ^const char) -> b32 = {150 at: u64 = schedule_find(s, name);151 if (at == s*.count) {152 return 0;153 }154 for (i: u64 = at; i + 1 < s*.count; i += 1) {155 s*.entries[i] = s*.entries[i + 1];156 }157 s*.count -= 1;158 s*.entries[s*.count] = {};159 return 1;160}161 162schedule_set_enabled: proc(s: *schedule, name: ^const char, on: b32) -> b32 = {163 at: u64 = schedule_find(s, name);164 if (at == s*.count) {165 return 0;166 }167 s*.entries[at].enabled = on;168 return 1;169}170 171// One frame: every enabled system, once, in order.172schedule_run: proc(s: *schedule, w: *world) -> void = {173 printfmt(" ran ");174 for (i: u64 = 0; i < s*.count; i += 1) {175 sys_entry: option<*sys in level> = s*.entries[i].system;176 if (sys_entry && s*.entries[i].enabled) {177 sys_entry*.run(w);178 printfmt(" {}", sys_entry*.name);179 }180 }181 printfmt("\n");182}183 184schedule_show: proc(s: *schedule) -> void = {185 printfmt(" schedule ");186 for (i: u64 = 0; i < s*.count; i += 1) {187 sys_entry: option<*sys in level> = s*.entries[i].system;188 if (!sys_entry) {189 continue;190 }191 if (s*.entries[i].enabled) {192 printfmt(" {}", sys_entry*.name);193 } else {194 printfmt(" ({} off)", sys_entry*.name);195 }196 }197 printfmt("\n");198}199 200show: proc(w: *world) -> void = {201 printfmt(" drone {} from the turret, health {}, ", w*.distance, w*.health);202 if (w*.alive) {203 printfmt("alive\n");204 } else {205 printfmt("dead\n");206 }207}208 209// The invariant, after every step: the drone is dead exactly when its health210// has run out.211verify: proc(w: *world) -> void = {212 if ((w*.alive != 0) == (w*.health <= 0)) {213 if (w*.alive) {214 printfmt(" verify: the drone has health {} and is still alive\n", w*.health);215 } else {216 printfmt(" verify: the drone is dead with health {}\n", w*.health);217 }218 }219}220 221frame: proc(s: *schedule, w: *world, n: u32) -> void = {222 printfmt("frame {}\n", n);223 schedule_run(s, w);224 show(w);225 verify(w);226}227 228main: proc() -> i32 = {229 level: region = {230 s: schedule = {};231 w: world = { .distance = 8, .health = 100, .alive = 1 };232 printfmt("a drone 8 from a turret, with health 100: it flies 2 closer a frame,\n");233 printfmt("and the turret hits it for 40 a frame once it's within 4\n");234 show(w&);235 236 printfmt("add move, shoot and die\n");237 schedule_add(s&, level, "move", move_system);238 schedule_add(s&, level, "shoot", shoot_system);239 schedule_add(s&, level, "die", die_system);240 schedule_show(s&);241 verify(w&);242 243 if (in_order) {244 printfmt("insert damage before die: die reads the health that damage writes\n");245 _ = schedule_insert_before(s&, level, "die", "damage", damage_system);246 } else {247 printfmt("add damage at the end, after die\n");248 schedule_add(s&, level, "damage", damage_system);249 }250 schedule_show(s&);251 verify(w&);252 253 frame(s&, w&, 1);254 frame(s&, w&, 2);255 256 printfmt("switch shoot off: the turret is reloading\n");257 _ = schedule_set_enabled(s&, "shoot", 0);258 schedule_show(s&);259 verify(w&);260 261 frame(s&, w&, 3);262 263 printfmt("switch shoot back on\n");264 _ = schedule_set_enabled(s&, "shoot", 1);265 schedule_show(s&);266 verify(w&);267 268 frame(s&, w&, 4);269 frame(s&, w&, 5);270 frame(s&, w&, 6);271 272 printfmt("remove shoot: the turret is gone\n");273 _ = schedule_remove(s&, "shoot");274 schedule_show(s&);275 verify(w&);276 }277 return 0;278}What play_dispatch prints. tools/check.py fails the playground if a single byte differs.
a drone 8 from a turret, with health 100: it flies 2 closer a frame,
and the turret hits it for 40 a frame once it's within 4
drone 8 from the turret, health 100, alive
add move, shoot and die
schedule move shoot die
insert damage before die: die reads the health that damage writes
schedule move shoot damage die
frame 1
ran move shoot damage die
drone 6 from the turret, health 100, alive
frame 2
ran move shoot damage die
drone 4 from the turret, health 60, alive
switch shoot off: the turret is reloading
schedule move (shoot off) damage die
frame 3
ran move damage die
drone 2 from the turret, health 60, alive
switch shoot back on
schedule move shoot damage die
frame 4
ran move shoot damage die
drone 0 from the turret, health 20, alive
frame 5
ran move shoot damage die
drone 0 from the turret, health -20, dead
frame 6
ran move shoot damage die
drone 0 from the turret, health -20, dead
remove shoot: the turret is gone
schedule move damage die
Advanced · seven lessons
Layout and SIMD
docs/README.md
particle is 68 bytes. Movement reads the first 24.In the array of structs, every cache line movement pulls in is about two thirds tint, age, life, size and name. In the struct of arrays, movement fetches only the position and velocity columns: 6,291,456 bytes a step instead of 17,825,792.
Seven lessons for after Build an ECS. The course builds the ECS. These go underneath it, to the memory and the machine code:
- Lesson 1, aos vs soa. The same particles in three layouts: what a loop fetches per cache line, what
noaliaspromises the compiler, which loops it vectorizes, what the bounds checks cost, and how to check that in the assembly. - Lessons 2 to 6 build an archetype ECS again, compressed into five steps, and end with a small game. They came first and move fast. After the course, read them as a recap, or as a second take on the same design. Lesson 4 builds the world in
includes/kayan.rin, and lessons 5 and 6 import it unchanged. - Lesson 7, layout follows access. Four layouts under streams and under random lookups, timed, and how an ECS decides where each component lives.
- Lesson 0, from kayan, reviews the C kayan these lessons started from: what they kept from it, what they changed, and why.
| lesson | about | |
|---|---|---|
| 0 | from kayan | what was kept from the C kayan, what changed, and why |
| 1 | aos vs soa | the same particles in three layouts, what a loop fetches, and what SIMD needs |
| 2 | archetype table | one table: columns, rows, spawn, swap-remove, systems over slices |
| 3 | many archetypes | one table type for every archetype: ids, signatures, queries |
| 4 | entity handles | generational handles, slots, moving between archetypes |
| 5 | inspector | printing and editing any entity from reflection records |
| 6 | kayan | a small game, and deferring structural changes to a command buffer |
| 7 | access patterns | four layouts under streams and lookups, and how an ECS decides |
How to read them
Read each lesson's .rin file from the top. It opens with the problem it solves, and the code follows in the order it runs. Then run it and read the output against the code: every lesson prints what it does, step by step. The guides here give the background that doesn't fit in a file header, and end with exercises.
Every lesson is in rin's safe subset. The memory they share lives in a region, level, that main opens; the checker follows every pointer into it. Two things it cannot follow stay in code marked for it: unsafe { } blocks over raw bytes, and proc[unsafe] kernels that make a noalias promise, which are called from inside unsafe { }.
Timings only mean something in a Release build. Debug does not optimize, and every slice index is a function call. Release keeps the bounds checks: they run in every build, because the safety rules rest on them. Lesson 1's guide shows how to build one, what the checks cost, and what to look for in the disassembly.
Advanced · lesson 0 · review
From kayan
docs/0_from_kayan.md
kayan is a sketch of an archetype entity component system in C, about 200 lines in src/kayan.h, with haikal generating its containers. These lessons start from its idea and rebuild the rest. This page says what was kept, what was changed, and why, so the lessons can be read as a review of it as much as a rewrite.
What kayan does
Components are bits in an enum, positions = 1 << 0, velocities = 1 << 1 and so on. Each archetype is a C struct written by hand (A, B, C, D) that starts with an ArchetypeHeader and then holds one haikal Array_T per component. The header has a u64 signature and a Map_u64 from column name to the column's byte offset in the struct.
A query walks an array of void * pointing at the archetypes, reads each header, and tests (signature & want) == want. For a match it looks up "positions" and "velocities" in the offset map, adds the offsets to the struct's address, and runs the loop over the two columns.
Kept
- An archetype is a struct of arrays. Each component is a contiguous column and every entity in the table has every column. That is the whole point, and kayan had it right from the start.
- A signature is a u64 of component bits, and a query is the test
(signature & all) == all. Lesson 3 adds anonemask beside it. - Arenas for all of it. Nothing in these lessons is freed one at a time. In safe rin an arena is a region:
level: region;names it,mainopens it, and the whole of it ends at one closing brace.
Changed
- One table type for every archetype (lesson 3). kayan needs a new C struct, a new signature line and new offset-map lines for every archetype. Here an archetype is data: a signature, and the world makes its table the first time something needs it (lesson 4).
- Columns by component id, not by name (lesson 3). kayan hashes the strings
"positions"and"velocities"on every query. Here a table's columns are an array indexed by component id, so finding one is an index. Names are kept for tools, through reflection (lesson 5). - The signature comes from the columns (lesson 3). In kayan they are written separately and can disagree. In
Dthey do: it haspositionsandvelocitiescolumns, but its signature ishealth | damage | alive, so the movement query never visits it. - A row is added and removed in one place (lesson 2). kayan appends to each column at the call site, and keeps
entity_countapart from the columns. ForBandDit incrementsa's count instead of their own (kayan.hlines 142, 145, 168, 174), soAreports seven entities and the others report none. Nothing breaks only because the query loop happens to use the column's own length. - Entities have handles (lesson 4). kayan's entity id is the value of an archetype's
entity_countbefore it is bumped: meant per table, in practiceA's, as above. Nothing records where an entity lives, so there is no way to refer to one entity from elsewhere, to remove it, or to move it to another archetype. The lessons use an{index, generation}handle and a slot list that tracks each entity's table and row. - Entities move between archetypes (lesson 4). Adding or removing a component moves the entity to the matching table, copying the components both tables share.
- Structural changes wait for the end of the frame (lesson 6), through a command buffer in a frame region. kayan never changes structure while a query runs, but only because it never changes structure at all.
- Reflection replaces hand-written tables (lessons 3 to 5). A component's size and alignment come from
T<>, and so do the names a tool prints. The offset map goes away entirely. - The compiler checks the memory. kayan's query reads through
void *and adds byte offsets it trusts. Here thevoid *is gone: a pointer says which region it points into, a lookup that can miss returnsoption<*T>, and a slice is bounds checked. A type-erased column is still bytes, because that is what lets one table type hold every archetype, but it is a[]u8that knows its length. Reading it as a[]Tis one smallunsafe { }block that checks what it is given (lesson 3), and everything above it is checked.
Not carried over
- **
vec4sfor positions.** kayan pads 3D vectors to 16 bytes, which suits SIMD loads. The lessons usevec2andvec3and plain loops, and leave vectorizing to the C compiler.noaliason the kernels' parameters, in lessons 1, 2, 3, 6 and 7, lets it do that without run-time overlap checks. Lesson 1 adds a third layout, onef32array per coordinate, which is the one SIMD runs cleanly over, and its exercises come back tovec4. Lesson 7 weighs that layout against lookups, which it is bad at. - cglm and the haikal containers. rin's generics and std (
vec<T>,hashmap<K, V>, slices) cover what they did.
Small things in kayan.h
- The query's inner loop declares
i32 i, shadowing the outeri(line 201). - Each lookup on lines 199 and 200 hashes and compares a string, once per column per archetype per query. That is cheap next to a large loop, but it is work that an array index does for nothing.
Advanced · lesson 1
AoS vs SoA
src/1.aos_vs_soa/aos_vs_soa.rin
src/1.aos_vs_soa/aos_vs_soa.rin
The same 262,144 particles, stored three times: as an array of 68-byte particle structs (AoS), as seven arrays, one per field (SoA), and with position and velocity split one step further into six arrays of f32, one per coordinate (split). Sixty steps of movement run over each, and the program checks that all three end with the same positions.
Where the particles live
All three layouts are in one region. A region is an arena the compiler can see:
level: region;
Each array is a slice, and a slice kept in memory says which region holds its elements:
particles: struct = {
positions: []vec3 in level;
velocities: []vec3 in level;
...
}
main opens the region, and new_slice is an arena push that keeps the length:
level: region = {
aos: []particle in level = new_slice<particle>(level, particle_count);
soa: *particles in level = new<particles>(level, {
.positions = new_slice<vec3>(level, particle_count),
...
});
Everything lives exactly as long as that block. Nothing is freed one at a time, and the checker will not let a pointer into level outlive the closing brace. The movement procs take plain slices and know nothing about any of this.
What the loop fetches
Memory reaches the CPU in 64-byte cache lines. A loop that reads 24 bytes of each 68-byte particle still pulls every byte of every line those particles sit in, so about two thirds of what movement fetches in the AoS loop is tint, age, lifetime, size and name. In the SoA loop the two arrays it reads are the only thing it fetches, and the hardware prefetcher sees two simple streams. The split loop fetches the same 24 bytes per particle as SoA, as six streams.
The program prints that arithmetic:
bytes fetched per step: 17825792 (AoS: 68 bytes x 262144)
bytes fetched per step: 6291456 (SoA: 24 bytes x 262144)
bytes fetched per step: 6291456 (split: 6 x 4 bytes x 262144)
What the CPU does with each layout
SSE registers hold four floats, AVX registers eight. One instruction adds four (or eight) pairs of floats at once, but only if they arrive side by side. Here is what clang-cl /O2 makes of each loop for Windows x64, read from the object file:
| loop | particles per pass | instructions per pass | per particle | shuffles |
|---|---|---|---|---|
move_aos | 1 | 16 | 16 | 1 |
move_soa | 4 | 77 | about 19 | 30 |
move_split | 4 | 20 | 5 | 0 |
move_split, /arch:AVX2 | 8 | 14 | under 2 | 0 |
- AoS goes one particle at a time: each particle's fields are 68 bytes from the next particle's, so there is nothing to load four of.
- SoA is vectorized, four particles per pass, but each
vec3keeps x, y and z side by side. To get four x's into one register the compiler loads them one float at a time and stitches them together, then takes them apart again to store. About a tenth of the loop is arithmetic, and per particle it costs about what AoS does. - Split is what SIMD wants. Each line of
move_splitbecomes one load of four consecutive floats, one multiply, one add, one store:
movups (%r10,%rsi,4), %xmm4 four vx
movups (%r8,%rsi,4), %xmm5 four px
mulps %xmm1, %xmm4 vx * dt
addps %xmm5, %xmm4 px + vx * dt
movups %xmm4, (%r8,%rsi,4) four px back
A layout is right for SIMD when its vector loop is mostly full-width loads, math and stores. When it is mostly movss, unpcklps and shufps, the data is fighting the registers.
Split is right for this loop, which runs over every particle in order. It is wrong for a loop that reaches particles by index, where x, y and z in three arrays are three trips to memory instead of one. Lesson 7 measures both kinds of loop over four layouts.
The bounds checks
Every positions[i] is bounds checked, in every build, Release included. The safety rules rest on it, so it does not hang on NDEBUG. A check is a compare and a branch to a call that never returns, in the middle of a loop that wants to do four particles at once, so you would expect it to cost the vector loop.
On clang it does not. The table above is with the checks on, and the passes are the same with them off. The compiler sees that the loop runs i up to px.length, works out before the loop how many full passes fit inside every column's length, runs the vector loop for those with no checks in it, and leaves the rest to a one-at-a-time loop that does check. A column shorter than px stops the program there, with a message. What the checks add to move_split is around the loop: 257 instructions in the whole proc instead of 79, and 27 in the one-at-a-time loop instead of 17.
gcc does not do this for move_soa or move_split. With the checks on, -fopt-info-vec reports no vector loop over either, and split, the layout that needs one most, is its slowest. -DRIN_NO_BOUNDS_CHECKS removes the checks and brings the vector loop back, as the timings below show. A program built that way has given up rin's guarantee. Try 8 is the other way, which keeps it.
Timings
--time prints how long each layout took. Release builds on one core, best of five runs, on a Ryzen 9 9950X3D: 1 MB of L2 a core and 96 MB of L3. clang-cl 22 on Windows, gcc 16 under WSL on the same machine. Each run did ten times the work and is divided by ten, because clock() on Windows ticks in milliseconds. ms for sixty steps:
| build | AoS | SoA | split |
|---|---|---|---|
clang-cl /O2 | 12.2 | 10.8 | 6.7 |
clang-cl /O2 /arch:AVX2 | 9.8 | 7.2 | 5.5 |
gcc -O3 | 31.4 | 11.6 | 26.5 |
gcc -O2 | 12.5 | 11.2 | 27.5 |
gcc -O3 -DRIN_NO_BOUNDS_CHECKS | 55.4 | 9.0 | 4.8 |
gcc -O2 -DRIN_NO_BOUNDS_CHECKS | 9.9 | 8.5 | 4.4 |
The clang-cl rows do not change with -DRIN_NO_BOUNDS_CHECKS, beyond the noise, and clang -O3 is the same as -O2.
The same loops with 8,192 particles, so everything stays in L2, and 1,920 steps so the work is the same:
| build | AoS | SoA | split |
|---|---|---|---|
clang-cl /O2 | 11.8 | 9.9 | 4.5 |
clang-cl /O2 /arch:AVX2 | 9.2 | 5.7 | 3.9 |
gcc -O3 | 30.7 | 11.6 | 10.6 |
gcc -O3 -DRIN_NO_BOUNDS_CHECKS | 53.4 | 8.9 | 2.7 |
And with 2,097,152 particles and 7.5 steps' worth, so the work is the same again. AoS, at 143 MB, no longer fits in L3:
| build | AoS | SoA | split |
|---|---|---|---|
clang-cl /O2 | 27.9 | 10.8 | 7.9 |
clang-cl /O2 /arch:AVX2 | 28.0 | 8.2 | 6.0 |
- In cache, split is the only layout SSE helps much. SoA and AoS come out about even in L2 on clang-cl, as the instruction counts say they should. AVX2's eight-float registers help SoA too.
- Out of cache, the bytes show. At 262,144 particles all three sit in this machine's 96 MB of L3, and AoS fetching three times the bytes costs little. At 2,097,152, AoS has left L3 and takes more than twice as long, while SoA and split, at 50 MB, are where they were. How much the layout matters here depends on the size of the cache you run on.
- **gcc's AoS at
-O3is the odd one.** Two and a half to five times its-O2, and it is the loop vectorizer:-fno-tree-loop-vectorizebrings it back to about 12 ms. It vectorizes the loop with 8-byte vectors, and I have not explained why that is so slow on this machine.
The machine was busy; expect about 20 percent noise between runs, more for AoS, which came out up to twice as slow in unlucky runs. Debug numbers are not worth comparing: nothing is optimized there, and every check is a function call.
For a Release build, set build_type to "Release" in build.rin and run rin build again. Release optimizes and defines NDEBUG, and the checks stay. To take them out, add "RIN_NO_BOUNDS_CHECKS" to build_defines, or compile the C that tools/check.py leaves in build/check/aos_vs_soa/ with -O2 -DRIN_NO_BOUNDS_CHECKS.
The parameter list is the data
move_aos(ps: noalias []particle, ...) says it needs particles. move_soa(positions: noalias []vec3, velocities: noalias []vec3, ...) and move_split(px, py, pz, vx, vy, vz, ...) say exactly which data they read and write. In lessons 2 and 3 every system is written that way. In lesson 6 each system runs its own query, then works only on the columns it named.
Naming the columns is what lets a scheduler see that two systems touch different data, and it is what makes a system easy to test: hand it two small arrays.
noalias
As far as the C compiler can tell, two []vec3 slices could overlap. So before it works on several particles at once, it checks at run time that the two are apart, and keeps a one-at-a-time copy of the loop for when they are not, or it does not vectorize at all. noalias on a parameter promises that, for the call, nothing else reaches the memory behind it. The check and the second copy go away.
It is a promise the checker cannot keep, so it lives in code marked for it. A proc with a noalias parameter is proc[unsafe]:
move_soa: proc[unsafe](positions: noalias []vec3, velocities: noalias []vec3, dt: f32) -> void = {
and a call to it sits in unsafe { }:
unsafe { move_soa(soa*.positions, soa*.velocities, dt); }
Take the [unsafe] off and rin stops:
noalias is a promise to the C optimiser that the checker cannot keep: it belongs on a 'proc[unsafe]'
So the three movement kernels are outside what the checker vouches for. That is the trade, and it is yours to make per kernel: take off both noalias and [unsafe], and the same loop is checked like everything else, at the cost below.
The SoA loop without and with it, Release, ms:
| build | without noalias | with noalias |
|---|---|---|
clang-cl /O2 | 23.2 | 10.8 |
clang-cl /O2 -DRIN_NO_BOUNDS_CHECKS | 10.8 | 10.8 |
clang -O2, Linux | 9.9 | 11.0 |
gcc -O3 | 11.8 | 11.6 |
gcc -O3 -DRIN_NO_BOUNDS_CHECKS | 9.1 | 9.0 |
- clang-cl is where it shows. Without the promise it does not vectorize
move_soaat all, and with the checks on it reads both slices from memory on every pass: one particle, 47 instructions. Without the checks it is 26 instructions and costs what the vector loop does, since fourvec3s gain little from SSE anyway. - clang on Linux vectorized the loop either way. Without the promise it checks once per call, so removing the check saves nothing that shows here.
- gcc puts the loop behind a run-time check only when the bounds checks are gone and it vectorizes at all: "loop versioned for vectorization because of possible aliasing". The promise removes the check, and it does not show here either.
On Windows, one slice needs it too. move_aos has a single slice, which cannot overlap another, and it still takes noalias. Windows passes a slice to a proc by reference, as a pointer to a copy, and clang-cl does not use C's type rules to decide what can overlap (it matches MSVC). So, without the promise, a float written through ps might be the slice's own pointer or length, and the loop reads both again after every particle:
movq (%rcx), %r8 ps.data, every particle
...
cmpq 0x8(%rcx), %rdx ps.length, every particle
With noalias the compiler gets the pointer and the length once, in registers, and the loop is 16 instructions instead of 18.
**And gravity is a const.** rin computes a constant and hands C the number, so the loops say -(9.8f) and there is no variable for a write to change:
gravity: static const f32 = -9.8;
A plain global is different. gravity: f32 = -9.8; is something other files could reach, and to gcc that is something a write through a slice might change, even through a noalias one, so gcc -O3 without the bounds checks puts a run-time check in front of move_split's vector loop. static f32 keeps it to this file, nothing takes its address, and the check goes away. Passing it as a parameter would do the same, which is what dt does.
The promise is checked at run time, in every build. Pass soa*.positions as both arguments to move_soa and the program stops before the loop runs:
aos_vs_soa.rin:153: noalias parameters 'positions' and 'velocities' overlap
That is one check per call, not per particle. Only -DRIN_NO_BOUNDS_CHECKS turns it off. That build runs on, and the lesson prints same result: no.
Fused multiply-add
With -mfma, -march=native or clang-cl's /arch:AVX2, clang fuses p + v * dt into one instruction that rounds once instead of twice. All three layouts are fused the same way and still agree, but the checksum's last digits move, from 38572460.505 to 38572460.519, and the byte-for-byte check against aos_vs_soa.expected fails. tools/check.py builds without those flags.
When AoS is right
When the code that touches a record touches most of its fields, AoS fetches nothing wasted, and keeping a record together is simpler. Data-oriented design does not mean SoA everywhere. It means choosing the layout from the loops that run over it. An entity component system is the case where many small loops each touch a few fields.
Try
- Run with
--timein Release. Then removetint,age,lifetimeandnamefromparticleand run again. The AoS loop should close most of the gap, because the struct shrinks toward what the loop reads. - Add
/arch:AVX2tobuild_c_flagsand compare--time. Then readmove_splitin the disassembly: eight floats per register, andvfmadd. - kayan stored positions as 16-byte
vec4s. Changevec3to four floats and see what it does to the bytes-per-step line, tomove_soa's disassembly and to the timing. - Store the split columns in blocks of eight,
{ x: [8]f32; y: [8]f32; ... }, and write the loop over blocks. Each block is a whole AVX register per coordinate, and a particle's coordinates stay near each other. - Remove
noaliasfrommove_soaand build Release with gcc-O3 -DRIN_NO_BOUNDS_CHECKS. Compare--time, then compile with-fopt-info-vecand find "loop versioned for vectorization because of possible aliasing". Then take off[unsafe]and theunsafe { }around its call, and read what clang-cl makes of a kernel the checker can vouch for. - Change
gravityto a plaingravity: f32 = -9.8;, build with gcc-O3 -DRIN_NO_BOUNDS_CHECKS -fopt-info-vec, and find the same message formove_split. Then make itstatic f32. - Pass
soa*.positionsas both arguments tomove_soa. Run it in Debug, in Release, then with-DRIN_NO_BOUNDS_CHECKS. - Get gcc's vector loop back with the checks on. At the top of
move_split, cut every column to one length,ys: []f32 = py[0..px.length];and so on, and loop over those. gcc can now see that one bound covers them all, and it vectorizes, behind an overlap check, since the cuts carry nonoalias. Split goes from 26.5 ms to about 8 on gcc-O3, a column that is too short stops the program at the cut, before the first particle, and nothing has been given up.
1// 1. AoS vs SoA2//3// The first decision in data-oriented code is how a collection sits in memory,4// and it is made before any system is written.5//6// An array of structs (AoS) keeps each particle's fields together: position,7// velocity, tint, age, lifetime, size and a name, one particle after another.8// A struct of arrays (SoA) keeps each field together instead: every position,9// then every velocity, and so on, each in an array of its own.10//11// Both hold exactly the same data. What differs is what a loop drags through12// the cache. Movement reads position and velocity, 24 bytes of each particle.13// The CPU fetches memory 64 bytes at a time, so in AoS most of every line it14// fetches for movement is tint, age and name, which the loop never looks at. In15// SoA the two arrays movement reads are all it fetches.16//17// SoA still stores each position as a vec3, x y z side by side. A third layout18// splits those too: one array of f32 per coordinate. Each array is then a19// straight run of floats, which is what a SIMD register loads: four floats at20// once with SSE, eight with AVX, nothing to rearrange first.21//22// The movement procs below also differ in what they ask for. move_aos takes the23// whole particle. move_soa and move_split take the arrays they use and nothing24// else, so their parameter lists are lists of their data dependencies. That25// habit is what the rest of these lessons build on.26//27// The program runs the same movement over all three layouts and checks that28// they agree. With `--time` it also times them. Build Release before believing29// the numbers: Debug checks every slice index.30//31// The arena is a region, `level`, that main opens; every array is32// a slice that names it, `[]vec3 in level`, made by `new_slice`. The kernels33// take plain slices and are `proc[unsafe]`, because `noalias` is a promise the34// checker cannot check: each call says `unsafe`. One other proc has an unsafe35// block, `has_flag`, which reads argv, a C array of C strings, which safe code36// cannot index.37 38cinclude "time.h"39import "std/print.rin"40 41clock: proc[external]() -> long = {}42CLOCKS_PER_SEC: const long;43 44level: region;45 46// argv is C's: `^^c8` with no lengths. Trusted, because strcmp reads to the47// terminator and the checker cannot see that argv's strings have one.48has_flag: proc(argc: i32, argv: ^^c8, flag: ^const c8) -> b32 = {49 unsafe {50 for (i: i32 = 1; i < argc; i += 1) {51 if (strcmp(argv[i], flag) == 0) {52 return 1;53 }54 }55 return 0;56 }57}58 59particle_count: u64 = 262144;60steps: u64 = 60;61dt: f32 = 1.0 / 60.0;62// const: rin writes the value into the loop, so no write through a slice can63// change it, and gcc needs no run-time check in front of move_split's vector loop.64gravity: static const f32 = -9.8;65 66vec3: struct = {67 x: f32;68 y: f32;69 z: f32;70}71 72color: struct = {73 r: f32;74 g: f32;75 b: f32;76 a: f32;77}78 79particle_name: struct = {80 text: [16]c8;81}82 83// One particle, the way a particle system tends to grow.84particle: struct = {85 position: vec3;86 velocity: vec3;87 tint: color;88 age: f32;89 lifetime: f32;90 size: f32;91 name: particle_name;92}93 94// The same particles, one array per field. Every array has the same length, and95// index i in each of them is particle i. Each slice says which region holds96// its elements: a slice kept in memory always does.97particles: struct = {98 positions: []vec3 in level;99 velocities: []vec3 in level;100 tints: []color in level;101 ages: []f32 in level;102 lifetimes: []f32 in level;103 sizes: []f32 in level;104 names: []particle_name in level;105}106 107// Position and velocity split one step further, one array per coordinate.108split_columns: struct = {109 px: []f32 in level;110 py: []f32 in level;111 pz: []f32 in level;112 vx: []f32 in level;113 vy: []f32 in level;114 vz: []f32 in level;115}116 117// A small LCG, so both layouts start from the same particles on every compiler.118random_state: u64 = 12345;119 120random01: proc() -> f32 = {121 random_state = random_state * 6364136223846793005'u64 + 1442695040888963407'u64;122 return cast(random_state shr 40, f32) / 16777216.0;123}124 125make_particle: proc(i: u64) -> particle = {126 p: particle = {};127 p.position = { .x = random01() * 100.0, .y = random01() * 100.0, .z = random01() * 100.0 };128 p.velocity = { .x = random01() - 0.5, .y = random01() * 4.0, .z = random01() - 0.5 };129 p.tint = { .r = random01(), .g = random01(), .b = random01(), .a = 1.0 };130 p.lifetime = 2.0 + random01() * 3.0;131 p.size = 0.1 + random01();132 p.name.text[0] = 'p';133 p.name.text[1] = cast('0' + i % 10, c8);134 return p;135}136 137// noalias even on a single slice: on Windows a slice is passed by reference,138// and clang-cl does not use C's type rules, so without it a float written139// through ps might be the slice's own pointer or length. The loop would read140// both again after every particle.141move_aos: proc[unsafe](ps: noalias []particle, dt: f32) -> void = {142 for (i: u64 = 0; i < ps.length; i += 1) {143 p: *particle = ps[i]&;144 p*.velocity.y += gravity * dt;145 p*.position.x += p*.velocity.x * dt;146 p*.position.y += p*.velocity.y * dt;147 p*.position.z += p*.velocity.z * dt;148 }149}150 151// noalias: positions and velocities never overlap, so the compiler can work on152// several particles at once without first checking that they don't.153move_soa: proc[unsafe](positions: noalias []vec3, velocities: noalias []vec3, dt: f32) -> void = {154 for (i: u64 = 0; i < positions.length; i += 1) {155 velocities[i].y += gravity * dt;156 positions[i].x += velocities[i].x * dt;157 positions[i].y += velocities[i].y * dt;158 positions[i].z += velocities[i].z * dt;159 }160}161 162// Every column it writes is noalias; vx and vz are only read. Each line of the163// loop is one SIMD load, multiply, add and store over consecutive floats.164move_split: proc[unsafe](px: noalias []f32, py: noalias []f32, pz: noalias []f32,165 vx: []const f32, vy: noalias []f32, vz: []const f32, dt: f32) -> void = {166 for (i: u64 = 0; i < px.length; i += 1) {167 vy[i] += gravity * dt;168 px[i] += vx[i] * dt;169 py[i] += vy[i] * dt;170 pz[i] += vz[i] * dt;171 }172}173 174checksum_aos: proc(ps: []particle) -> f64 = {175 sum: f64 = 0.0;176 for (i: u64 = 0; i < ps.length; i += 1) {177 sum += cast(ps[i].position.x + ps[i].position.y + ps[i].position.z, f64);178 }179 return sum;180}181 182checksum_soa: proc(positions: []vec3) -> f64 = {183 sum: f64 = 0.0;184 for (i: u64 = 0; i < positions.length; i += 1) {185 sum += cast(positions[i].x + positions[i].y + positions[i].z, f64);186 }187 return sum;188}189 190checksum_split: proc(xs: []const f32, ys: []const f32, zs: []const f32) -> f64 = {191 sum: f64 = 0.0;192 for (i: u64 = 0; i < xs.length; i += 1) {193 sum += cast(xs[i] + ys[i] + zs[i], f64);194 }195 return sum;196}197 198seconds: proc() -> f64 = {199 return cast(clock(), f64) / cast(CLOCKS_PER_SEC, f64);200}201 202main: proc(argc: i32, argv: ^^c8) -> i32 = {203 timing: b32 = has_flag(argc, argv, "--time");204 same: b32 = 0;205 206 // Everything below lives exactly as long as this block: the region is the207 // arena, and new_slice is push_slice with the length kept.208 level: region = {209 aos: []particle in level = new_slice<particle>(level, particle_count);210 soa: *particles in level = new<particles>(level, {211 .positions = new_slice<vec3>(level, particle_count),212 .velocities = new_slice<vec3>(level, particle_count),213 .tints = new_slice<color>(level, particle_count),214 .ages = new_slice<f32>(level, particle_count),215 .lifetimes = new_slice<f32>(level, particle_count),216 .sizes = new_slice<f32>(level, particle_count),217 .names = new_slice<particle_name>(level, particle_count),218 });219 split: *split_columns in level = new<split_columns>(level, {220 .px = new_slice<f32>(level, particle_count),221 .py = new_slice<f32>(level, particle_count),222 .pz = new_slice<f32>(level, particle_count),223 .vx = new_slice<f32>(level, particle_count),224 .vy = new_slice<f32>(level, particle_count),225 .vz = new_slice<f32>(level, particle_count),226 });227 228 // The same particle goes into every layout: whole into aos, field by field229 // into soa, coordinate by coordinate into split.230 for (i: u64 = 0; i < particle_count; i += 1) {231 p: particle = make_particle(i);232 aos[i] = p;233 soa*.positions[i] = p.position;234 soa*.velocities[i] = p.velocity;235 soa*.tints[i] = p.tint;236 soa*.ages[i] = p.age;237 soa*.lifetimes[i] = p.lifetime;238 soa*.sizes[i] = p.size;239 soa*.names[i] = p.name;240 split*.px[i] = p.position.x;241 split*.py[i] = p.position.y;242 split*.pz[i] = p.position.z;243 split*.vx[i] = p.velocity.x;244 split*.vy[i] = p.velocity.y;245 split*.vz[i] = p.velocity.z;246 }247 248 t0: f64 = seconds();249 for (s: u64 = 0; s < steps; s += 1) {250 unsafe { move_aos(aos, dt); }251 }252 t1: f64 = seconds();253 for (s: u64 = 0; s < steps; s += 1) {254 unsafe { move_soa(soa*.positions, soa*.velocities, dt); }255 }256 t2: f64 = seconds();257 for (s: u64 = 0; s < steps; s += 1) {258 unsafe { move_split(split*.px, split*.py, split*.pz, split*.vx, split*.vy, split*.vz, dt); }259 }260 t3: f64 = seconds();261 262 used: u64 = sizeof(vec3) * 2;263 printfmt("{} particles, {} steps\n", particle_count, steps);264 printfmt("array of structs: a particle is {} bytes, movement uses {} of them\n", sizeof(particle), used);265 printfmt(" bytes fetched per step: {}\n", sizeof(particle) * particle_count);266 printfmt("struct of arrays: movement reads two arrays of {}-byte vec3\n", sizeof(vec3));267 printfmt(" bytes fetched per step: {}\n", used * particle_count);268 printfmt("split columns: movement reads six arrays of {}-byte f32\n", sizeof(f32));269 printfmt(" bytes fetched per step: {}\n", used * particle_count);270 271 a: f64 = checksum_aos(aos);272 b: f64 = checksum_soa(soa*.positions);273 c: f64 = checksum_split(split*.px, split*.py, split*.pz);274 printfmt("checksum aos {:.3}\n", a);275 printfmt("checksum soa {:.3}\n", b);276 printfmt("checksum split {:.3}\n", c);277 same = a == b && b == c;278 printfmt("same result: {}\n", same ? "yes" : "no");279 280 if (timing) {281 printfmt("aos {:.1} ms, soa {:.1} ms, split {:.1} ms\n",282 (t1 - t0) * 1000.0, (t2 - t1) * 1000.0, (t3 - t2) * 1000.0);283 }284 }285 return same ? 0 : 1;286}What aos_vs_soa prints. tools/check.py fails the lesson if a single byte differs.
262144 particles, 60 steps
array of structs: a particle is 68 bytes, movement uses 24 of them
bytes fetched per step: 17825792
struct of arrays: movement reads two arrays of 12-byte vec3
bytes fetched per step: 6291456
split columns: movement reads six arrays of 4-byte f32
bytes fetched per step: 6291456
checksum aos 38572460.505
checksum soa 38572460.505
checksum split 38572460.505
same result: yes
Advanced · lesson 2
One archetype table
src/2.archetype_table/archetype_table.rin
src/2.archetype_table/archetype_table.rin
In the dojo: 2. swap-remove and the back-reference, 4. a type-erased column, each on its own, to watch and to practise.
Eight sparks in one table with three component columns, positions, velocities and lives, plus a numbers column that exists only so the output can say which spark is which. Each frame they move, they age, and the ones that ran out are removed.
Two rules
Columns never disagree. spark_spawn and spark_despawn are the only places rows come and go, and each touches every column. A new component means a new column, one line in each of those two procs, and one more argument to spark_spawn. A call site cannot add a row to some columns and forget the others.
Rows are packed. Systems loop over 0..count and never test whether a row is alive. Removing a row moves the last row into the hole:
spark 0 in row 0 expired; spark 7 moved from row 7 into it
That one copy per column is the whole cost of a despawn, whatever the table's size. The price is that a row number is not a name for a spark. After that line, "row 0" means spark 7. Lesson 4 adds names that do not move.
The table lives in a region
The table and its columns come from the region level, and their types say so:
spark_table_create: proc(r: region level, capacity: u64) -> *spark_table in level = {
return new<spark_table>(r, {
.capacity = capacity,
.numbers = new_slice<u32>(r, capacity),
.positions = new_slice<vec2>(r, capacity),
.velocities = new_slice<vec2>(r, capacity),
.lives = new_slice<f32>(r, capacity),
});
}
main opens level and holds the result as *spark_table in level, so the checker will not let it be kept past the region's end. Every column here is typed, so no byte is ever reinterpreted, and the only unsafe in the file is for noalias, below.
Removing while iterating
expire_system walks the rows backwards. When it removes row i, the row that moves in comes from the end, and the end has already been tested. A forward loop would have to stay on i after a removal, and it is easy to get wrong. Lesson 6 removes the question entirely by not removing anything during a loop.
Systems take slices
move_system(positions: noalias []vec2, velocities: []const vec2, dt) gets the rows in use, sparks*.positions[0..sparks*.count], and knows nothing about tables. []const says it only reads velocities. Each positions[i] is bounds checked, in every build, against the count rather than the capacity, so a system that runs past the live rows stops with a message instead of reading stale data.
noalias says that for the call, nothing else reaches the memory positions covers. Only the column the system writes needs it: a promise on the written column already rules out a write landing in the column being read. Without it, clang -O2 on Linux vectorized this loop behind a run-time check that the two columns are apart, and so did gcc -O3 with the bounds checks compiled out. With it, neither checks. A build with the bounds checks in checks the promise instead, once per call.
The promise is one the checker cannot keep, so the two kernels are proc[unsafe], and main calls each from inside unsafe { }:
unsafe { move_system(sparks*.positions[0..sparks*.count], sparks*.velocities[0..sparks*.count], dt); }
age_system(lives: noalias []f32, dt) has one slice, and still takes it. Windows passes a slice by reference, as a pointer to a copy, and clang-cl does not use C's type rules to decide what can overlap. So without the promise, a float written through lives might be the slice's own pointer or length. clang-cl then reads both again after every element and does one float at a time. With it, the loop is eight floats per pass:
movups (%rcx,%r8,4), %xmm2 four lives
movups 0x10(%rcx,%r8,4), %xmm3 four more
subps %xmm0, %xmm2 minus dt
subps %xmm0, %xmm3
movups %xmm2, (%rcx,%r8,4)
movups %xmm3, 0x10(%rcx,%r8,4)
The bounds check does not cost this loop anything: lives.length is already the loop's bound, so the compiler drops it. move_system, with two lengths to check, keeps its vector loop on clang too, as lesson 1 explains; clang-cl does not vectorize it at all without noalias, and gcc -O3 does not with the checks in.
The rule that holds on every compiler: noalias on every slice a loop writes, even when it is the only one.
lives is a plain run of floats. positions and velocities are vec2, x and y side by side, but both get the same math, so the compiler keeps the pairs together and move_system's vector loop needs no shuffles at all: four vec2s a pass in two registers. A vec3, which does not divide into a register, is another matter. Lesson 1 measures what that costs.
Status codes
spark_spawn returns spawn_status, an enum[must_use]: calling it and ignoring the result is a compile error. That is rin's convention for a proc that can fail: a status, with the result through an out-parameter. The ninth spark shows the full case.
Try
- Rewrite
expire_systemto walk forwards with a plainfor. The output does not change, because no frame here expires a spark together with the one in the last row. Change two lives so that one does, and find the spark that outlives its life by a frame. Then fix the loop. - Delete the line in
spark_despawnthat copieslives. What does the output show, and how long did it take to notice? - Call
spark_spawnwithout using its result and read the compiler's error. Then discard it on purpose with_ = spark_spawn(...). - Make the table grow instead of reporting
full:spark_spawnnow needs the region,r: region level, tonew_slicedouble the capacity, and a plain loop copies every column across. It is all checked, since the columns are typed. Lesson 4'skayan_table_growis the same idea over columns of bytes.
1// 2. One archetype table2//3// An archetype is one combination of components. Its table holds every entity4// that has exactly that combination: one column per component, one row per5// entity. It is lesson 1's struct of arrays with two rules added.6//7// Every column has the same length, always. Spawn appends to every column in8// one place and despawn removes from every column in one place. kayan appended9// to each column at the call site, and three appends that must happen together10// are three chances to forget one.11//12// Rows are packed. Despawn moves the last row into the hole it leaves (a13// swap-remove), so a system loops over 0..count with no gaps and no "alive"14// flag to test. The price is that a row is not a stable name for an entity: the15// last entity changes rows every time something before it dies. Lesson 4 deals16// with that.17//18// The sparks here have a position, a velocity and a life in seconds. Each frame19// they move, they age, and the ones whose life ran out are removed. Removal20// walks the rows backwards, so the row swapped into a hole has already been21// looked at this frame.22//23// The arena is the region `level`; the table is made in it and24// its columns are slices that name it. The two kernels are `proc[unsafe]`, because25// `noalias` is a promise the checker cannot check: each call says `unsafe`, and the26// checker sees every other line of this lesson.27 28import "std/print.rin"29 30level: region;31 32vec2: struct = {33 x: f32;34 y: f32;35}36 37// The table for the archetype {position, velocity, life}. Each column is sized38// to the capacity once; count says how many rows are in use. `numbers` is not a39// component: it only lets the output say which spark sits in which row.40spark_table: struct = {41 count: u64;42 capacity: u64;43 numbers: []u32 in level;44 positions: []vec2 in level;45 velocities: []vec2 in level;46 lives: []f32 in level;47}48 49spawn_status: enum[must_use] = {50 ok,51 full,52}53 54// The table and its columns come from the region: the pointer says so, and55// the checker will not let it be kept past the region's end.56spark_table_create: proc(r: region level, capacity: u64) -> *spark_table in level = {57 return new<spark_table>(r, {58 .capacity = capacity,59 .numbers = new_slice<u32>(r, capacity),60 .positions = new_slice<vec2>(r, capacity),61 .velocities = new_slice<vec2>(r, capacity),62 .lives = new_slice<f32>(r, capacity),63 });64}65 66// The only place a row is added: every column gets its value here.67spark_spawn: proc(t: *spark_table, number: u32, position: vec2, velocity: vec2, life: f32,68 row: *u64) -> spawn_status = {69 if (t*.count == t*.capacity) {70 return spawn_status.full;71 }72 r: u64 = t*.count;73 t*.numbers[r] = number;74 t*.positions[r] = position;75 t*.velocities[r] = velocity;76 t*.lives[r] = life;77 t*.count += 1;78 row* = r;79 return spawn_status.ok;80}81 82// The only place a row is removed. The last row moves into `row`, in every83// column, and the table shrinks by one. Returns the row that moved, which is84// `row` itself when the removed row was the last one.85spark_despawn: proc(t: *spark_table, row: u64) -> u64 = {86 last: u64 = t*.count - 1;87 t*.numbers[row] = t*.numbers[last];88 t*.positions[row] = t*.positions[last];89 t*.velocities[row] = t*.velocities[last];90 t*.lives[row] = t*.lives[last];91 t*.count = last;92 return last;93}94 95// Systems. Each takes the columns it reads and writes, cut to the rows in use.96// noalias on the column a system writes promises nothing else reaches it, so97// the loop can be vectorized without runtime overlap checks.98move_system: proc[unsafe](positions: noalias []vec2, velocities: []const vec2, dt: f32) -> void = {99 for (i: u64 = 0; i < positions.length; i += 1) {100 positions[i].x += velocities[i].x * dt;101 positions[i].y += velocities[i].y * dt;102 }103}104 105// One slice, and still noalias: on Windows the slice arrives by reference, and106// without the promise a float written through lives might be its own pointer107// or length, so clang-cl reads both again after every element.108age_system: proc[unsafe](lives: noalias []f32, dt: f32) -> void = {109 for (i: u64 = 0; i < lives.length; i += 1) {110 lives[i] -= dt;111 }112}113 114// Structural change: rows leave the table. Backwards, so the row a swap-remove115// pulls in has already been tested.116expire_system: proc(t: *spark_table) -> void = {117 i: u64 = t*.count;118 while (i > 0) {119 i -= 1;120 if (t*.lives[i] <= 0.0) {121 number: u32 = t*.numbers[i];122 moved: u64 = spark_despawn(t, i);123 if (moved != i) {124 printfmt(" spark {} in row {} expired; spark {} moved from row {} into it\n",125 number, i, t*.numbers[i], moved);126 } else {127 printfmt(" spark {} in row {} expired; it was the last row\n", number, i);128 }129 }130 }131}132 133print_table: proc(t: *spark_table) -> void = {134 for (i: u64 = 0; i < t*.count; i += 1) {135 printfmt(" row {} spark {} position ({:6.2}, {:6.2}) life {:.2}\n",136 cast(i, i32), cast(t*.numbers[i], i32),137 cast(t*.positions[i].x, f64), cast(t*.positions[i].y, f64), cast(t*.lives[i], f64));138 }139}140 141main: proc() -> i32 = {142 level: region = {143 sparks: *spark_table in level = spark_table_create(level, 8);144 printfmt("a table with room for {} sparks, {} bytes of components a row\n", sparks*.capacity,145 sizeof(vec2) * 2 + sizeof(f32));146 147 // Eight sparks with lives of 0.5s to 4s, plus one more than fits.148 for (i: u64 = 0; i < 9; i += 1) {149 f: f32 = cast(i, f32);150 row: u64 = 0;151 life: f32 = 0.5 + cast((i * 5) % 8, f32) * 0.5;152 status: spawn_status = spark_spawn(sparks, cast(i, u32), { .x = f, .y = 0.0 }, { .x = 1.0, .y = f },153 life, row&);154 if (status == spawn_status.full) {155 printfmt("spark {} did not fit: the table is full\n", i);156 }157 }158 print_table(sparks);159 160 dt: f32 = 0.5;161 for (frame: i32 = 1; frame <= 4; frame += 1) {162 printfmt("frame {}\n", frame);163 unsafe { move_system(sparks*.positions[0..sparks*.count], sparks*.velocities[0..sparks*.count], dt); }164 unsafe { age_system(sparks*.lives[0..sparks*.count], dt); }165 expire_system(sparks);166 print_table(sparks);167 }168 printfmt("{} sparks left\n", sparks*.count);169 }170 return 0;171}What archetype_table prints. tools/check.py fails the lesson if a single byte differs.
a table with room for 8 sparks, 20 bytes of components a row
spark 8 did not fit: the table is full
row 0 spark 0 position ( 0.00, 0.00) life 0.50
row 1 spark 1 position ( 1.00, 0.00) life 3.00
row 2 spark 2 position ( 2.00, 0.00) life 1.50
row 3 spark 3 position ( 3.00, 0.00) life 4.00
row 4 spark 4 position ( 4.00, 0.00) life 2.50
row 5 spark 5 position ( 5.00, 0.00) life 1.00
row 6 spark 6 position ( 6.00, 0.00) life 3.50
row 7 spark 7 position ( 7.00, 0.00) life 2.00
frame 1
spark 0 in row 0 expired; spark 7 moved from row 7 into it
row 0 spark 7 position ( 7.50, 3.50) life 1.50
row 1 spark 1 position ( 1.50, 0.50) life 2.50
row 2 spark 2 position ( 2.50, 1.00) life 1.00
row 3 spark 3 position ( 3.50, 1.50) life 3.50
row 4 spark 4 position ( 4.50, 2.00) life 2.00
row 5 spark 5 position ( 5.50, 2.50) life 0.50
row 6 spark 6 position ( 6.50, 3.00) life 3.00
frame 2
spark 5 in row 5 expired; spark 6 moved from row 6 into it
row 0 spark 7 position ( 8.00, 7.00) life 1.00
row 1 spark 1 position ( 2.00, 1.00) life 2.00
row 2 spark 2 position ( 3.00, 2.00) life 0.50
row 3 spark 3 position ( 4.00, 3.00) life 3.00
row 4 spark 4 position ( 5.00, 4.00) life 1.50
row 5 spark 6 position ( 7.00, 6.00) life 2.50
frame 3
spark 2 in row 2 expired; spark 6 moved from row 5 into it
row 0 spark 7 position ( 8.50, 10.50) life 0.50
row 1 spark 1 position ( 2.50, 1.50) life 1.50
row 2 spark 6 position ( 7.50, 9.00) life 2.00
row 3 spark 3 position ( 4.50, 4.50) life 2.50
row 4 spark 4 position ( 5.50, 6.00) life 1.00
frame 4
spark 7 in row 0 expired; spark 4 moved from row 4 into it
row 0 spark 4 position ( 6.00, 8.00) life 0.50
row 1 spark 1 position ( 3.00, 2.00) life 1.00
row 2 spark 6 position ( 8.00, 12.00) life 1.50
row 3 spark 3 position ( 5.00, 6.00) life 2.00
4 sparks left
Advanced · lesson 3
Many archetypes
src/3.many_archetypes/many_archetypes.rin
src/3.many_archetypes/many_archetypes.rin
In the dojo: 4. a type-erased column, 6. bitsets, each on its own, to watch and to practise.
Four archetypes (rock, ship, bullet, turret) and four components (position, velocity, health, damage). One table type holds all four archetypes, and two queries decide which tables each system visits.
A table without a type
Lesson 2's table had one typed field per column. That cannot work for every archetype at once, so the columns become bytes:
table: struct = {
name: ^const c8;
signature: u64;
count: u64;
capacity: u64;
columns: [component<>.count][]u8 in level;
}
columns[id] is empty when the table has no such component, and table_has reads that from the signature. Its element size is component_info[id]*.size, where component_info holds each component's reflect record, position<>& and so on. table_add_row and lesson 4's moves work only from sizes, so they never need to know a component's type. A column is a slice of bytes, so all of that is checked: a row past the end stops the program like any index.
slice<u8>zero(t*.columns[c][row * size..(row + 1) * size]);
Types come back at the edges. column<T>(t) asks component_of<T>() for T's id, views that column's bytes as T, and returns a []T of the rows in use. component_of is one specialisation per component, one line each. Systems take []position, []velocity and so on, exactly as in lesson 2.
What the checker cannot see
Two things are unsafe { } here. table_create allocates columns with a component's alignment, which only the arena can promise. column<T> turns bytes into a type, which no checker can vouch for:
column: proc<T>(t: *const table) -> []T in level = {
c: component = component_of<T>();
if (!table_has(t, c)) {
printfmt("table {} has no {} column\n", t*.name, component_info[c]*.name);
exit(1);
}
if (!reflect_is_plain(T<>&)) {
printfmt("a column holds plain data, and {} is not\n", T<>.name);
exit(1);
}
bytes: []u8 in level = t*.columns[c];
if (t*.count > bytes.length / sizeof(T)) {
printfmt("table {} has {} rows and room for {}\n", t*.name, t*.count, bytes.length / sizeof(T));
exit(1);
}
unsafe {
if (cast(bytes.data, usize) % alignof(T) != 0) {
printfmt("table {}: a column that is not aligned for {}\n", t*.name, T<>.name);
exit(1);
}
raw: ^T = cast(bytes.data, ^T);
return cast(raw[0..t*.count], []T in level);
}
}
Most of that is checks, and each one is there for a reason.
rin has no private fields. Any checked proc can write t*.count, so an unsafe block must not trust a field or an argument unless the value carries its own bound, as a slice does. Here the bytes know their length, and count is measured against them. Write tables[1]*.count = 1000; in main and the program stops rather than hand out a slice past the column:
table ship has 1000 rows and room for 16
Set a bit in a signature by hand and it stops the same way, since the column behind that bit is empty.
A plain-data T means any bytes are a valid T. reflect_is_plain accepts numbers, enums, and arrays and structs of them, and refuses a pointer. So reading the wrong type gives wrong numbers, never a bad pointer. Make component_of<health> return component.position and the program runs to the end, with ships at ( 62.0, 100.0). That is a bug, and it is one the output shows. It is not a memory error.
What comes back is []T in level, and it is a borrow of the table, good until a row is added or removed. Everything above those two procs is checked, and works in slices and *table. A struct holding a raw pointer, here the table's name, is only ever made by a proc that vouches for it, so main holds the tables by pointer, in the region:
tables: [4]*table in level = {
table_create(level, "rock", rock_components, 16),
table_create(level, "ship", ship_components, 16),
table_create(level, "bullet", bullet_components, 16),
table_create(level, "turret", turret_components, 16),
};
Sized by reflection
component has no count member at the end. component<>.count is the number of members, from the enum's reflect record, and it sizes both columns and component_info. With a hand-added count member the reflected count is one more than the number of components. That is how the bounds check found a bug while this lesson was being written.
component_info is a global, and globals live in permanent, the region of the whole program, so its type says that of the records it holds:
component_info: [component<>.count]*const reflect in permanent = {
components_check compares each enum member's name with the name in component_info. Swap two entries in component_info and the program stops before anything runs:
component 2 is health in the enum but damage in component_info
Different types, fewer checks
move_system takes noalias []position, as lesson 2's takes noalias []vec2, and for the same reason, so it is proc[unsafe] too. regen_system takes noalias []health, its only slice, for the reason lesson 2 gives for age_system.
One difference shows without noalias: clang on Linux vectorizes this move_system without a run-time overlap check, and lesson 2's only with one. Here the two columns are different struct types, position and velocity, and clang reads C's type rules as saying they cannot share memory; built with -fno-strict-aliasing, it checks here too. In lesson 2 both are vec2. gcc checks in both, once it vectorizes at all (lesson 1). clang-cl, rin's default compiler, does not use those rules, to match MSVC, so on Windows neither lesson vectorizes without it. noalias makes the answer the same everywhere, whatever the column types are.
Queries
moving: query = { .all = bit(component.position) | bit(component.velocity) };
healing: query = { .all = bit(component.health), .none = bit(component.damage) };
A table matches when it has every bit in all and none in none. moving visits ships and bullets. healing visits ships but not turrets, which have health but also deal damage. The output prints each visit.
The signature cannot lie
table_create builds the signature from the component list it allocates columns for. kayan wrote them separately, and archetype D's signature left out two of its columns (see 0_from_kayan.md).
Try
- Add an
armorcomponent, give it to turrets, and make a query that visits only tables with bothhealthandarmor. - Ask a rock for
column<velocity>. Read the message, then find where it comes from. - Change
columnsto[4][]u8 in level. Then add a fifth component, give it to a table, and see what happens, and where. - Write
table_remove_rowwith the swap-remove from lesson 2, working only fromcomponent_infosizes andslice<u8>copy. It needs nounsafe, since both ends of each copy are slices. - Give
damagea field that holds a pointer,owner: option<*position in level>, and read whatcolumn<damage>says. Now see whatcolumn<T>'s checks are for. The rest of this exercise misuses memory on purpose, so run it in a VM, not on your machine.
Take out the plain-data check. The program prints what it always did, because nothing reads owner. So read one: declare lookalike: struct = { owner: option<*position in level>; }, make its component_of return component.position, and read owner from the second ship's row. rin accepts it, and the ship's position, (0.0, 10.0), faults as an address. That is the bad pointer plain data rules out.
Put that check back and take out the check of count against the bytes. Set a ship table's count to 1000, call table_set<position> at row 16, and set count back to 2. Under AddressSanitizer it stops at that write. Without the sanitizers nothing reports it: row 16 is the first ship's velocity, and that ship ends frame 1 at (0.5, 0.5) instead of (1.0, 0.0). Keep writing to row 199 and you pass the health column and reach the bullet table's own fields at row 48, its name pointer among them, and the program dies with no message.
1// 3. Many archetypes2//3// A game has more than one combination of components. Rocks only have a4// position. Ships move and can be hurt. Bullets move and hurt. Turrets stand5// still, can be hurt and hurt back. Each combination is a table of its own, and6// a system has to visit every table that has the columns it needs.7//8// kayan wrote a C struct per archetype, gave each a hand-set signature, and at9// query time looked each column up by name in a string map. Here one table type10// serves every archetype:11//12// * A component is a member of the `component` enum, so its id is a small13// integer, and a set of components fits in a u64 mask, one bit each.14// * A table's columns are `[component<>.count][]u8 in level`, indexed by15// component id and empty where the table has no such column. Finding a column16// is an array index, not a string lookup.17// * A column's element size comes from reflection: `component_info[id]` is the18// component type's reflect record, `position<>&` and so on.19// * A table's signature is computed from the components it is created with,20// so the signature cannot disagree with the columns. In kayan it could, and21// it did: archetype D held positions and velocities, its signature left them22// out, and the movement query skipped it.23//24// A query is two masks: a table matches when it has every component in `all`25// and none in `none`. Typed access comes back through generics: `column<T>`26// asks `component_of<T>()` which id T has, one line per component type, and27// returns that column as a []T cut to the rows in use.28//29// A type-erased column is bytes with a size, and reading it as a []T is the one30// thing the checker cannot vouch for. So there is a small unsafe core, two procs:31// table_create (allocates columns with the right alignment) and column<T> (the32// cast), which checks what it is given rather than trust it. Everything above33// them is checked.34 35import "std/print.rin"36import "std/memops.rin"37import "std/slice.rin"38import "std/reflect.rin"39 40level: region;41 42vec2: struct = {43 x: f32;44 y: f32;45}46 47// The components. Each enum member is named after its type, so the program can48// check at startup that this list and `component_info` agree. There is no49// `count` member: `component<>.count` is the number of members, from reflection.50component: enum = {51 position,52 velocity,53 health,54 damage,55}56 57position: struct = { at: vec2; }58velocity: struct = { per_second: vec2; }59health: struct = { points: f32; max: f32; }60damage: struct = { per_hit: f32; }61 62// Globals live in `permanent`, the region of the whole program.63component_info: [component<>.count]*const reflect in permanent = {64 position<>&,65 velocity<>&,66 health<>&,67 damage<>&,68};69 70component_of: proc<position>() -> component = { return component.position; }71component_of: proc<velocity>() -> component = { return component.velocity; }72component_of: proc<health>() -> component = { return component.health; }73component_of: proc<damage>() -> component = { return component.damage; }74 75bit: proc(c: component) -> u64 = {76 return 1'u64 shl cast(c, u64);77}78 79// One archetype. Every column has `capacity` elements of its component's size.80table: struct = {81 name: ^const c8;82 signature: u64;83 count: u64;84 capacity: u64;85 columns: [component<>.count][]u8 in level;86}87 88query: struct = {89 all: u64;90 none: u64;91}92 93// Stops the program if the enum and component_info were written in different94// orders: a column would silently get another component's size. The enum's95// members come from its reflect record, one at a time, as `option<*const>`: null past96// the end, which the loop bound already rules out.97components_check: proc() -> void = {98 for (i: u64 = 0; i < component<>.count; i += 1) {99 member: option<*const reflect_value> = reflect_value_at(component<>&, i);100 if (member && !reflect_cstr_equal(member*.name, component_info[i]*.name)) {101 printfmt("component {} is {} in the enum but {} in component_info\n", i,102 member*.name, component_info[i]*.name);103 exit(1);104 }105 }106}107 108// Unsafe: the columns are bytes with the component's alignment, which is an arena109// push the checker cannot type. The table itself is made here too, since a struct110// holding a raw pointer, its name, is only ever made by a proc that vouches for it.111table_create: proc(r: region level, name: ^const c8, components: []const component,112 capacity: u64) -> *table in level = {113 unsafe {114 t: *table in level = new<table>(r);115 t*.name = name;116 t*.capacity = capacity;117 for (i: u64 = 0; i < components.length; i += 1) {118 c: component = components[i];119 info: *const reflect = component_info[c];120 t*.signature |= bit(c);121 raw: ^u8 = cast(memops_arena_push_zero(r, info*.size * capacity, info*.align), ^u8);122 t*.columns[c] = cast(raw[0..info*.size * capacity], []u8 in level);123 }124 return t;125 }126}127 128table_has: proc(t: *const table, c: component) -> b32 = {129 return (t*.signature & bit(c)) != 0;130}131 132table_matches: proc(t: *const table, q: query) -> b32 = {133 return (t*.signature & q.all) == q.all && (t*.signature & q.none) == 0;134}135 136// A new row with every component zeroed. The caller fills it in.137table_add_row: proc(t: *table) -> u64 = {138 if (t*.count == t*.capacity) {139 printfmt("table {} is full\n", t*.name);140 exit(1);141 }142 row: u64 = t*.count;143 for (c: u64 = 0; c < component<>.count; c += 1) {144 if ((t*.signature & (1'u64 shl c)) != 0) {145 size: u64 = component_info[c]*.size;146 slice<u8>zero(t*.columns[c][row * size..(row + 1) * size]);147 }148 }149 t*.count += 1;150 return row;151}152 153// The rows in use of T's column. Asking a table for a column it does not have154// is a bug in the caller, so it stops the program rather than return nothing.155// This is the one place bytes become a type, so it takes nothing on trust: T must156// be plain data, so reading the wrong type gives wrong numbers and never a bad157// pointer, and the rows must fit in the bytes, on an address T may live at. The158// slice is a borrow of the table: good until a row is added or removed.159column: proc<T>(t: *const table) -> []T in level = {160 c: component = component_of<T>();161 if (!table_has(t, c)) {162 printfmt("table {} has no {} column\n", t*.name, component_info[c]*.name);163 exit(1);164 }165 if (!reflect_is_plain(T<>&)) {166 printfmt("a column holds plain data, and {} is not\n", T<>.name);167 exit(1);168 }169 bytes: []u8 in level = t*.columns[c];170 if (t*.count > bytes.length / sizeof(T)) {171 printfmt("table {} has {} rows and room for {}\n", t*.name, t*.count, bytes.length / sizeof(T));172 exit(1);173 }174 unsafe {175 if (cast(bytes.data, usize) % alignof(T) != 0) {176 printfmt("table {}: a column that is not aligned for {}\n", t*.name, T<>.name);177 exit(1);178 }179 raw: ^T = cast(bytes.data, ^T);180 return cast(raw[0..t*.count], []T in level);181 }182}183 184table_set: proc<T>(t: *table, row: u64, value: T) -> void = {185 values: []T = column<T>(t);186 values[row] = value;187}188 189// Systems take columns, as in lesson 2. The loops that feed them are queries.190move_system: proc[unsafe](positions: noalias []position, velocities: []const velocity, dt: f32) -> void = {191 for (i: u64 = 0; i < positions.length; i += 1) {192 positions[i].at.x += velocities[i].per_second.x * dt;193 positions[i].at.y += velocities[i].per_second.y * dt;194 }195}196 197regen_system: proc[unsafe](healths: noalias []health, per_second: f32, dt: f32) -> void = {198 for (i: u64 = 0; i < healths.length; i += 1) {199 healths[i].points += per_second * dt;200 if (healths[i].points > healths[i].max) {201 healths[i].points = healths[i].max;202 }203 }204}205 206print_tables: proc(tables: []*table in level) -> void = {207 for (ti: u64 = 0; ti < tables.length; ti += 1) {208 t: *table in level = tables[ti];209 for (row: u64 = 0; row < t*.count; row += 1) {210 printfmt(" {:-7} row {}", t*.name, cast(row, i32));211 if (table_has(t, component.position)) {212 ps: []position = column<position>(t);213 printfmt(" at ({:5.1}, {:5.1})", cast(ps[row].at.x, f64), cast(ps[row].at.y, f64));214 }215 if (table_has(t, component.health)) {216 hs: []health = column<health>(t);217 printfmt(" health {:5.1}/{:.0}", cast(hs[row].points, f64), cast(hs[row].max, f64));218 }219 if (table_has(t, component.damage)) {220 ds: []damage = column<damage>(t);221 printfmt(" damage {:.0}", cast(ds[row].per_hit, f64));222 }223 printfmt("\n");224 }225 }226}227 228spawn_ship: proc(t: *table, x: f32, y: f32) -> void = {229 row: u64 = table_add_row(t);230 table_set<position>(t, row, { .at = { .x = x, .y = y } });231 table_set<velocity>(t, row, { .per_second = { .x = 2.0, .y = 0.0 } });232 table_set<health>(t, row, { .points = 50.0, .max = 100.0 });233}234 235spawn_bullet: proc(t: *table, x: f32, y: f32, dy: f32) -> void = {236 row: u64 = table_add_row(t);237 table_set<position>(t, row, { .at = { .x = x, .y = y } });238 table_set<velocity>(t, row, { .per_second = { .x = 0.0, .y = dy } });239 table_set<damage>(t, row, { .per_hit = 10.0 });240}241 242main: proc() -> i32 = {243 components_check();244 245 level: region = {246 rock_components: [1]component = { component.position };247 ship_components: [3]component = { component.position, component.velocity, component.health };248 bullet_components: [3]component = { component.position, component.velocity, component.damage };249 turret_components: [3]component = { component.position, component.health, component.damage };250 251 // Four tables in the region, held by pointer: a table holds a raw C string,252 // so safe code never holds one by value.253 tables: [4]*table in level = {254 table_create(level, "rock", rock_components, 16),255 table_create(level, "ship", ship_components, 16),256 table_create(level, "bullet", bullet_components, 16),257 table_create(level, "turret", turret_components, 16),258 };259 260 for (ti: u64 = 0; ti < 4; ti += 1) {261 printfmt("{} signature {}:", tables[ti]*.name, tables[ti]*.signature);262 for (c: u64 = 0; c < component<>.count; c += 1) {263 if ((tables[ti]*.signature & (1'u64 shl c)) != 0) {264 printfmt(" {}", component_info[c]*.name);265 }266 }267 printfmt("\n");268 }269 270 rock: u64 = table_add_row(tables[0]);271 table_set<position>(tables[0], rock, { .at = { .x = 5.0, .y = 5.0 } });272 spawn_ship(tables[1], 0.0, 0.0);273 spawn_ship(tables[1], 0.0, 10.0);274 spawn_bullet(tables[2], 1.0, 0.0, 8.0);275 spawn_bullet(tables[2], 2.0, 0.0, 8.0);276 spawn_bullet(tables[2], 3.0, 0.0, -8.0);277 turret: u64 = table_add_row(tables[3]);278 table_set<position>(tables[3], turret, { .at = { .x = 20.0, .y = 0.0 } });279 table_set<health>(tables[3], turret, { .points = 40.0, .max = 200.0 });280 table_set<damage>(tables[3], turret, { .per_hit = 25.0 });281 print_tables(tables);282 283 moving: query = { .all = bit(component.position) | bit(component.velocity) };284 // Turrets have health too, but they repair by other means: no regen for285 // anything that deals damage.286 healing: query = { .all = bit(component.health), .none = bit(component.damage) };287 288 dt: f32 = 0.5;289 for (frame: i32 = 1; frame <= 2; frame += 1) {290 printfmt("frame {}\n", frame);291 for (ti: u64 = 0; ti < 4; ti += 1) {292 t: *table in level = tables[ti];293 if (table_matches(t, moving)) {294 printfmt(" move visits {}\n", t*.name);295 // move_system takes []const velocity. Widening a call's result296 // reads it twice, so rin wants it in a local first.297 velocities: []velocity = column<velocity>(t);298 unsafe { move_system(column<position>(t), velocities, dt); }299 }300 if (table_matches(t, healing)) {301 printfmt(" regen visits {}\n", t*.name);302 unsafe { regen_system(column<health>(t), 10.0, dt); }303 }304 }305 print_tables(tables);306 }307 }308 return 0;309}What many_archetypes prints. tools/check.py fails the lesson if a single byte differs.
rock signature 1: position
ship signature 7: position velocity health
bullet signature 11: position velocity damage
turret signature 13: position health damage
rock row 0 at ( 5.0, 5.0)
ship row 0 at ( 0.0, 0.0) health 50.0/100
ship row 1 at ( 0.0, 10.0) health 50.0/100
bullet row 0 at ( 1.0, 0.0) damage 10
bullet row 1 at ( 2.0, 0.0) damage 10
bullet row 2 at ( 3.0, 0.0) damage 10
turret row 0 at ( 20.0, 0.0) health 40.0/200 damage 25
frame 1
move visits ship
regen visits ship
move visits bullet
rock row 0 at ( 5.0, 5.0)
ship row 0 at ( 1.0, 0.0) health 55.0/100
ship row 1 at ( 1.0, 10.0) health 55.0/100
bullet row 0 at ( 1.0, 4.0) damage 10
bullet row 1 at ( 2.0, 4.0) damage 10
bullet row 2 at ( 3.0, -4.0) damage 10
turret row 0 at ( 20.0, 0.0) health 40.0/200 damage 25
frame 2
move visits ship
regen visits ship
move visits bullet
rock row 0 at ( 5.0, 5.0)
ship row 0 at ( 2.0, 0.0) health 60.0/100
ship row 1 at ( 2.0, 10.0) health 60.0/100
bullet row 0 at ( 1.0, 8.0) damage 10
bullet row 1 at ( 2.0, 8.0) damage 10
bullet row 2 at ( 3.0, -8.0) damage 10
turret row 0 at ( 20.0, 0.0) health 40.0/200 damage 25
Advanced · lesson 4
Entity handles
src/4.entity_handles/entity_handles.rin
src/4.entity_handles/entity_handles.rin, includes/kayan.rin
In the dojo: 1. free list and generations, 3. the slot map, 9. interning, each on its own, to watch and to practise.
The world: tables made on demand, entity handles that survive row moves, despawning with index reuse, and moving an entity between archetypes when a component is added or removed. The code is includes/kayan.rin. The lesson program calls each part and prints what happened.
Handles, slots, rows
kayan_entity: struct = { index: u32; generation: u32; }
kayan_slot: struct = { generation: u32; table: u32; row: u32; }
A handle is what the rest of a program keeps. Its index picks a slot, and the slot says where the entity lives now. Rows move all the time: every despawn and every archetype change swap-removes. When one does, the table's entities column says which entity owned the moved row, and that entity's slot gets the new row number. Handles never change.
generation is what makes a handle safe to keep after its entity dies. Despawning bumps the slot's generation and frees the index, and the next spawn reuses the index with the new generation:
spawn d: it reuses a's index with the next generation
d = 0:2 in table 0 row 2 {position velocity} at (4.0, 0.0)
a = 0:1 is dead
kayan_alive is one comparison: the handle's generation against the slot's. A free slot already holds its next generation, which no handle carries yet, so no separate "alive" flag is needed. a cannot reach d's data, kayan_get returns null for it, and despawning it again returns kayan_status.dead and changes nothing.
Generations start at 1. A zeroed handle, {}, is 0:0, which never matches a slot, so it can stand for "no entity" without a separate flag.
A handle to keep, a pointer to borrow
kayan_get can miss, and its type says so. It returns option<*T>, which the checker will not let you read until you have tested it:
p: option<*position> = kayan_get<position>(w, e);
if (p) {
printfmt(" at ({:.1}, {:.1})", cast(p*.at.x, f64), cast(p*.at.y, f64));
}
The pointer is a borrow of the world, good until the next spawn, despawn, add or remove, which can move what it points at. The checker holds you to that: use one after a call that can change the world and rin says
'kept' may be stale: 'world', which it was taken from, has been changed since; take it again
So a pointer is for now, and the handle is for later. That was already this lesson's advice; now it is checked.
The world in a region
world: *kayan_world in level = kayan_world_make(level);
kayan_world_make makes the world whole, inside the region, and every list it holds says where it lives:
tables: *vec<kayan_table> in level;
by_signature: *hashmap<u64, u32> in level;
slots: *vec<kayan_slot> in level;
free: *vec<u32> in level;
A table holds lesson 3's columns of bytes, and its entities, as slices:
kayan_table: struct = {
signature: u64;
count: u64;
capacity: u64;
entities: []kayan_entity in level;
columns: [64][]u8 in level;
}
Each slice knows its own length, so nothing here depends on a count being right. Almost all of includes/kayan.rin is checked because of that: spawning, despawning, moving, growing a table, adding and removing a row are plain indexing and slice<u8>copy, and a row past the end stops the program. Only three procs need unsafe { }. kayan_world_make keeps the region's arena, a raw pointer, so columns can grow without being handed the region on each call. kayan_bytes takes a block from that arena with a component's alignment. kayan_view<T> is the one place bytes become a type:
kayan_view: proc<T>(bytes: []u8 in level, count: u64) -> []T in level = {
if (!reflect_is_plain(T<>&)) {
printfmt("kayan: a column holds plain data, and {} is not\n", T<>.name);
exit(1);
}
if (count > bytes.length / sizeof(T)) {
printfmt("kayan: a view of {} {} on {} bytes\n", count, T<>.name, bytes.length);
exit(1);
}
unsafe {
if (cast(bytes.data, usize) % alignof(T) != 0) {
printfmt("kayan: bytes that are not aligned for {}\n", T<>.name);
exit(1);
}
raw: ^T = cast(bytes.data, ^T);
return cast(raw[0..count], []T in level);
}
}
It takes nothing on trust, for the reasons lesson 3 gives. rin has no private fields: a program can write kayan_table_at(world, 0)*.count = 1000;, and an unsafe block must not trust a field or an argument unless the value carries its own bound, as a slice does. That write stops at the next view:
kayan: a view of 1000 position on 128 bytes
And a component must be plain data, so any bytes are a valid T: reading the wrong type gives wrong numbers, never a bad pointer. Everything else, kayan_column and kayan_get included, is written in terms of this. The lesson program itself has no unsafe at all.
Statuses
kayan_despawn, kayan_add, kayan_remove and kayan_move return kayan_status, an enum[must_use]: ok, or dead when the handle no longer names a live entity. Ignoring one is a compile error. The lesson discards them on purpose with _ = kayan_add<health>(...) where the handle is known to be alive.
Tables on demand
kayan_table_for(w, signature) looks the signature up in a hashmap<u64, u32> and makes the table if it is new. Nobody declares archetypes; they appear when an entity first needs that combination. Tables live in a vec<kayan_table> and are referred to by index, because adding one can move the whole list.
Tables start with room for 16 rows and double when full. The old columns stay in the region unused, which is a waste these lessons accept. The last part of the program spawns 40 entities so the first table grows twice, despawns every other one, and checks the result two ways.
Checking the world
Reading back each survivor's position is a weak check. Swap-remove copies rows, so a stale row often still holds the right numbers. kayan_verify checks the invariant instead: every row of every table belongs to a live entity whose slot points back at that table and row, and the rows add up to the live count. It prints each problem it finds and returns how many there were. The lesson prints kayan_verify: 0 problems and fails if that is not so.
kayan_get also checks the slot's row against the rows in use, in every build, so a slot that points past the end of its table stops the program at the first lookup.
Moving between archetypes
give b health: b moves to a new table and keeps its position
kayan_add<health> computes signature | bit(health), finds or makes that table, appends a row, copies every component the two tables share, then swap-removes the old row. kayan_remove is the same move with the bit cleared. A move costs two row copies, one into the new table and one to fill the hole in the old, so components that come and go every frame are better as a field than as a component. A burning flag in health, say, instead of a burning component.
Try
- Right after the first three spawns, despawn
b, spawn anx, which reusesb's index, and keep usingb's handle. Every call notices:kayan_alivesays no,kayan_getreturns null,kayan_setrefuses,kayan_signatureis 0,kayan_get_bytesis empty, andkayan_add,kayan_removeandkayan_despawnreturndead.xis untouched. Now makekayan_alivestop checking the generation. None of them notices, and they all land onx:kayan_setoverwrites it,kayan_addandkayan_removemove it, andkayan_despawnkills it.kayan_verifystill says 0 problems, because the world is consistent. It is only the wrong entity, a wrong answer and not a memory error. Further on, the lesson's own print of deadastops on a bounds check, since its row is gone. - Remove the line in
kayan_table_remove_rowthat updates the moved entity's slot. The program stops at the firstkayan_getthrough a stale row, withindex 2 out of bounds (length 2), in Debug and in Release. Printkayan_verify(world)right after the first despawn, and it names the slot:2:1 is in table 0 row 0, its slot says table 0 row 2. Now build it with-DRIN_NO_BOUNDS_CHECKS, without that print.creads back(3.0, 0.0)from a row nobody owns, the right numbers from the wrong place, and the program runs on untilkayan_table_remove_row's own check stops it:kayan: row 30 is past the 28 rows in use. - At the end of the lesson, table 0 has 22 rows and room for 64. Hold a
[]positionfromkayan_column, spawn at least 43 more so the table grows, then write through the old slice. rin refuses, with the same message. Compile it withrin compile --unsafe, which turns the rules off, and read the position back throughkayan_get. The write is lost, and nothing reports it. That stale slice is why lesson 6 defers spawns. Now do what the message says, and take the slice again after the spawns. - Add
kayan_has<T>(w, e) -> b32without moving anything. - Make
component_of<velocity>returncomponent.position. Both are 8 bytes, so nothing stops it: position and velocity now share a column, every position reads(0.0, 1.0), and the lesson's own position check fails, with no crash anywhere. Then makecomponent_of<health>returncomponent.velocity, which is a different size, and read whatkayan_viewsays.
Imports the world from includes/kayan.rin.
1// 4. Entity handles2//3// Lesson 2 ended on a problem: swap-remove keeps a table packed, but it moves4// rows, so "spark 7 is row 7" stops being true the moment something before it5// dies. Anything that remembers an entity by its row (a target, a parent, a UI6// selection) ends up pointing at whoever moved in.7//8// The fix is one level of indirection that does not move:9//10// * A handle is {index, generation}. The index picks a slot in the world's11// slot list; the slot says which table and row the entity is in now.12// * Every table keeps an `entities` column: the handle that owns each row.13// When a swap-remove moves the last row, that column says whose slot to14// update. That is the line lesson 2 could not write.15// * Despawning bumps the slot's generation and puts the index on a free list.16// The next spawn reuses the index with the new generation, so every old17// handle to the dead entity fails the generation check instead of quietly18// finding the newcomer. Generations start at 1, so a zeroed handle is never19// alive and can mean "no entity".20//21// The same slot is what lets an entity change archetype. Adding a component22// moves the entity to the table with one more bit: a row is appended there,23// the components both tables share are copied, and the old row is24// swap-removed. Removing a component is the same move the other way. kayan had25// no way to do either: its entity id was a running count, and nothing recorded26// where an entity lived.27//28// The world itself is in includes/kayan.rin, which lessons 5 and 6 use as29// it is. This program walks through what it does and prints each step.30//31// Nothing in this file is unsafe. The world is made in the32// region `level` and handed around as `*kayan_world`; `kayan_get` returns33// `option<*T>`, so every read tests it, and `kayan_set` writes a component the same34// way. The library holds the small unsafe core; see its header.35 36import "../../includes/kayan.rin"37 38vec2: struct = {39 x: f32;40 y: f32;41}42 43component: enum = {44 position,45 velocity,46 health,47}48 49position: struct = { at: vec2; }50velocity: struct = { per_second: vec2; }51health: struct = { points: f32; }52 53component_of: proc<position>() -> component = { return component.position; }54component_of: proc<velocity>() -> component = { return component.velocity; }55component_of: proc<health>() -> component = { return component.health; }56 57register_components: proc(w: *kayan_world) -> void = {58 kayan_register(w, kayan_id_of<position>(), position<>&);59 kayan_register(w, kayan_id_of<velocity>(), velocity<>&);60 kayan_register(w, kayan_id_of<health>(), health<>&);61}62 63print_entity: proc(w: *kayan_world, name: ^const c8, e: kayan_entity) -> void = {64 printfmt(" {} = {}:{}", name, e.index, e.generation);65 if (!kayan_alive(w, e)) {66 printfmt(" is dead\n");67 return;68 }69 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));70 printfmt(" in table {} row {} ", slot*.table, slot*.row);71 kayan_print_signature(w, kayan_signature(w, e));72 p: option<*position> = kayan_get<position>(w, e);73 if (p) {74 printfmt(" at ({:.1}, {:.1})", cast(p*.at.x, f64), cast(p*.at.y, f64));75 }76 h: option<*health> = kayan_get<health>(w, e);77 if (h) {78 printfmt(" health {:.0}", cast(h*.points, f64));79 }80 printfmt("\n");81}82 83spawn_mover: proc(w: *kayan_world, x: f32) -> kayan_entity = {84 e: kayan_entity = kayan_spawn(w, kayan_mask_of<position>() | kayan_mask_of<velocity>());85 _ = kayan_set<position>(w, e, { .at = { .x = x, .y = 0.0 } });86 _ = kayan_set<velocity>(w, e, { .per_second = { .x = 0.0, .y = 1.0 } });87 return e;88}89 90main: proc() -> i32 = {91 wrong: u32 = 0;92 problems: u64 = 0;93 level: region = {94 world: *kayan_world in level = kayan_world_make(level);95 register_components(world);96 97 printfmt("spawn a, b, c\n");98 a: kayan_entity = spawn_mover(world, 1.0);99 b: kayan_entity = spawn_mover(world, 2.0);100 c: kayan_entity = spawn_mover(world, 3.0);101 print_entity(world, "a", a);102 print_entity(world, "b", b);103 print_entity(world, "c", c);104 105 printfmt("despawn a: c is swapped into a's row, and its slot follows\n");106 despawned: kayan_status = kayan_despawn(world, a);107 printfmt(" despawn returned {}\n", despawned);108 print_entity(world, "a", a);109 print_entity(world, "c", c);110 111 printfmt("spawn d: it reuses a's index with the next generation\n");112 d: kayan_entity = spawn_mover(world, 4.0);113 print_entity(world, "d", d);114 print_entity(world, "a", a);115 printfmt(" a's position through the old handle is {}\n",116 kayan_get<position>(world, a) == null ? "null" : "found (a bug)");117 printfmt(" despawning a again returns {}\n", kayan_despawn(world, a));118 nobody: kayan_entity = {};119 printfmt(" a zeroed handle {}:{} is {}\n", nobody.index, nobody.generation,120 kayan_alive(world, nobody) ? "alive (a bug)" : "dead");121 122 printfmt("give b health: b moves to a new table and keeps its position\n");123 _ = kayan_add<health>(world, b, { .points = 75.0 });124 print_entity(world, "b", b);125 print_entity(world, "d", d);126 127 printfmt("take b's velocity: another move\n");128 _ = kayan_remove<velocity>(world, b);129 print_entity(world, "b", b);130 131 printfmt("the tables, with the handles in each row\n");132 kayan_print_tables(world);133 134 // Enough spawns to make a table grow past its first 16 rows. Then two135 // checks: every survivor still reads its own position, and kayan_verify136 // finds every row's owner and every owner's slot in agreement.137 printfmt("spawn 40 more\n");138 many: [40]kayan_entity = {};139 for (i: u64 = 0; i < 40; i += 1) {140 many[i] = spawn_mover(world, cast(i, f32));141 }142 for (i: u64 = 0; i < 40; i += 2) {143 _ = kayan_despawn(world, many[i]);144 }145 for (i: u64 = 1; i < 40; i += 2) {146 p: option<*position> = kayan_get<position>(world, many[i]);147 if (!p || p*.at.x != cast(i, f32)) {148 wrong += 1;149 }150 }151 problems = kayan_verify(world);152 printfmt(" despawned every other one; {} of the 20 left read their own position\n", 20 - wrong);153 printfmt(" kayan_verify: {} problems\n", problems);154 printfmt(" {} entities alive, {} tables\n", world*.live, world*.tables*.length);155 }156 return wrong == 0 && problems == 0 ? 0 : 1;157}What entity_handles prints. tools/check.py fails the lesson if a single byte differs.
spawn a, b, c
a = 0:1 in table 0 row 0 {position velocity} at (1.0, 0.0)
b = 1:1 in table 0 row 1 {position velocity} at (2.0, 0.0)
c = 2:1 in table 0 row 2 {position velocity} at (3.0, 0.0)
despawn a: c is swapped into a's row, and its slot follows
despawn returned ok
a = 0:1 is dead
c = 2:1 in table 0 row 0 {position velocity} at (3.0, 0.0)
spawn d: it reuses a's index with the next generation
d = 0:2 in table 0 row 2 {position velocity} at (4.0, 0.0)
a = 0:1 is dead
a's position through the old handle is null
despawning a again returns dead
a zeroed handle 0:0 is dead
give b health: b moves to a new table and keeps its position
b = 1:1 in table 1 row 0 {position velocity health} at (2.0, 0.0) health 75
d = 0:2 in table 0 row 1 {position velocity} at (4.0, 0.0)
take b's velocity: another move
b = 1:1 in table 2 row 0 {position health} at (2.0, 0.0) health 75
the tables, with the handles in each row
table 0 {position velocity}: 2:1 0:2
table 1 {position velocity health}:
table 2 {position health}: 1:1
spawn 40 more
despawned every other one; 20 of the 20 left read their own position
kayan_verify: 0 problems
23 entities alive, 3 tables
Advanced · lesson 5
The inspector
src/5.inspector/inspector.rin
src/5.inspector/inspector.rin
In the dojo: 11. a reflection walk, on its own, to watch and to practise.
A soldier and a crate, printed field by field by code that names no component type, then edited by component and field name, then a table layout report. Everything comes from the reflect records the world already holds for sizes.
What a reflect record has
T<> is a reflect record: name, size, align, kind, count, and for a struct, count fields. Each field has its name, offset, size, type ("f32" for a builtin, "array_16_c8" for a [16]c8), elem_type and array_count for arrays, and info, the record of the field's own type when it has one. A nested struct or an enum field has info; a builtin like f32 does not.
The walk asks for these through procs that return option<*const>, because a record can run out of fields and a builtin has no record of its own:
f: option<*const reflect_field> = reflect_field_at(type, i);
...
info: option<*const reflect> = reflect_field_info(f);
print_fields walks that. It takes the component's bytes as a []const u8, from kayan_get_bytes, and cuts each field's bytes out of it by offset and size, so a record that disagreed with the data would stop the program in any build. Struct fields recurse through info, enum fields print their member name through reflect_name_from_value_or, a [16]c8 prints as text, and builtins print by type name. Because components are structs, even a single float like health.points has a name to print.
Bytes as a value
The walk is checked. The last step is not: a field's bytes may sit at any address, so they cannot simply be cast to a *f32. They go through a local, which is aligned, with memcpy, and memcpy is C's, so that is unsafe. These are the lesson's small unsafe procs, one a type, each checking the length it is given:
bytes_as_f32: proc(at: []const u8) -> f32 = {
unsafe {
if (at.length < sizeof(f32)) { printfmt("inspector: too few bytes for an f32\n"); exit(1); }
value: f32 = 0.0;
memcpy(value&, at.data, sizeof(f32));
return value;
}
}
bytes_as_u32, bytes_as_i32 and bytes_set_f32 are the same, print_text prints a [N]c8, and cstr_into copies a C string into one. The bytes come from kayan_get_bytes, which is checked: a []u8 cut from the column.
Tags
health: struct = { points: f32; max: f32; regen_timer: f32 @ "hidden"; }
@ "hidden" is a string attached to the field, read with reflect_field_has_attr. The inspector skips it. The component decides what a tool shows, and the tool does not need to know the component.
Names belong in tools
set_f32_by_name(world, soldier, "health", "points", 55.0) finds the component by name, the field by name, checks the field is an f32, and writes it. It refuses weapon.ammo, a u32, and the crate's health, which it does not have. That is the flexibility an editor or a console needs. It is also the string comparison kayan did in every query, once per archetype. Here it runs once per edit, and systems keep using ids.
Layout
print_layout lists each table's columns with size and alignment, and each column's bytes in use out of the bytes allocated for the table's capacity. It is the kind of report that shows when a component has grown padding, when a table has far more room than rows, or when a table carries a column most of its systems never read.
Try
- Add a
bool-likeb32field toweaponand teachprint_builtinto print it asyesorno. It needs a reader besidebytes_as_u32, through a local in the same way. - Give
position'sheadingfield a@ "readonly"tag, the wayhealthtagsregen_timer, and makeset_f32_by_namerefuse fields that carry it. - Write
set_by_nameforu32andi32as well, picking the write fromf*.type. Each new write is another small unsafe proc, likebytes_set_f32. - Print a whole table as rows and columns: one line per entity, one column per field.
Imports the world from includes/kayan.rin.
1// 5. The inspector2//3// Type-erased columns are fast and opaque: to a debugger a column is `[]u8`.4// But every component already has a reflect record, registered in lesson 4 so5// the world knew its size. The same record lists the component's fields with6// their names, offsets, sizes and types, and links each struct or enum field to7// a record of its own. That is enough to print any entity, field by field,8// without writing a print proc for any component.9//10// This is where looking things up by name belongs. kayan looked up columns by11// name in every query, once per archetype, to find what to move. Here systems still12// find columns by id, as in lesson 3, and names are only used by tools: this13// inspector, and the editor-style write at the end that sets a field from its14// component and field names.15//16// Fields can carry tags with `@`. The inspector skips anything tagged "hidden",17// which is how a component keeps bookkeeping out of a tool without the tool18// knowing that component exists.19//20// Reading a field's bytes as an f32, a u32 or an i32, writing an f32 into them,21// printing them as text and copying a C string into a fixed array are the22// unsafe procs of this lesson; each checks the length it is given. The walk23// over the reflect records is checked: `reflect_field_at` and24// `reflect_field_info` return `option<*const>`, since a record can run out of fields25// and a builtin has no record of its own.26 27import "../../includes/kayan.rin"28 29vec2: struct = {30 x: f32;31 y: f32;32}33 34side: enum = {35 red,36 blue,37}38 39component: enum = {40 position,41 health,42 team,43 weapon,44 label,45}46 47position: struct = { at: vec2; heading: f32; }48health: struct = { points: f32; max: f32; regen_timer: f32 @ "hidden"; }49team: struct = { side: side; }50weapon: struct = { damage: f32; cooldown: f32; ammo: u32; }51label: struct = { text: [16]c8; }52 53component_of: proc<position>() -> component = { return component.position; }54component_of: proc<health>() -> component = { return component.health; }55component_of: proc<team>() -> component = { return component.team; }56component_of: proc<weapon>() -> component = { return component.weapon; }57component_of: proc<label>() -> component = { return component.label; }58 59register_components: proc(w: *kayan_world) -> void = {60 kayan_register(w, kayan_id_of<position>(), position<>&);61 kayan_register(w, kayan_id_of<health>(), health<>&);62 kayan_register(w, kayan_id_of<team>(), team<>&);63 kayan_register(w, kayan_id_of<weapon>(), weapon<>&);64 kayan_register(w, kayan_id_of<label>(), label<>&);65}66 67indent: proc(depth: u32) -> void = {68 for (i: u32 = 0; i < depth; i += 1) {69 printfmt(" ");70 }71}72 73// It takes `*const` because it only reads.74is_type: proc(field: *const reflect_field, name: ^const char) -> b32 = {75 return reflect_cstr_equal(field*.type, name);76}77 78// std's reflect_field_is_array, with the same `*const` promise.79is_array: proc(field: *const reflect_field) -> b32 = {80 return field*.kind == reflect_field_kind_array || field*.array_count > 0;81}82 83// Bytes read as a builtin, or written as one, through a local: the bytes may sit84// at any address, and a local is aligned. Each checks that the bytes are at least85// that wide.86bytes_as_f32: proc(at: []const u8) -> f32 = {87 unsafe {88 if (at.length < sizeof(f32)) { printfmt("inspector: too few bytes for an f32\n"); exit(1); }89 value: f32 = 0.0;90 memcpy(value&, at.data, sizeof(f32));91 return value;92 }93}94bytes_as_u32: proc(at: []const u8) -> u32 = {95 unsafe {96 if (at.length < sizeof(u32)) { printfmt("inspector: too few bytes for a u32\n"); exit(1); }97 value: u32 = 0;98 memcpy(value&, at.data, sizeof(u32));99 return value;100 }101}102bytes_as_i32: proc(at: []const u8) -> i32 = {103 unsafe {104 if (at.length < sizeof(i32)) { printfmt("inspector: too few bytes for an i32\n"); exit(1); }105 value: i32 = 0;106 memcpy(value&, at.data, sizeof(i32));107 return value;108 }109}110bytes_set_f32: proc(at: []u8, value: f32) -> void = {111 unsafe {112 if (at.length < sizeof(f32)) { printfmt("inspector: too few bytes for an f32\n"); exit(1); }113 memcpy(at.data, value&, sizeof(f32));114 }115}116// Up to `at.length` bytes of text, printed as they are: a [N]c8 field.117print_text: proc(at: []const u8) -> void = {118 unsafe {119 printf("%.*s", cast(at.length, i32), cast(at.data, ^const c8));120 }121}122// A C string into a fixed array: up to dst.length - 1 characters, the rest123// zero, so a shorter label does not keep the tail of a longer one.124cstr_into: proc(dst: []c8, text: ^const c8) -> void = {125 unsafe {126 for (i: u64 = 0; i < dst.length; i += 1) { dst[i] = 0; }127 for (i: u64 = 0; i + 1 < dst.length && text[i] != 0; i += 1) {128 dst[i] = text[i];129 }130 }131}132 133// One value of a builtin type.134print_builtin: proc(type_name: ^const char, at: []const u8) -> void = {135 if (reflect_cstr_equal(type_name, "f32")) {136 printfmt("{:.2}", cast(bytes_as_f32(at), f64));137 } else if (reflect_cstr_equal(type_name, "u32")) {138 printfmt("{}", bytes_as_u32(at));139 } else if (reflect_cstr_equal(type_name, "i32")) {140 printfmt("{}", bytes_as_i32(at));141 } else {142 printfmt("<{}>", type_name);143 }144}145 146// Every field of a struct whose bytes are `base`, recursing into struct147// fields. Enum fields print their member name; a [N]c8 prints as text. Each148// field's bytes are cut from `base` by its offset and size, so a record that149// disagreed with the data would stop the program in Debug.150print_fields: proc(type: *const reflect, base: []const u8, depth: u32) -> void = {151 for (i: u64 = 0; i < type*.count; i += 1) {152 f: option<*const reflect_field> = reflect_field_at(type, i);153 if (!f || reflect_field_has_attr(f, "hidden")) {154 continue;155 }156 at: []const u8 = base[f*.offset..f*.offset + f*.size];157 indent(depth);158 printfmt("{}:", f*.name);159 info: option<*const reflect> = reflect_field_info(f);160 if (is_array(f) && reflect_cstr_equal(f*.elem_type, "c8")) {161 printfmt(" \"");162 print_text(at);163 printfmt("\"\n");164 } else if (info && info*.kind == reflect_kind_struct) {165 printfmt("\n");166 print_fields(info, at, depth + 1);167 } else if (info && info*.kind == reflect_kind_enum) {168 value: i32 = bytes_as_i32(at);169 printfmt(" {}\n", reflect_name_from_value_or(info, value, "(not a member)"));170 } else {171 printfmt(" ");172 print_builtin(f*.type, at);173 printfmt("\n");174 }175 }176}177 178inspect: proc(w: *kayan_world, name: ^const char, e: kayan_entity) -> void = {179 printfmt("{} ({}:{})\n", name, e.index, e.generation);180 signature: u64 = kayan_signature(w, e);181 for (id: u32 = 0; id < w*.component_count; id += 1) {182 if ((signature & kayan_bit(id)) != 0) {183 info: *const reflect in permanent = kayan_info(w, id);184 printfmt(" {}\n", info*.name);185 // A local first: widening a call's []u8 to []const u8 reads it twice.186 bytes: []u8 = kayan_get_bytes(w, e, id);187 print_fields(info, bytes, 2);188 }189 }190}191 192// What each table holds: per component, its size, and its column's bytes in use193// out of the bytes allocated for the table's capacity.194print_layout: proc(w: *kayan_world) -> void = {195 for (ti: u64 = 0; ti < w*.tables*.length; ti += 1) {196 t: *kayan_table = vec<kayan_table>at(w*.tables, ti);197 printfmt("table {} ", cast(ti, i32));198 kayan_print_signature(w, t*.signature);199 printfmt(", rows in use: {} of {}\n", cast(t*.count, i32), cast(t*.capacity, i32));200 for (id: u32 = 0; id < w*.component_count; id += 1) {201 if (kayan_table_has(t, id)) {202 info: *const reflect in permanent = kayan_info(w, id);203 printfmt(" {:-8} {:2} bytes, align {}, column {} of {} bytes\n", info*.name,204 cast(info*.size, i32), cast(info*.align, i32),205 cast(info*.size * t*.count, i32), cast(info*.size * t*.capacity, i32));206 }207 }208 }209}210 211// The editor's write: set an f32 field by component and field name. Slow and212// flexible, which is right for a tool and wrong for a system.213set_f32_by_name: proc(w: *kayan_world, e: kayan_entity, component_name: ^const char,214 field_name: ^const char, value: f32) -> b32 = {215 for (id: u32 = 0; id < w*.component_count; id += 1) {216 info: *const reflect in permanent = kayan_info(w, id);217 if (!reflect_cstr_equal(info*.name, component_name)) {218 continue;219 }220 bytes: []u8 = kayan_get_bytes(w, e, id);221 f: option<*const reflect_field> = reflect_find_field(info, field_name);222 if (bytes.length == 0 || !f || !is_type(f, "f32")) {223 return 0;224 }225 bytes_set_f32(bytes[f*.offset..f*.offset + f*.size], value);226 return 1;227 }228 return 0;229}230 231// Up to 15 characters, the rest of the array zeroed, so a shorter label does232// not keep the tail of a longer one.233set_label: proc(w: *kayan_world, e: kayan_entity, text: ^const c8) -> void = {234 l: option<*label> = kayan_get<label>(w, e);235 if (!l) {236 return;237 }238 cstr_into(l*.text, text);239}240 241main: proc() -> i32 = {242 level: region = {243 world: *kayan_world in level = kayan_world_make(level);244 register_components(world);245 246 soldier: kayan_entity = kayan_spawn(world, kayan_mask_of<position>() | kayan_mask_of<health>()247 | kayan_mask_of<team>() | kayan_mask_of<weapon>() | kayan_mask_of<label>());248 _ = kayan_set<position>(world, soldier, { .at = { .x = 3.0, .y = 4.0 }, .heading = 90.0 });249 _ = kayan_set<health>(world, soldier, { .points = 80.0, .max = 100.0, .regen_timer = 2.5 });250 _ = kayan_set<team>(world, soldier, { .side = side.blue });251 _ = kayan_set<weapon>(world, soldier, { .damage = 12.5, .cooldown = 0.25, .ammo = 30 });252 set_label(world, soldier, "soldier");253 254 crate: kayan_entity = kayan_spawn(world, kayan_mask_of<position>() | kayan_mask_of<label>());255 _ = kayan_set<position>(world, crate, { .at = { .x = -1.0, .y = 0.5 } });256 set_label(world, crate, "crate");257 258 inspect(world, "soldier", soldier);259 inspect(world, "crate", crate);260 261 printfmt("\nthe editor writes three fields by name\n");262 printfmt(" soldier health.points = 55: {}\n",263 set_f32_by_name(world, soldier, "health", "points", 55.0) ? "written" : "refused");264 printfmt(" soldier weapon.ammo = 1: {}, ammo is a u32, not an f32\n",265 set_f32_by_name(world, soldier, "weapon", "ammo", 1.0) ? "written" : "refused");266 printfmt(" crate health.points = 1: {}, the crate has no health\n",267 set_f32_by_name(world, crate, "health", "points", 1.0) ? "written" : "refused");268 inspect(world, "soldier", soldier);269 270 printfmt("\n");271 print_layout(world);272 }273 return 0;274}What inspector prints. tools/check.py fails the lesson if a single byte differs.
soldier (0:1)
position
at:
x: 3.00
y: 4.00
heading: 90.00
health
points: 80.00
max: 100.00
team
side: blue
weapon
damage: 12.50
cooldown: 0.25
ammo: 30
label
text: "soldier"
crate (1:1)
position
at:
x: -1.00
y: 0.50
heading: 0.00
label
text: "crate"
the editor writes three fields by name
soldier health.points = 55: written
soldier weapon.ammo = 1: refused, ammo is a u32, not an f32
crate health.points = 1: refused, the crate has no health
soldier (0:1)
position
at:
x: 3.00
y: 4.00
heading: 90.00
health
points: 55.00
max: 100.00
team
side: blue
weapon
damage: 12.50
cooldown: 0.25
ammo: 30
label
text: "soldier"
table 0 {position health team weapon label}, rows in use: 1 of 16
position 12 bytes, align 4, column 12 of 192 bytes
health 12 bytes, align 4, column 12 of 192 bytes
team 4 bytes, align 4, column 4 of 64 bytes
weapon 12 bytes, align 4, column 12 of 192 bytes
label 16 bytes, align 1, column 16 of 256 bytes
table 1 {position label}, rows in use: 1 of 16
position 12 bytes, align 4, column 12 of 192 bytes
label 16 bytes, align 1, column 16 of 256 bytes
Advanced · lesson 6
kayan
src/6.kayan/kayan.rin
src/6.kayan/kayan.rin
In the dojo: 5. a command buffer, 7. an arena, each on its own, to watch and to practise.
A turret at the centre, five wings of three drones, 140 frames. Seven systems run each frame, and every structural change they want goes through a command buffer applied at the end of the frame.
The frame
spawner wings of drones enter records spawns
turret shoot at the nearest drone records spawns
movement position += velocity * dt writes
lifetime bullets expire records despawns
collision live bullets against drones writes health, records despawn and add
burning burning drones lose health writes
outcome destroy dead drones, catch leaks records despawns
blast a destroyed drone hurts its neighbours writes health, records adds
apply every recorded change, in order
Writes to components happen in place, during queries. Changes to which table an entity is in are recorded, and applied once nothing is iterating.
Lifetime runs before collision, and collision skips bullets whose lifetime is used up. Those bullets are only still in their table because nothing structural happens until the end of the frame. Without the skip, a bullet that hit on its last frame would be despawned twice and counted as both a hit and a miss.
The frame is a region inside level, declared frame: region in level; and opened once per frame, around the systems and the apply:
frame: region = {
c: *commands in frame = commands_make(frame);
spawner_system(g, c);
turret_system(g, c, dt);
movement_system(g*.world, dt);
lifetime_system(g, c, dt);
collision_system(g, c);
burning_system(g*.world, dt);
outcome_system(g, c);
result: apply_result = commands_apply(c, g*.world);
skipped += result.skipped;
}
The world outlives every frame, so it is in level. The buffer belongs to one frame, so it is made inside the block and passed to the systems beside the game.
Systems and kernels
movement_system and burning_system are each a query and a kernel:
integrate: proc[unsafe](ps: noalias []position, vs: []const velocity, dt: f32) -> void
burn: proc[unsafe](hs: noalias []health, bs: []const burning, dt: f32) -> void
The system walks the tables and calls the kernel once for each table that matches:
unsafe { integrate(kayan_column<position>(t), kayan_column<velocity>(t), dt); }
The kernel sees two columns, so the compiler sees a straight loop over two arrays. noalias qualifies parameters, so the loop needs a proc of its own to carry it, and since noalias is a promise the checker cannot keep, that proc is proc[unsafe]. The split is what keeps the promise small: the query around it stays checked. The output does not change.
With the bounds checks compiled out, gcc -O3 vectorized both loops behind a run-time check that the columns were apart before the split, and without the check after it. With the checks in, it vectorizes neither here, as lesson 1 says of gcc. clang on Linux had no check either way, for the reason lesson 3 gives: the columns are different types. clang-cl, which does not use C's type rules, needs noalias here too.
burn is the cleanest loop in the game. health and burning are each one f32, so their columns are plain runs of floats, and the kernel is eight floats per pass with nothing to rearrange: loads, mulps, subps, stores. integrate works on vec2s, both coordinates get the same math, and it is just as clean at four vec2s per pass; lesson 1 shows what a vec3 does to that, and what the fully split layout looks like.
integrate and burn are streams: every row of every matching table, in order. collision_system and blast_system are closer to lookups: for each bullet or each blast, they visit drones that could be anywhere. Lesson 7 measures what each layout costs for both kinds of loop, and why the components here stay small structs.
lifetime_system stays one proc. Its loop records a despawn for each expired bullet, and a loop that calls something per row is not vectorized whatever its parameters say. The other systems branch, search or record in their loops too.
Why not change tables during a query
Each of these would be a real bug with the world from lesson 4:
- Despawn. The swap-remove moves the last row into the hole, behind the loop, so that entity is not processed this frame. Meanwhile the slice the system cut before the despawn still reaches the old last row, a stale copy, and anything written to it is lost.
- Spawn. A table that is full grows, and its columns move to new memory. A system holding a
[]positionfrom before the spawn keeps writing to the old copy. Nothing crashes; the writes are just lost. - Add or remove. The entity moves to another table, which the same query may reach later. That entity is processed twice this frame.
The checker catches the stale slices, the first two. It reads the body of a checked proc to see what the proc can move. kayan_spawn and kayan_despawn can grow or shrink what the world holds, so a column held across one is an error at compile time. Put a kayan_spawn or a kayan_despawn in lifetime_system's loop and rin says:
'ls' may be stale: 'g.world.tables', which it was taken from, has been changed since; take it again
blast_system and nearest_drone take the game or the world, but nothing in them can move a table, so a column stays good across them. turret_system and outcome_system take their columns again at the top of every pass all the same, which is more than the checker asks today.
It does not catch the rest. Take the columns again as it says, and the loop compiles, and you are back to visiting an entity twice or not at all, or to a bounds check stopping the program: a wrong answer or a stop, not a memory error. The checker rules out the lost write. The design rule, that nothing changes structure during a query, is still yours.
The buffer
command: struct = { kind; entity; pending; signature; id; payload; }
Spawned entities have no handle until the buffer runs, so commands_spawn returns a pending number, and commands_set_spawned<T> writes a component into "the entity that pending spawn will create". T has to be in the spawn's signature, and applying the buffer stops the program with a message if it is not. Payloads are copies of the component value, kept as []u8 in frame next to the commands, and the commands themselves are *vec<command> in frame. commands_apply runs everything in order. Nothing clears anything: the frame region ends at its closing brace in main, and every command and payload go with it.
So a command cannot outlive its frame, by construction. Try to keep the buffer in the game, in a field that says level, and rin refuses:
this points into region 'frame', which may end before the memory in region 'level' that holds it
Copying a component into bytes is the one unsafe act here, in commands_payload. commands_make keeps the region's arena, and commands_spawned takes a block from it. Everything else is checked. write_component copies a payload out with slice<u8>copy, so both ends are slices and the copy cannot run off either. The systems are checked too, but for their kernels' noalias.
Skipped commands
Frame 113 is the interesting one. A bullet destroys one drone, and its blast takes two burning neighbours below zero in the same loop. Each of them is destroyed in turn, and each blast reaches back to drones already recorded as destroyed:
bullet 8:9 hits drone 6:1, health -13.2
drone 6:1 is destroyed
drone 7:1 is caught in the blast, health -14.4
drone 4:2 is caught in the blast, health -4.8
drone 7:1 is destroyed
drone 6:1 is caught in the blast, health -54.0
...
The blast cannot know 6:1 is doomed: its despawn is only recorded, and it is still in its table. So it records an add<burning> against it. When the buffer is applied the despawn comes first, and kayan_move reports kayan_status.dead for the add, so the command is skipped. The summary counts three.
Try
- Call
kayan_despawndirectly inoutcome_systeminstead of recording it. It compiles, because that loop takes its columns again each pass. Then the program stops at frame 56 withindex 1 out of bounds (length 1). Explain why, from the swap-remove anddrones_here, the count the loop took before it started. - Make
blast_systemskip neighbours whose health is already at or below zero. The skip count drops to zero. Is the buffer's check still needed? - Give the turret a
healthcomponent and let leaking drones damage it. When it reaches zero, despawn it through the buffer and end the game. - Print the buffer before applying it, as a log of every structural change the frame made.
- Inline
integrateback intomovement_system, build with gcc-O3 -fopt-info-vec, and find the loop it now versions for aliasing. It needs nounsafenow: withoutnoalias, the checker can vouch for all of it. - Hold a column across a
kayan_spawnthat grows its table, and read the message. Then translate it withRIN_UNSAFE=1, which turns the rules off, and build it with AddressSanitizer, in a VM, since it misuses memory. AddressSanitizer says nothing, because the old memory is still valid memory in the region: a write through the held column lands there, and the table never sees it. That is why this is a rule in the checker: no sanitizer would have found it.
Imports the world from includes/kayan.rin.
1// 6. kayan2//3// Everything so far, in one small game. A turret at the centre shoots at wings4// of drones flying in from the edges. Bullets hurt drones, a badly hurt drone5// catches fire, and a drone at zero health is destroyed in a blast that hurts6// and ignites the drones around it. A drone that reaches the turret is a leak.7//8// The new idea is when structural changes happen. Spawning, despawning, adding9// and removing components all change tables, and systems are in the middle of10// walking tables when they decide to make those changes:11//12// * a despawn swap-removes: the last row moves into the hole, behind the13// loop, so that entity is not processed this frame, while a slice cut14// before the despawn still reaches the old last row, a stale copy whose15// writes are lost;16// * a spawn can grow a table, which moves every column of it to new memory,17// while the system still holds a slice of the old one;18// * an add or a remove moves an entity into another table, which the same19// query may walk later, so the entity is visited twice.20//21// So systems do not make structural changes. They record them in a command22// buffer, and the frame applies the buffer once every system has run. Plain23// writes to components are fine during a query: a hit lowers a drone's health24// in place. Only changes to which table an entity is in wait for the end of25// the frame.26//27// The buffer lives in a frame region. Commands and their payloads are pushed28// into it during the frame and the whole region ends after it is applied, so29// nothing in it is freed one at a time. A command against an entity that30// died earlier in the same buffer is skipped, which the handle's generation31// check makes cheap to detect.32//33// The frame arena is now the region `frame`, opened once per34// game frame around the systems and the apply; the command buffer is made in35// it and cannot outlive it, which the checker enforces. The buffer holds a36// payload's bytes, and copying a component into bytes is the one unsafe act37// here, in commands_payload; commands_make keeps the region's arena, and38// commands_spawned takes a block from it. The systems are checked but for their39// kernels' `noalias`, and the checker's one demand on them is worth seeing: a column40// slice is dead after a call that could move the table, so a system that41// calls out while walking columns takes them again afterwards.42 43cinclude "math.h"44import "../../includes/kayan.rin"45import "std/slice.rin"46 47sqrtf: proc[external](x: f32) -> f32 = {}48 49vec2: struct = {50 x: f32;51 y: f32;52}53 54component: enum = {55 position,56 velocity,57 health,58 damage,59 lifetime,60 turret,61 burning,62}63 64position: struct = { at: vec2; }65velocity: struct = { per_second: vec2; }66health: struct = { points: f32; }67damage: struct = { amount: f32; }68lifetime: struct = { seconds: f32; }69turret: struct = { cooldown: f32; timer: f32; }70burning: struct = { per_second: f32; }71 72component_of: proc<position>() -> component = { return component.position; }73component_of: proc<velocity>() -> component = { return component.velocity; }74component_of: proc<health>() -> component = { return component.health; }75component_of: proc<damage>() -> component = { return component.damage; }76component_of: proc<lifetime>() -> component = { return component.lifetime; }77component_of: proc<turret>() -> component = { return component.turret; }78component_of: proc<burning>() -> component = { return component.burning; }79 80register_components: proc(w: *kayan_world) -> void = {81 kayan_register(w, kayan_id_of<position>(), position<>&);82 kayan_register(w, kayan_id_of<velocity>(), velocity<>&);83 kayan_register(w, kayan_id_of<health>(), health<>&);84 kayan_register(w, kayan_id_of<damage>(), damage<>&);85 kayan_register(w, kayan_id_of<lifetime>(), lifetime<>&);86 kayan_register(w, kayan_id_of<turret>(), turret<>&);87 kayan_register(w, kayan_id_of<burning>(), burning<>&);88}89 90// ---------------------------------------------------------------------------91// The command buffer92// ---------------------------------------------------------------------------93 94command_kind: enum = {95 spawn,96 set_spawned,97 add,98 remove,99 despawn,100}101 102// One recorded change. `pending` names an entity spawned by this same buffer,103// which has no handle until the buffer is applied. The payload's bytes live in104// the frame, like the command itself.105command: struct = {106 kind: command_kind;107 entity: kayan_entity;108 pending: u32;109 signature: u64;110 id: u32;111 payload: []u8 in frame;112}113 114commands: struct = {115 // The frame's arena, for the payload copies; set by commands_make.116 arena: ^memops_arena;117 list: *vec<command> in frame;118 spawned: u32;119}120 121apply_result: struct = {122 applied: u32;123 skipped: u32;124}125 126// A buffer in the frame region. Trusted: it holds the arena as a raw pointer.127commands_make: proc(r: region frame) -> *commands in frame = {128 unsafe {129 c: *commands in frame = new<commands>(r, { .list = vec<command>make(r, 64) });130 c*.arena = r;131 return c;132 }133}134 135commands_record: proc(c: *commands, cmd: command) -> void = {136 _ = vec<command>push(c*.list, cmd);137}138 139// A copy of `value` in the frame region, kept until the buffer is applied, as140// the bytes the component's column will receive. Trusted: the view of a T as141// bytes.142commands_payload: proc<T>(c: *commands, value: T) -> []u8 in frame = {143 unsafe {144 copy: *T = cast(memops_arena_push(c*.arena[0]&, sizeof(T), alignof(T)), *T);145 copy* = value;146 bytes: ^u8 = cast(copy, ^u8);147 return cast(bytes[0..sizeof(T)], []u8 in frame);148 }149}150 151// Trusted: it counts in a field of the buffer, which the checker reserves for152// the struct's own procs.153commands_spawn: proc(c: *commands, signature: u64) -> u32 = {154 cmd: command = { .kind = command_kind.spawn, .signature = signature, .pending = c*.spawned };155 c*.spawned += 1;156 commands_record(c, cmd);157 return cmd.pending;158}159 160// Room for the handles of this frame's spawns, in the frame. Trusted: a raw161// push.162commands_spawned: proc(c: *commands) -> []kayan_entity in frame = {163 unsafe {164 raw: ^kayan_entity = cast(memops_arena_push_zero(c*.arena[0]&,165 sizeof(kayan_entity) * cast(c*.spawned, u64), alignof(kayan_entity)), ^kayan_entity);166 return cast(raw[0..cast(c*.spawned, u64)], []kayan_entity in frame);167 }168}169 170// T must be in the pending spawn's signature: the spawn decides the table.171commands_set_spawned: proc<T>(c: *commands, pending: u32, value: T) -> void = {172 commands_record(c, { .kind = command_kind.set_spawned, .pending = pending,173 .id = kayan_id_of<T>(), .payload = commands_payload<T>(c, value) });174}175 176commands_add: proc<T>(c: *commands, e: kayan_entity, value: T) -> void = {177 commands_record(c, { .kind = command_kind.add, .entity = e,178 .id = kayan_id_of<T>(), .payload = commands_payload<T>(c, value) });179}180 181commands_remove: proc<T>(c: *commands, e: kayan_entity) -> void = {182 commands_record(c, { .kind = command_kind.remove, .entity = e, .id = kayan_id_of<T>() });183}184 185commands_despawn: proc(c: *commands, e: kayan_entity) -> void = {186 commands_record(c, { .kind = command_kind.despawn, .entity = e });187}188 189// Copy a payload into e's component. Returns 0 if e has no such component.190// Both ends are slices, so the copy cannot run off either.191write_component: proc(w: *kayan_world, e: kayan_entity, id: u32, payload: []const u8) -> b32 = {192 target: []u8 = kayan_get_bytes(w, e, id);193 if (target.length == 0) {194 return 0;195 }196 if (payload.length < target.length) {197 printfmt("write_component: payload of {} bytes for a component of {}\n", payload.length, target.length);198 exit(1);199 }200 slice<u8>copy(target, payload[0..target.length]);201 return 1;202}203 204// Every command, in the order it was recorded. The frame region ends right205// after, in main, which frees every command and payload at once.206commands_apply: proc(c: *commands, w: *kayan_world) -> apply_result = {207 result: apply_result = {};208 spawned: []kayan_entity in frame = commands_spawned(c);209 list: []command = vec<command>items(c*.list);210 for (i: u64 = 0; i < list.length; i += 1) {211 cmd: command = list[i];212 switch (cmd.kind) {213 case command_kind.spawn: {214 spawned[cmd.pending] = kayan_spawn(w, cmd.signature);215 result.applied += 1;216 }217 case command_kind.set_spawned: {218 if (!write_component(w, spawned[cmd.pending], cmd.id, cmd.payload)) {219 printfmt("set_spawned: spawn {} has no {} in its signature\n", cmd.pending,220 kayan_component_name(w, cmd.id));221 exit(1);222 }223 result.applied += 1;224 }225 case command_kind.add: {226 signature: u64 = kayan_signature(w, cmd.entity) | kayan_bit(cmd.id);227 if (kayan_move(w, cmd.entity, signature) == kayan_status.ok) {228 _ = write_component(w, cmd.entity, cmd.id, cmd.payload);229 result.applied += 1;230 } else {231 result.skipped += 1;232 }233 }234 case command_kind.remove: {235 signature: u64 = kayan_signature(w, cmd.entity) & ~kayan_bit(cmd.id);236 if (kayan_move(w, cmd.entity, signature) == kayan_status.ok) {237 result.applied += 1;238 } else {239 result.skipped += 1;240 }241 }242 case command_kind.despawn: {243 if (kayan_despawn(w, cmd.entity) == kayan_status.ok) {244 result.applied += 1;245 } else {246 result.skipped += 1;247 }248 }249 }250 }251 return result;252}253 254// ---------------------------------------------------------------------------255// The game256// ---------------------------------------------------------------------------257 258stats: struct = {259 waves: u32;260 drones: u32;261 destroyed: u32;262 leaked: u32;263 shots: u32;264 hits: u32;265 misses: u32;266}267 268// The world outlives every frame, so it is in `level`; the command buffer is269// a frame's and is passed to the systems beside the game.270game: struct = {271 world: *kayan_world in level;272 stats: stats;273 frame: i32;274 printed_frame: i32;275}276 277drone_mask: proc() -> u64 = {278 return kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<health>();279}280 281bullet_mask: proc() -> u64 = {282 return kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<damage>()283 | kayan_mask_of<lifetime>();284}285 286// Each frame's events are printed under one "frame N" line. It takes the two287// fields it uses rather than the game, which is all it needs.288event: proc(frame_no: i32, printed_frame: *i32) -> void = {289 if (printed_frame* != frame_no) {290 printfmt("frame {}\n", frame_no);291 printed_frame* = frame_no;292 }293 printfmt(" ");294}295 296distance: proc(a: vec2, b: vec2) -> f32 = {297 dx: f32 = a.x - b.x;298 dy: f32 = a.y - b.y;299 return sqrtf(dx * dx + dy * dy);300}301 302toward: proc(from: vec2, to: vec2, speed: f32) -> vec2 = {303 d: f32 = distance(from, to);304 return { .x = (to.x - from.x) / d * speed, .y = (to.y - from.y) / d * speed };305}306 307// Drones come in wings of three, side by side, from each side in turn.308entry_points: [4]vec2 = {309 { .x = 20.0, .y = 0.0 }, { .x = 0.0, .y = -20.0 }, { .x = -20.0, .y = 0.0 }, { .x = 0.0, .y = 20.0 },310};311wave_every: i32 = 20;312waves: u32 = 5;313drone_speed: f32 = 2.0;314 315spawner_system: proc(g: *game, c: *commands) -> void = {316 if ((g*.frame - 1) % wave_every != 0 || g*.stats.waves == waves) {317 return;318 }319 centre: vec2 = entry_points[g*.stats.waves % 4];320 heading: vec2 = toward(centre, { .x = 0.0, .y = 0.0 }, 1.0);321 for (k: i32 = -1; k <= 1; k += 1) {322 // Side by side: 2 units apart, across the direction of flight.323 side: f32 = 2.0 * cast(k, f32);324 at: vec2 = { .x = centre.x - heading.y * side, .y = centre.y + heading.x * side };325 drone: u32 = commands_spawn(c, drone_mask());326 commands_set_spawned<position>(c, drone, { .at = at });327 commands_set_spawned<velocity>(c, drone,328 { .per_second = toward(at, { .x = 0.0, .y = 0.0 }, drone_speed) });329 commands_set_spawned<health>(c, drone, { .points = 100.0 });330 g*.stats.drones += 1;331 }332 g*.stats.waves += 1;333 event(g*.frame, g*.printed_frame&);334 printfmt("a wing of three drones enters at ({:.1}, {:.1})\n", cast(centre.x, f64), cast(centre.y, f64));335}336 337// The nearest drone to `from`, by walking every table a drone can be in.338nearest_drone: proc(w: *kayan_world, from: vec2, found: *vec2) -> b32 = {339 q: kayan_query = { .all = drone_mask() };340 best: f32 = 1000000.0;341 any: b32 = 0;342 for (ti: u64 = 0; ti < w*.tables*.length; ti += 1) {343 t: *kayan_table = vec<kayan_table>at(w*.tables, ti);344 if (!kayan_matches(t, q)) {345 continue;346 }347 ps: []position = kayan_column<position>(t);348 for (i: u64 = 0; i < ps.length; i += 1) {349 d: f32 = distance(from, ps[i].at);350 if (d < best) {351 best = d;352 found* = ps[i].at;353 any = 1;354 }355 }356 }357 return any;358}359 360turret_system: proc(g: *game, c: *commands, dt: f32) -> void = {361 w: *kayan_world in level = g*.world;362 q: kayan_query = { .all = kayan_mask_of<position>() | kayan_mask_of<turret>() };363 for (ti: u64 = 0; ti < w*.tables*.length; ti += 1) {364 t: *kayan_table = vec<kayan_table>at(w*.tables, ti);365 if (!kayan_matches(t, q)) {366 continue;367 }368 turrets: u64 = t*.count;369 for (i: u64 = 0; i < turrets; i += 1) {370 // nearest_drone walks the world. The table and its columns are taken371 // at the top of every pass, more often than the checker asks, so a372 // reader need not work out what each call may move.373 t = vec<kayan_table>at(w*.tables, ti);374 ps: []position = kayan_column<position>(t);375 ts: []turret = kayan_column<turret>(t);376 // The timer stops at zero, so an idle turret is ready to fire but377 // does not save up shots.378 ts[i].timer -= dt;379 if (ts[i].timer < 0.0) {380 ts[i].timer = 0.0;381 }382 target: vec2 = {};383 if (ts[i].timer > 0.0 || !nearest_drone(w, ps[i].at, target&)) {384 continue;385 }386 // And again after the walk, before the write.387 t = vec<kayan_table>at(w*.tables, ti);388 ps = kayan_column<position>(t);389 ts = kayan_column<turret>(t);390 ts[i].timer += ts[i].cooldown;391 bullet: u32 = commands_spawn(c, bullet_mask());392 commands_set_spawned<position>(c, bullet, ps[i]);393 commands_set_spawned<velocity>(c, bullet, { .per_second = toward(ps[i].at, target, 12.0) });394 commands_set_spawned<damage>(c, bullet, { .amount = 30.0 });395 commands_set_spawned<lifetime>(c, bullet, { .seconds = 2.0 });396 g*.stats.shots += 1;397 }398 }399}400 401// A system is a query and a kernel. The query finds the tables; the kernel runs402// over one table's columns. noalias is a parameter qualifier, so the loop has403// to be in a proc of its own to get it.404integrate: proc[unsafe](ps: noalias []position, vs: []const velocity, dt: f32) -> void = {405 for (i: u64 = 0; i < ps.length; i += 1) {406 ps[i].at.x += vs[i].per_second.x * dt;407 ps[i].at.y += vs[i].per_second.y * dt;408 }409}410 411movement_system: proc(w: *kayan_world, dt: f32) -> void = {412 q: kayan_query = { .all = kayan_mask_of<position>() | kayan_mask_of<velocity>() };413 for (ti: u64 = 0; ti < w*.tables*.length; ti += 1) {414 t: *kayan_table = vec<kayan_table>at(w*.tables, ti);415 if (kayan_matches(t, q)) {416 unsafe { integrate(kayan_column<position>(t), kayan_column<velocity>(t), dt); }417 }418 }419}420 421// Every bullet against every drone. A hit lowers health in place, which is not422// a structural change; removing the bullet and setting the drone on fire are.423// A bullet that expired this frame is skipped: lifetime_system has already424// recorded its despawn, and it is only still here because nothing structural425// happens until the end of the frame.426collision_system: proc(g: *game, c: *commands) -> void = {427 w: *kayan_world in level = g*.world;428 bullets: kayan_query = { .all = bullet_mask() };429 drones: kayan_query = { .all = drone_mask() };430 for (bi: u64 = 0; bi < w*.tables*.length; bi += 1) {431 bt: *kayan_table = vec<kayan_table>at(w*.tables, bi);432 if (!kayan_matches(bt, bullets)) {433 continue;434 }435 bullet_at: []position = kayan_column<position>(bt);436 bullet_damage: []damage = kayan_column<damage>(bt);437 bullet_life: []lifetime = kayan_column<lifetime>(bt);438 bullet_ids: []const kayan_entity = kayan_table_entities(bt);439 for (b: u64 = 0; b < bullet_at.length; b += 1) {440 if (bullet_life[b].seconds <= 0.0) {441 continue;442 }443 hit: b32 = 0;444 for (di: u64 = 0; di < w*.tables*.length && !hit; di += 1) {445 drone_table: *kayan_table = vec<kayan_table>at(w*.tables, di);446 if (!kayan_matches(drone_table, drones)) {447 continue;448 }449 drone_at: []position = kayan_column<position>(drone_table);450 drone_health: []health = kayan_column<health>(drone_table);451 drone_ids: []const kayan_entity = kayan_table_entities(drone_table);452 for (d: u64 = 0; d < drone_at.length && !hit; d += 1) {453 if (distance(bullet_at[b].at, drone_at[d].at) > 1.0 || drone_health[d].points <= 0.0) {454 continue;455 }456 hit = 1;457 before: f32 = drone_health[d].points;458 drone_health[d].points -= bullet_damage[b].amount;459 g*.stats.hits += 1;460 bullet: kayan_entity = bullet_ids[b];461 drone: kayan_entity = drone_ids[d];462 commands_despawn(c, bullet);463 health_now: f32 = drone_health[d].points;464 event(g*.frame, g*.printed_frame&);465 printfmt("bullet {}:{} hits drone {}:{}, health {:.1}\n", bullet.index,466 bullet.generation, drone.index, drone.generation, cast(health_now, f64));467 if (before >= 50.0 && drone_health[d].points < 50.0468 && (drone_table*.signature & kayan_mask_of<burning>()) == 0) {469 commands_add<burning>(c, drone, { .per_second = 8.0 });470 event(g*.frame, g*.printed_frame&);471 printfmt("drone {}:{} catches fire\n", drone.index, drone.generation);472 }473 }474 }475 }476 }477}478 479burn: proc[unsafe](hs: noalias []health, bs: []const burning, dt: f32) -> void = {480 for (i: u64 = 0; i < hs.length; i += 1) {481 hs[i].points -= bs[i].per_second * dt;482 }483}484 485burning_system: proc(w: *kayan_world, dt: f32) -> void = {486 q: kayan_query = { .all = kayan_mask_of<health>() | kayan_mask_of<burning>() };487 for (ti: u64 = 0; ti < w*.tables*.length; ti += 1) {488 t: *kayan_table = vec<kayan_table>at(w*.tables, ti);489 if (kayan_matches(t, q)) {490 unsafe { burn(kayan_column<health>(t), kayan_column<burning>(t), dt); }491 }492 }493}494 495// A destroyed drone's blast hurts every drone near it and sets it on fire.496// Lowering health is a plain write, so a blast can finish off a neighbour later497// in the same loop, and that neighbour's blast reaches back to the drone that498// started it. That drone's despawn is only recorded, so it is still in its499// table and gets an add<burning> too. By the time the buffer is applied it is500// gone, and commands_apply skips the add.501blast_system: proc(g: *game, c: *commands, from: kayan_entity, at: vec2) -> void = {502 w: *kayan_world in level = g*.world;503 q: kayan_query = { .all = drone_mask() };504 for (ti: u64 = 0; ti < w*.tables*.length; ti += 1) {505 t: *kayan_table = vec<kayan_table>at(w*.tables, ti);506 if (!kayan_matches(t, q)) {507 continue;508 }509 ps: []position = kayan_column<position>(t);510 hs: []health = kayan_column<health>(t);511 ids: []const kayan_entity = kayan_table_entities(t);512 for (i: u64 = 0; i < ps.length; i += 1) {513 if (ids[i].index != from.index && distance(ps[i].at, at) < 4.0) {514 hs[i].points -= 40.0;515 hurt: kayan_entity = ids[i];516 health_now: f32 = hs[i].points;517 commands_add<burning>(c, hurt, { .per_second = 8.0 });518 event(g*.frame, g*.printed_frame&);519 printfmt("drone {}:{} is caught in the blast, health {:.1}\n", hurt.index,520 hurt.generation, cast(health_now, f64));521 }522 }523 }524}525 526// Drones at zero health are destroyed; drones that reach the turret leak.527outcome_system: proc(g: *game, c: *commands) -> void = {528 w: *kayan_world in level = g*.world;529 q: kayan_query = { .all = drone_mask() };530 for (ti: u64 = 0; ti < w*.tables*.length; ti += 1) {531 t: *kayan_table = vec<kayan_table>at(w*.tables, ti);532 if (!kayan_matches(t, q)) {533 continue;534 }535 drones_here: u64 = t*.count;536 for (i: u64 = 0; i < drones_here; i += 1) {537 // blast_system walks the world, so the table and its columns are538 // taken at the top of every pass, as in turret_system.539 t = vec<kayan_table>at(w*.tables, ti);540 ps: []position = kayan_column<position>(t);541 hs: []health = kayan_column<health>(t);542 ids: []const kayan_entity = kayan_table_entities(t);543 if (hs[i].points <= 0.0) {544 gone: kayan_entity = ids[i];545 at: vec2 = ps[i].at;546 commands_despawn(c, gone);547 g*.stats.destroyed += 1;548 event(g*.frame, g*.printed_frame&);549 printfmt("drone {}:{} is destroyed\n", gone.index, gone.generation);550 blast_system(g, c, gone, at);551 } else if (distance(ps[i].at, { .x = 0.0, .y = 0.0 }) < 1.5) {552 leaked: kayan_entity = ids[i];553 health_now: f32 = hs[i].points;554 commands_despawn(c, leaked);555 g*.stats.leaked += 1;556 event(g*.frame, g*.printed_frame&);557 printfmt("drone {}:{} reaches the turret with {:.1} health\n", leaked.index,558 leaked.generation, cast(health_now, f64));559 }560 }561 }562}563 564lifetime_system: proc(g: *game, c: *commands, dt: f32) -> void = {565 w: *kayan_world in level = g*.world;566 q: kayan_query = { .all = kayan_mask_of<lifetime>() };567 for (ti: u64 = 0; ti < w*.tables*.length; ti += 1) {568 t: *kayan_table = vec<kayan_table>at(w*.tables, ti);569 if (!kayan_matches(t, q)) {570 continue;571 }572 ls: []lifetime = kayan_column<lifetime>(t);573 ids: []const kayan_entity = kayan_table_entities(t);574 for (i: u64 = 0; i < ls.length; i += 1) {575 ls[i].seconds -= dt;576 if (ls[i].seconds <= 0.0) {577 expired: kayan_entity = ids[i];578 commands_despawn(c, expired);579 g*.stats.misses += 1;580 }581 }582 }583}584 585main: proc() -> i32 = {586 level: region = {587 g: *game in level = new<game>(level, { .world = kayan_world_make(level) });588 register_components(g*.world);589 590 tower: kayan_entity = kayan_spawn(g*.world, kayan_mask_of<position>() | kayan_mask_of<turret>());591 _ = kayan_set<turret>(g*.world, tower, { .cooldown = 0.6, .timer = 0.6 });592 593 dt: f32 = 0.1;594 skipped: u32 = 0;595 frames: i32 = 140;596 for (frame_no: i32 = 1; frame_no <= frames; frame_no += 1) {597 g*.frame = frame_no;598 // One frame region per frame: the command buffer and every payload in599 // it are gone at the closing brace, and the checker makes sure nothing600 // kept a pointer in.601 frame: region = {602 c: *commands in frame = commands_make(frame);603 spawner_system(g, c);604 turret_system(g, c, dt);605 movement_system(g*.world, dt);606 lifetime_system(g, c, dt);607 collision_system(g, c);608 burning_system(g*.world, dt);609 outcome_system(g, c);610 result: apply_result = commands_apply(c, g*.world);611 skipped += result.skipped;612 }613 }614 615 printfmt("after {} frames\n", frames);616 printfmt(" drones: {} entered, {} destroyed, {} leaked\n",617 g*.stats.drones, g*.stats.destroyed, g*.stats.leaked);618 printfmt(" bullets: {} fired, {} hit, {} expired\n", g*.stats.shots, g*.stats.hits, g*.stats.misses);619 printfmt(" commands skipped because their entity was already gone: {}\n", skipped);620 printfmt(" {} entities alive in {} tables\n", g*.world*.live, g*.world*.tables*.length);621 for (ti: u64 = 0; ti < g*.world*.tables*.length; ti += 1) {622 t: *kayan_table = vec<kayan_table>at(g*.world*.tables, ti);623 printfmt(" table {} ", cast(ti, i32));624 kayan_print_signature(g*.world, t*.signature);625 printfmt(", rows in use: {}\n", cast(t*.count, i32));626 }627 }628 return 0;629}What kayan prints. tools/check.py fails the lesson if a single byte differs.
frame 1
a wing of three drones enters at (20.0, 0.0)
frame 20
bullet 4:1 hits drone 2:1, health 70.0
frame 21
a wing of three drones enters at (0.0, -20.0)
frame 26
bullet 5:1 hits drone 2:1, health 40.0
drone 2:1 catches fire
frame 32
bullet 8:1 hits drone 2:1, health 6.0
frame 38
bullet 5:2 hits drone 2:1, health -28.8
drone 2:1 is destroyed
drone 1:1 is caught in the blast, health 60.0
drone 3:1 is caught in the blast, health 60.0
frame 41
a wing of three drones enters at (-20.0, 0.0)
frame 50
bullet 10:1 hits drone 1:1, health 21.2
frame 56
bullet 11:1 hits drone 1:1, health -13.6
drone 1:1 is destroyed
drone 3:1 is caught in the blast, health 5.6
frame 61
a wing of three drones enters at (0.0, 20.0)
frame 64
drone 3:1 is destroyed
frame 77
bullet 3:2 hits drone 6:1, health 70.0
frame 81
a wing of three drones enters at (20.0, 0.0)
frame 83
bullet 8:4 hits drone 6:1, health 40.0
drone 6:1 catches fire
frame 89
bullet 8:5 hits drone 7:1, health 70.0
frame 95
bullet 8:6 hits drone 7:1, health 40.0
drone 7:1 catches fire
frame 101
bullet 8:7 hits drone 4:2, health 70.0
frame 107
bullet 8:8 hits drone 4:2, health 40.0
drone 4:2 catches fire
frame 113
bullet 8:9 hits drone 6:1, health -13.2
drone 6:1 is destroyed
drone 7:1 is caught in the blast, health -14.4
drone 4:2 is caught in the blast, health -4.8
drone 7:1 is destroyed
drone 6:1 is caught in the blast, health -54.0
drone 4:2 is caught in the blast, health -44.8
drone 4:2 is destroyed
drone 6:1 is caught in the blast, health -94.0
drone 7:1 is caught in the blast, health -54.4
frame 122
bullet 4:3 hits drone 2:2, health 70.0
frame 128
bullet 4:4 hits drone 2:2, health 40.0
drone 2:2 catches fire
frame 134
bullet 4:5 hits drone 5:3, health 70.0
drone 5:3 reaches the turret with 70.0 health
drone 9:1 reaches the turret with 100.0 health
drone 2:2 reaches the turret with 35.2 health
after 140 frames
drones: 15 entered, 6 destroyed, 3 leaked
bullets: 20 fired, 16 hit, 3 expired
commands skipped because their entity was already gone: 3
8 entities alive in 4 tables
table 0 {position turret}, rows in use: 1
table 1 {position velocity health}, rows in use: 6
table 2 {position velocity damage lifetime}, rows in use: 1
table 3 {position velocity health burning}, rows in use: 0
Advanced · lesson 7
Layout follows access
src/7.access_patterns/access_patterns.rin
src/7.access_patterns/access_patterns.rin
Lesson 1 found the layout SIMD wants: one array per float. This lesson asks what that layout costs everywhere else. 1,048,576 bodies are stored four ways, and two kinds of loop run over each, and the program checks that all four end with the same positions.
Two kinds of loop
Stream: every body, in order, the same math. Integration is the example:
for (i: u64 = 0; i < px.length; i += 1) {
px[i] += vx[i] * dt;
...
}
The next body is always right after the last one, so the hardware prefetcher has it ready, and SIMD can take four or eight at once if they sit side by side.
Lookup: bodies picked by index, two at a time, in no particular order. Contacts are the example, the way collision and constraints reach bodies:
a: u64 = cast(cs[c].a, u64);
b: u64 = cast(cs[c].b, u64);
dx: f32 = px[b] - px[a];
...
Each body is somewhere else in memory. What matters is how many cache lines one body's data sits in, because each one is a separate trip to memory. SIMD does not help: there are no four neighbours to load at once.
Four layouts
aos [ x y z vx vy vz m r ][ x y z vx vy vz m r ] ... 32 bytes a body
columns positions [ x y z ][ x y z ][ x y z ] ...
velocities [ vx vy vz ][ vx vy vz ] ... 24 bytes a body
split px [ x x x x ... ] py [ y y y y ... ] pz ... vx ... vy ... vz ...
blocks [ x0..x7 | y0..y7 | z0..z7 | vx0..vx7 | vy0..vy7 | vz0..vz7 ][ x8..x15 ...
- aos keeps everything about a body together, two fields neither loop uses included.
- columns is one array per component, each component a small struct. This is what an ECS table stores, and what lessons 2 to 6 use.
- split is one array per float, lesson 1's
move_split. - blocks is eight bodies at a time, each coordinate a run of eight floats. It is split cut into pieces of eight, with the pieces of one body kept near each other. The usual name is AoSoA, an array of structs of arrays.
As in lesson 1, the arena is the region level, every array is a slice that names it, and each loop is a proc[unsafe] kernel so that it can take noalias, called from inside unsafe { }.
The program prints how many 64-byte cache lines one body's position sits in:
cache lines one lookup touches: aos 1.000, columns 1.125, split 3.000, blocks 2.000
A 12-byte vec3 crosses a line boundary 2 times in 16, hence 1.125. In split, x, y and z are in three different arrays. In a block, x and y share a line and z is in the next.
The program works that out from byte offsets within an array, as if each array began on a line. new_slice aligns to the element, not to the line, so nothing promises that. In this build most arrays do start on a line, positions and velocities start 32 bytes into one, and the counts from the real addresses come out the same. A promise would take an arena push with an alignment of 64, which is unsafe.
Measured
Release, clang on Windows (clang -O2 and clang-cl /O2, which make the same loops) on one core, best of at least three runs, on a Ryzen 9 9950X3D: 1 MB of L2 a core and 96 MB of L3. The body count, the contact count and the steps were changed in the source for the first and last columns. At 16,384 bodies each layout fits in L2; at the lesson's 1,048,576 each is 24 to 32 MB and sits in L3; at 8,388,608 each is 192 to 256 MB and comes mostly from RAM.
Stream, ns per body:
| layout | 16,384 | 1,048,576 | 8,388,608 |
|---|---|---|---|
| aos | 0.48 | 0.57 | 1.26 |
| columns | 0.67 | 0.62 | 0.93 |
| split | 0.33 | 0.43 | 0.76 |
| blocks | 0.14 | 0.29 | 0.88 |
Lookup, ns per contact:
| layout | 16,384 | 1,048,576 | 8,388,608 |
|---|---|---|---|
| aos | 1.05 | 1.76 | 10.20 |
| columns | 1.05 | 1.72 | 7.25 |
| split | 1.67 | 2.72 | 11.01 |
| blocks | 1.62 | 2.67 | 14.00 |
clang-cl with /arch:AVX2 agrees on the main points: split streams fastest from RAM, and columns look up fastest. So does gcc -O3 on the lookups. On the stream it does not, with the checks on: it leaves split unvectorized, as in lesson 1, and split streams slowest from RAM, at 1.86 ns. With -DRIN_NO_BOUNDS_CHECKS it is fastest again, at 0.80. Where two numbers are within about 20 percent, the order changes from build to build. The machine was busy, and everything on it shares that L3: lookups at a million bodies came out up to three times slower in unlucky runs.
What it says:
- Split wins the stream from RAM and loses the lookup. There it streams 1.2 times faster than columns, and looks up 1.5 times slower: three lines a body instead of one.
- Columns win the lookup once the data leaves the cache, and tie with AoS before that, since both have about one line a body. From RAM, columns' positions are 100 MB, about all of L3, while AoS spreads the same lookups over 256 MB. In the stream they are slowest in cache: lesson 1 showed why a
vec3fights the registers. - Blocks are the compromise, mostly in the stream. In cache they streamed fastest of all, more than twice split's speed in L2. From RAM they fall a little behind split. They look up no better than split: two lines a body instead of three did not show here.
- Out of cache, a lookup costs about ten times a streamed body. From RAM, 7 to 14 ns a contact against about 1 ns a body. Turning lookups into something closer to a stream, by sorting contacts or walking space in order, usually saves more than any layout choice.
What the compiler makes of it
clang-cl /O2, Windows x64, from the object file:
| loop | bodies a pass | instructions a pass | shuffles |
|---|---|---|---|
integrate_aos | 1 | 16 | 1 |
integrate_columns | 4 | 77 | 30 |
integrate_split | 4 | 20 | 0 |
integrate_blocks | 8 | 37 | 0 |
integrate_split, /arch:AVX2 | 8 | 14 | 0 |
integrate_blocks, /arch:AVX2 | 8 | 14 | 0 |
contacts_columns | 1 contact | 29 | 0 |
contacts_split | 1 contact | 44 | 0 |
Blocks vectorize as cleanly as split: the inner loop over eight lanes is a fixed-size loop over consecutive floats, which compilers turn into whole registers. The contact loops are one contact a pass in every layout.
Bounds checks are on in all of these, and the stream loops are as they would be without them, as lesson 1 explains. The contact loops are where the checks stay. A contact's two indexes come out of data, so nothing can prove them in range, and each index stays a compare and a branch per use: contacts_columns would be 25 instructions without them, contacts_split 32. It hardly shows in the time, since the trip to memory is most of a lookup. With -DRIN_NO_BOUNDS_CHECKS the lookup table moves by about 10 percent from RAM and not at all in cache.
In between
Engines that have both kinds of loop do one of these:
- Blocks, as above. One layout, decent at both.
- Gather, work, scatter. Storage stays in columns or AoS, good for lookups. The hot loop copies a batch of bodies into a wide layout, x's in one register and y's in another, does its math, and copies them back. The copy pays off when the same batch is worked on many times, as a constraint solver does over several iterations. Box2D v3 solves contacts this way, four or eight at a time.
- Two layouts for two jobs. A particle system that only ever streams stores split. The bodies that collision looks up stay in columns.
How an ECS decides
An ECS does not decide where each float goes. The component is the unit it stores together, and the programmer decides what a component is.
- Tables. kayan, Unity's entities, Bevy and flecs keep one table per archetype, with one column per component. That is this lesson's columns layout: an array per component, and each component's fields side by side inside it. Unity cuts its tables into 16 KB chunks, each holding every component array for as many entities as fit, which is close to blocks at the level of components.
- Sparse sets. EnTT stores each component in its own dense array with a lookup table from entity to index, and can keep several components in the same order ("groups") so a system over them streams. Bevy lets each component choose: table storage, or a sparse set for components added and removed often.
- Systems are streams. A system is a query and a kernel, as in lesson 6: the ECS finds the tables that match, and hands the kernel each table's columns. That stream over columns is the fast path every ECS is built for. Systems that say which columns they read and which they write are what lets a scheduler run them in parallel.
- Lookups go through the handle. Reaching one entity is handle, slot, table and row, then the column:
kayan_getin lesson 4. That is this lesson's lookup loop with an extra hop in front. ECS code keeps lookups for the places that need them, such as relationships between entities and collision, and finds the rest by query.
So the decisions that are yours:
- Keep components small, to what one stream system reads. A component a system reads half of is AoS's wasted bytes again.
- Leave components as small structs,
position { x, y, z }, unless a profiler shows a stream over them is the bottleneck. They are the best layout for lookups and good enough for most streams. - For a hot stream over huge counts, particles above all, store split or blocks, or gather into them for the hot loop.
- Make lookups rarer and less random before tuning their layout.
Try
- Sort
csbyabefore the timing loops and run--timeagain. Which lookup numbers change, and by how much? - Set
body_countandcontact_countto 8388608 and run--timein Release. Compare with the tables above. - Make the contacts push velocities too,
vx,vyandvzof both bodies. How many lines does a lookup touch in each layout now? - Add
/arch:AVX2tobuild_c_flagsand compare the stream column. - Write
contacts_gather: for every eight contacts, copy the sixteen bodies' positions into[8]f32arrays, do the arithmetic over the arrays, and copy back. Contacts that share a body in the same eight break it; find the case and decide what to do about it. - Make
body64 bytes with fields neither loop uses, and watch aos in both tables.
1// 7. Layout follows access2//3// Lesson 1 found the layout SIMD wants: one array per float. This lesson asks4// what that layout costs everywhere else. The same bodies are stored four ways,5// and two kinds of loop run over each:6//7// stream every body, in order, the same math: integration.8// lookup bodies picked by index, two at a time, in no order: contacts,9// the way collision and constraints reach them.10//11// The four layouts:12//13// aos []body, everything about a body together14// columns []vec3 per field, as an ECS table stores components15// split []f32 per coordinate, as lesson 1's move_split16// blocks []body_block, eight bodies per block, one [8]f32 per coordinate17//18// A stream wants consecutive floats, so SIMD can load four or eight at once.19// A lookup wants one body's fields close together, so one cache line brings20// them all. No layout is best at both; the program prints how many cache lines21// a lookup touches in each, and with `--time` how long each loop takes.22//23// As in lesson 1, the arena is the region `level` and every array24// is a slice that names it. The kernels are `proc[unsafe]`, for `noalias`, and one25// other proc has an unsafe block: `has_flag` reads argv.26 27cinclude "time.h"28import "std/print.rin"29 30clock: proc[external]() -> long = {}31CLOCKS_PER_SEC: const long;32 33level: region;34 35// argv is C's: `^^c8` with no lengths. Trusted, because strcmp reads to the36// terminator and the checker cannot see that argv's strings have one.37has_flag: proc(argc: i32, argv: ^^c8, flag: ^const c8) -> b32 = {38 unsafe {39 for (i: i32 = 1; i < argc; i += 1) {40 if (strcmp(argv[i], flag) == 0) {41 return 1;42 }43 }44 return 0;45 }46}47 48body_count: u64 = 1048576;49contact_count: u64 = 1048576;50dt: static f32 = 1.0 / 60.0;51gravity: static const f32 = -9.8;52stiffness: static const f32 = 0.01;53 54vec3: struct = {55 x: f32;56 y: f32;57 z: f32;58}59 60// aos: 32 bytes, the two fields both loops use and two they do not.61body: struct = {62 position: vec3;63 velocity: vec3;64 inverse_mass: f32;65 radius: f32;66}67 68// blocks: eight bodies, each coordinate a run of eight floats. 192 bytes.69body_block: struct = {70 px: [8]f32;71 py: [8]f32;72 pz: [8]f32;73 vx: [8]f32;74 vy: [8]f32;75 vz: [8]f32;76}77 78// Two bodies a contact pushes apart.79contact: struct = {80 a: u32;81 b: u32;82}83 84columns: struct = {85 positions: []vec3 in level;86 velocities: []vec3 in level;87}88 89split: struct = {90 px: []f32 in level;91 py: []f32 in level;92 pz: []f32 in level;93 vx: []f32 in level;94 vy: []f32 in level;95 vz: []f32 in level;96}97 98// new_slice aligns to the element. The line counts below are computed from byte99// offsets within an array, as if it started on a line, not from addresses.100 101random_state: u64 = 12345;102 103random01: proc() -> f32 = {104 random_state = random_state * 6364136223846793005'u64 + 1442695040888963407'u64;105 return cast(random_state shr 40, f32) / 16777216.0;106}107 108random_index: proc(n: u64) -> u32 = {109 random_state = random_state * 6364136223846793005'u64 + 1442695040888963407'u64;110 return cast((random_state shr 33) % n, u32);111}112 113// ---------------------------------------------------------------- stream114// Every body, in order, the same math. noalias on what each kernel writes.115 116integrate_aos: proc[unsafe](bodies: noalias []body, dt: f32) -> void = {117 for (i: u64 = 0; i < bodies.length; i += 1) {118 b: *body = bodies[i]&;119 b*.velocity.y += gravity * dt;120 b*.position.x += b*.velocity.x * dt;121 b*.position.y += b*.velocity.y * dt;122 b*.position.z += b*.velocity.z * dt;123 }124}125 126integrate_columns: proc[unsafe](positions: noalias []vec3, velocities: noalias []vec3, dt: f32) -> void = {127 for (i: u64 = 0; i < positions.length; i += 1) {128 velocities[i].y += gravity * dt;129 positions[i].x += velocities[i].x * dt;130 positions[i].y += velocities[i].y * dt;131 positions[i].z += velocities[i].z * dt;132 }133}134 135integrate_split: proc[unsafe](px: noalias []f32, py: noalias []f32, pz: noalias []f32,136 vx: []const f32, vy: noalias []f32, vz: []const f32, dt: f32) -> void = {137 for (i: u64 = 0; i < px.length; i += 1) {138 vy[i] += gravity * dt;139 px[i] += vx[i] * dt;140 py[i] += vy[i] * dt;141 pz[i] += vz[i] * dt;142 }143}144 145// The inner loop is eight floats per coordinate: one AVX register, or two SSE.146integrate_blocks: proc[unsafe](blocks: noalias []body_block, dt: f32) -> void = {147 for (b: u64 = 0; b < blocks.length; b += 1) {148 k: *body_block = blocks[b]&;149 for (l: u64 = 0; l < 8; l += 1) {150 k*.vy[l] += gravity * dt;151 k*.px[l] += k*.vx[l] * dt;152 k*.py[l] += k*.vy[l] * dt;153 k*.pz[l] += k*.vz[l] * dt;154 }155 }156}157 158// ---------------------------------------------------------------- lookup159// Each contact moves its two bodies a little toward each other. The bodies are160// anywhere in the arrays, so every contact is two jumps through memory. The161// arithmetic is the same in every layout, in the same order, so the results162// match to the bit.163 164contacts_aos: proc[unsafe](bodies: noalias []body, cs: []const contact) -> void = {165 for (c: u64 = 0; c < cs.length; c += 1) {166 p: *vec3 = bodies[cast(cs[c].a, u64)].position&;167 q: *vec3 = bodies[cast(cs[c].b, u64)].position&;168 dx: f32 = q*.x - p*.x;169 dy: f32 = q*.y - p*.y;170 dz: f32 = q*.z - p*.z;171 p*.x += dx * stiffness;172 p*.y += dy * stiffness;173 p*.z += dz * stiffness;174 q*.x -= dx * stiffness;175 q*.y -= dy * stiffness;176 q*.z -= dz * stiffness;177 }178}179 180contacts_columns: proc[unsafe](positions: noalias []vec3, cs: []const contact) -> void = {181 for (c: u64 = 0; c < cs.length; c += 1) {182 p: *vec3 = positions[cast(cs[c].a, u64)]&;183 q: *vec3 = positions[cast(cs[c].b, u64)]&;184 dx: f32 = q*.x - p*.x;185 dy: f32 = q*.y - p*.y;186 dz: f32 = q*.z - p*.z;187 p*.x += dx * stiffness;188 p*.y += dy * stiffness;189 p*.z += dz * stiffness;190 q*.x -= dx * stiffness;191 q*.y -= dy * stiffness;192 q*.z -= dz * stiffness;193 }194}195 196contacts_split: proc[unsafe](px: noalias []f32, py: noalias []f32, pz: noalias []f32, cs: []const contact) -> void = {197 for (c: u64 = 0; c < cs.length; c += 1) {198 a: u64 = cast(cs[c].a, u64);199 b: u64 = cast(cs[c].b, u64);200 dx: f32 = px[b] - px[a];201 dy: f32 = py[b] - py[a];202 dz: f32 = pz[b] - pz[a];203 px[a] += dx * stiffness;204 py[a] += dy * stiffness;205 pz[a] += dz * stiffness;206 px[b] -= dx * stiffness;207 py[b] -= dy * stiffness;208 pz[b] -= dz * stiffness;209 }210}211 212contacts_blocks: proc[unsafe](blocks: noalias []body_block, cs: []const contact) -> void = {213 for (c: u64 = 0; c < cs.length; c += 1) {214 a: u64 = cast(cs[c].a, u64);215 b: u64 = cast(cs[c].b, u64);216 p: *body_block = blocks[a / 8]&;217 q: *body_block = blocks[b / 8]&;218 la: u64 = a % 8;219 lb: u64 = b % 8;220 dx: f32 = q*.px[lb] - p*.px[la];221 dy: f32 = q*.py[lb] - p*.py[la];222 dz: f32 = q*.pz[lb] - p*.pz[la];223 p*.px[la] += dx * stiffness;224 p*.py[la] += dy * stiffness;225 p*.pz[la] += dz * stiffness;226 q*.px[lb] -= dx * stiffness;227 q*.py[lb] -= dy * stiffness;228 q*.pz[lb] -= dz * stiffness;229 }230}231 232// ---------------------------------------------------------------- measuring233 234// How many 64-byte lines hold one body's x, y and z, given each one's byte235// offset from the start of a line. Each float is 4 bytes at a multiple of 4,236// so none of them crosses a line on its own.237lines_for: proc(x: u64, y: u64, z: u64) -> u64 = {238 lx: u64 = x / 64;239 ly: u64 = y / 64;240 lz: u64 = z / 64;241 n: u64 = 1;242 if (ly != lx) { n += 1; }243 if (lz != lx && lz != ly) { n += 1; }244 return n;245}246 247// The average over the first 4096 bodies, which covers every repeating pattern.248lines_per_lookup: proc(layout: u32) -> f64 = {249 total: u64 = 0;250 for (i: u64 = 0; i < 4096; i += 1) {251 if (layout == 0) {252 base: u64 = i * sizeof(body);253 total += lines_for(base, base + 4, base + 8);254 } else if (layout == 1) {255 base: u64 = i * sizeof(vec3);256 total += lines_for(base, base + 4, base + 8);257 } else if (layout == 2) {258 // three separate arrays: counted as never the same line259 total += 3;260 } else {261 base: u64 = (i / 8) * sizeof(body_block) + (i % 8) * 4;262 total += lines_for(base, base + 32, base + 64);263 }264 }265 return cast(total, f64) / 4096.0;266}267 268checksum_aos: proc(bodies: []const body) -> f64 = {269 sum: f64 = 0.0;270 for (i: u64 = 0; i < bodies.length; i += 1) {271 sum += cast(bodies[i].position.x + bodies[i].position.y + bodies[i].position.z, f64);272 }273 return sum;274}275 276checksum_columns: proc(positions: []const vec3) -> f64 = {277 sum: f64 = 0.0;278 for (i: u64 = 0; i < positions.length; i += 1) {279 sum += cast(positions[i].x + positions[i].y + positions[i].z, f64);280 }281 return sum;282}283 284checksum_split: proc(xs: []const f32, ys: []const f32, zs: []const f32) -> f64 = {285 sum: f64 = 0.0;286 for (i: u64 = 0; i < xs.length; i += 1) {287 sum += cast(xs[i] + ys[i] + zs[i], f64);288 }289 return sum;290}291 292checksum_blocks: proc(blocks: []const body_block) -> f64 = {293 sum: f64 = 0.0;294 for (b: u64 = 0; b < blocks.length; b += 1) {295 for (l: u64 = 0; l < 8; l += 1) {296 sum += cast(blocks[b].px[l] + blocks[b].py[l] + blocks[b].pz[l], f64);297 }298 }299 return sum;300}301 302seconds: proc() -> f64 = {303 return cast(clock(), f64) / cast(CLOCKS_PER_SEC, f64);304}305 306main: proc(argc: i32, argv: ^^c8) -> i32 = {307 timing: b32 = has_flag(argc, argv, "--time");308 steps: u64 = timing ? 20 : 2;309 same: b32 = 0;310 311 level: region = {312 aos: []body in level = new_slice<body>(level, body_count);313 col: *columns in level = new<columns>(level, {314 .positions = new_slice<vec3>(level, body_count),315 .velocities = new_slice<vec3>(level, body_count),316 });317 spl: *split in level = new<split>(level, {318 .px = new_slice<f32>(level, body_count),319 .py = new_slice<f32>(level, body_count),320 .pz = new_slice<f32>(level, body_count),321 .vx = new_slice<f32>(level, body_count),322 .vy = new_slice<f32>(level, body_count),323 .vz = new_slice<f32>(level, body_count),324 });325 block_count: u64 = body_count / 8;326 blocks: []body_block in level = new_slice<body_block>(level, block_count);327 328 for (i: u64 = 0; i < body_count; i += 1) {329 p: vec3 = { .x = random01() * 100.0, .y = random01() * 100.0, .z = random01() * 100.0 };330 v: vec3 = { .x = random01() - 0.5, .y = random01() * 4.0, .z = random01() - 0.5 };331 aos[i] = { .position = p, .velocity = v, .inverse_mass = 1.0, .radius = 0.5 };332 col*.positions[i] = p;333 col*.velocities[i] = v;334 spl*.px[i] = p.x; spl*.py[i] = p.y; spl*.pz[i] = p.z;335 spl*.vx[i] = v.x; spl*.vy[i] = v.y; spl*.vz[i] = v.z;336 k: *body_block = blocks[i / 8]&;337 l: u64 = i % 8;338 k*.px[l] = p.x; k*.py[l] = p.y; k*.pz[l] = p.z;339 k*.vx[l] = v.x; k*.vy[l] = v.y; k*.vz[l] = v.z;340 }341 cs: []contact in level = new_slice<contact>(level, contact_count);342 for (c: u64 = 0; c < contact_count; c += 1) {343 cs[c] = { .a = random_index(body_count), .b = random_index(body_count) };344 }345 346 // stream, then lookup, for each layout in turn347 t: [9]f64 = {};348 t[0] = seconds();349 for (s: u64 = 0; s < steps; s += 1) {350 unsafe {351 integrate_aos(aos, dt);352 }353 }354 t[1] = seconds();355 for (s: u64 = 0; s < steps; s += 1) {356 unsafe {357 contacts_aos(aos, cs);358 }359 }360 t[2] = seconds();361 for (s: u64 = 0; s < steps; s += 1) {362 unsafe {363 integrate_columns(col*.positions, col*.velocities, dt);364 }365 }366 t[3] = seconds();367 for (s: u64 = 0; s < steps; s += 1) {368 unsafe {369 contacts_columns(col*.positions, cs);370 }371 }372 t[4] = seconds();373 for (s: u64 = 0; s < steps; s += 1) {374 unsafe { integrate_split(spl*.px, spl*.py, spl*.pz, spl*.vx, spl*.vy, spl*.vz, dt); }375 }376 t[5] = seconds();377 for (s: u64 = 0; s < steps; s += 1) {378 unsafe {379 contacts_split(spl*.px, spl*.py, spl*.pz, cs);380 }381 }382 t[6] = seconds();383 for (s: u64 = 0; s < steps; s += 1) {384 unsafe {385 integrate_blocks(blocks, dt);386 }387 }388 t[7] = seconds();389 for (s: u64 = 0; s < steps; s += 1) {390 unsafe {391 contacts_blocks(blocks, cs);392 }393 }394 t[8] = seconds();395 396 printfmt("{} bodies, {} contacts, {} steps\n", body_count, contact_count, steps);397 printfmt("bytes per body: aos {}, columns {}, split {}, blocks {}\n",398 sizeof(body), sizeof(vec3) * 2, sizeof(f32) * 6, sizeof(body_block) / 8);399 printfmt("cache lines one lookup touches: aos {:.3}, columns {:.3}, split {:.3}, blocks {:.3}\n",400 lines_per_lookup(0), lines_per_lookup(1), lines_per_lookup(2), lines_per_lookup(3));401 402 sums: [4]f64 = {403 checksum_aos(aos),404 checksum_columns(col*.positions),405 checksum_split(spl*.px, spl*.py, spl*.pz),406 checksum_blocks(blocks),407 };408 printfmt("checksum aos {:.3}\n", sums[0]);409 printfmt("checksum columns {:.3}\n", sums[1]);410 printfmt("checksum split {:.3}\n", sums[2]);411 printfmt("checksum blocks {:.3}\n", sums[3]);412 same = sums[0] == sums[1] && sums[1] == sums[2] && sums[2] == sums[3];413 printfmt("same result: {}\n", same ? "yes" : "no");414 415 if (timing) {416 n_stream: f64 = cast(body_count * steps, f64);417 n_lookup: f64 = cast(contact_count * steps, f64);418 printfmt("ns per body stream ns per contact lookup\n");419 printfmt(" aos {:6.2} {:6.2}\n", (t[1] - t[0]) * 1.0e9 / n_stream, (t[2] - t[1]) * 1.0e9 / n_lookup);420 printfmt(" columns {:6.2} {:6.2}\n", (t[3] - t[2]) * 1.0e9 / n_stream, (t[4] - t[3]) * 1.0e9 / n_lookup);421 printfmt(" split {:6.2} {:6.2}\n", (t[5] - t[4]) * 1.0e9 / n_stream, (t[6] - t[5]) * 1.0e9 / n_lookup);422 printfmt(" blocks {:6.2} {:6.2}\n", (t[7] - t[6]) * 1.0e9 / n_stream, (t[8] - t[7]) * 1.0e9 / n_lookup);423 }424 }425 return same ? 0 : 1;426}What access_patterns prints. tools/check.py fails the lesson if a single byte differs.
1048576 bodies, 1048576 contacts, 2 steps
bytes per body: aos 32, columns 24, split 24, blocks 24
cache lines one lookup touches: aos 1.000, columns 1.125, split 3.000, blocks 2.000
checksum aos 157337021.562
checksum columns 157337021.562
checksum split 157337021.562
checksum blocks 157337021.562
same result: yes
Advanced · library · built in lesson 4
kayan.rin
includes/kayan.rin · 529 lines
1// kayan: archetype tables, entity handles, and moving entities between tables.2//3// This is the world lesson 4 builds, kept as a library so lessons 5 and 6 can4// use it as it is. Read it with lesson 4: the lesson explains the why, this5// file is the how.6//7// What a program using it provides:8//9// * its components, as structs, and an enum with one member per component;10// * `component_of: proc<T>() -> <its enum>` for each component type, which11// the generic procs here call to find a type's id;12// * a call to `kayan_register` for each component, with its reflect record.13//14// Ids are u32 here, so the library does not need to know the program's enum.15// Up to 64 components, because a signature is a u64.16//17// Not here, on purpose: threads, change tracking, chunked storage, relations.18// Each is worth a lesson of its own and none is needed to see the idea.19//20// A table's columns are bytes addressed by a component's size and alignment, and21// almost all of this is checked anyway: the bytes are a `[]u8`, so a row past the22// end stops the program like any index. Three procs need unsafe { }: making the23// world, which keeps the region's arena; taking an aligned block from it; and24// viewing bytes as T. What a lesson can hold is said in the types: `option<*T>`25// where null is possible, `*const` where nothing is written, a `[]T` cut to the26// rows in use.27//28// The world lives in the region `level`, declared here for every program that29// imports this file; lesson 6 also uses `frame`, one region per game frame.30 31import "std/print.rin"32import "std/memops.rin"33import "std/vec.rin"34import "std/hashmap.rin"35import "std/slice.rin"36import "std/reflect.rin"37 38level: region;39frame: region in level;40 41// A handle. `index` picks a slot in the world's slot list and `generation` says42// which life of that slot the handle was made for. Despawning bumps the slot's43// generation, so every handle to the dead entity stops matching, even after the44// index is reused. Generations start at 1, so a zeroed handle, `{}`, never45// names a live entity and can stand for "none".46kayan_entity: struct = {47 index: u32;48 generation: u32;49}50 51// Where an entity lives now: which table, which row. Rows move (swap-remove,52// lesson 2), so this is updated whenever one does. Handles never move. A free53// slot already holds its next generation, which no handle has yet, so the54// generation check alone says whether a handle is alive.55kayan_slot: struct = {56 generation: u32;57 table: u32;58 row: u32;59}60 61// One archetype. `entities[row]` is the handle of the entity in that row, which62// is how a swap-remove finds the slot to fix. `columns[id]` holds `capacity` rows63// of component `id` when the signature has bit `id`, and is empty otherwise. Each64// slice knows its own length, so nothing here depends on a count being right.65kayan_table: struct = {66 signature: u64;67 count: u64;68 capacity: u64;69 entities: []kayan_entity in level;70 columns: [64][]u8 in level;71}72 73kayan_query: struct = {74 all: u64;75 none: u64;76}77 78// What a structural change reports. `dead` means the handle no longer names a79// live entity, so nothing was changed.80kayan_status: enum[must_use] = {81 ok,82 dead,83}84 85kayan_world: struct = {86 // The region's arena, kept so columns can grow without being handed the87 // region each call; set by kayan_world_make.88 arena: ^memops_arena;89 // Reflect records are globals, so they live in `permanent`; null until90 // the component is registered.91 infos: [64]option<*const reflect in permanent>;92 component_count: u32;93 // Tables are referred to by index. A pointer from kayan_table_at is good94 // until the next table is created, which can move the list; the checker95 // knows this and makes the caller take it again after any call that may.96 tables: *vec<kayan_table> in level;97 by_signature: *hashmap<u64, u32> in level;98 slots: *vec<kayan_slot> in level;99 free: *vec<u32> in level;100 live: u64;101}102 103// ---------------------------------------------------------------------------104// Components105// ---------------------------------------------------------------------------106 107kayan_bit: proc(id: u32) -> u64 = {108 return 1'u64 shl cast(id, u64);109}110 111kayan_id_of: proc<T>() -> u32 = {112 return cast(component_of<T>(), u32);113}114 115kayan_mask_of: proc<T>() -> u64 = {116 return kayan_bit(kayan_id_of<T>());117}118 119// The world and its lists, in the region. Unsafe: the world keeps the region's120// arena, a raw pointer, and such a value is only made by a proc that vouches for it.121kayan_world_make: proc(r: region level) -> *kayan_world in level = {122 unsafe {123 w: *kayan_world in level = new<kayan_world>(r, {124 .tables = vec<kayan_table>make(r, 8),125 .by_signature = hashmap<u64, u32>make(r),126 .slots = vec<kayan_slot>make(r, 64),127 .free = vec<u32>make(r, 16),128 });129 w*.arena = r;130 return w;131 }132}133 134kayan_register: proc(w: *kayan_world, id: u32, info: *const reflect in permanent) -> void = {135 if (id >= 64) {136 printfmt("kayan: component id {} is past the 64 a signature can hold\n", id);137 exit(1);138 }139 w*.infos[id] = info;140 if (id + 1 > w*.component_count) {141 w*.component_count = id + 1;142 }143}144 145// The reflect record of a registered component. Stops the program for an id146// that was never registered, so callers get a `*const`, not a `option<*const>`.147kayan_info: proc(w: *const kayan_world, id: u32) -> *const reflect in permanent = {148 info: option<*const reflect in permanent> = w*.infos[id];149 if (!info) {150 printfmt("kayan: component {} was never registered\n", id);151 exit(1);152 }153 return info;154}155 156kayan_component_name: proc(w: *const kayan_world, id: u32) -> ^const char = {157 return kayan_info(w, id)*.name;158}159 160// ---------------------------------------------------------------------------161// Tables162// ---------------------------------------------------------------------------163 164kayan_table_has: proc(t: *const kayan_table, id: u32) -> b32 = {165 return (t*.signature & kayan_bit(id)) != 0;166}167 168// Borrowed from the table list: good until the next kayan_table_for or spawn.169// It takes `*const` because it reads the world and moves nothing.170kayan_table_at: proc(w: *const kayan_world, index: u32) -> *kayan_table in level = {171 return vec<kayan_table>at(w*.tables, cast(index, u64));172}173 174// A block of zeroed bytes in the region, aligned for a component. Unsafe: only the175// arena can promise the alignment.176kayan_bytes: proc(w: *kayan_world, length: u64, align: u64) -> []u8 in level = {177 unsafe {178 raw: ^u8 = cast(memops_arena_push_zero(w*.arena[0]&, length, align), ^u8);179 return cast(raw[0..length], []u8 in level);180 }181}182 183// Bytes as `count` T's: the one place bytes become a type, so it takes nothing on184// trust. T must be plain data: any bytes are then a valid T, so reading the wrong185// type gives wrong numbers, never a bad pointer. The rows must fit in the bytes,186// on an address T may live at.187kayan_view: proc<T>(bytes: []u8 in level, count: u64) -> []T in level = {188 if (!reflect_is_plain(T<>&)) {189 printfmt("kayan: a column holds plain data, and {} is not\n", T<>.name);190 exit(1);191 }192 if (count > bytes.length / sizeof(T)) {193 printfmt("kayan: a view of {} {} on {} bytes\n", count, T<>.name, bytes.length);194 exit(1);195 }196 unsafe {197 if (cast(bytes.data, usize) % alignof(T) != 0) {198 printfmt("kayan: bytes that are not aligned for {}\n", T<>.name);199 exit(1);200 }201 raw: ^T = cast(bytes.data, ^T);202 return cast(raw[0..count], []T in level);203 }204}205 206// Double the table's columns. The old columns stay in the arena unused: an207// arena does not free single allocations, and these lessons accept the waste.208kayan_table_grow: proc(w: *kayan_world, t: *kayan_table) -> void = {209 capacity: u64 = t*.capacity == 0 ? 16 : t*.capacity * 2;210 entities: []kayan_entity in level = kayan_view<kayan_entity>(211 kayan_bytes(w, sizeof(kayan_entity) * capacity, alignof(kayan_entity)), capacity);212 slice<kayan_entity>copy(entities[0..t*.count], t*.entities[0..t*.count]);213 t*.entities = entities;214 for (id: u32 = 0; id < w*.component_count; id += 1) {215 if (kayan_table_has(t, id)) {216 info: *const reflect in permanent = kayan_info(w, id);217 column: []u8 in level = kayan_bytes(w, info*.size * capacity, info*.align);218 kept: u64 = info*.size * t*.count;219 slice<u8>copy(column[0..kept], t*.columns[id][0..kept]);220 t*.columns[id] = column;221 }222 }223 t*.capacity = capacity;224}225 226// The table for a signature, made the first time anything needs it.227kayan_table_for: proc(w: *kayan_world, signature: u64) -> u32 = {228 found: option<*u32> = hashmap<u64, u32>get(w*.by_signature, signature);229 if (found) {230 return found*;231 }232 for (id: u32 = 0; id < 64; id += 1) {233 if ((signature & kayan_bit(id)) != 0 && (id >= w*.component_count || w*.infos[id] == null)) {234 printfmt("kayan: signature {} uses component {}, which was never registered\n", signature, id);235 exit(1);236 }237 }238 t: kayan_table = {};239 t.signature = signature;240 index: u32 = cast(w*.tables*.length, u32);241 _ = vec<kayan_table>push(w*.tables, t);242 _ = hashmap<u64, u32>set(w*.by_signature, signature, index);243 return index;244}245 246// A new row with zeroed components, owned by `e`.247kayan_table_add_row: proc(w: *kayan_world, t: *kayan_table, e: kayan_entity) -> u64 = {248 if (t*.count == t*.capacity) {249 kayan_table_grow(w, t);250 }251 row: u64 = t*.count;252 t*.entities[row] = e;253 for (id: u32 = 0; id < w*.component_count; id += 1) {254 if (kayan_table_has(t, id)) {255 size: u64 = kayan_info(w, id)*.size;256 slice<u8>zero(t*.columns[id][row * size..(row + 1) * size]);257 }258 }259 t*.count += 1;260 return row;261}262 263// Swap-remove: the last row moves into `row`, and the slot of the entity that264// moved is told its new row. This is the line lesson 2 could not write.265kayan_table_remove_row: proc(w: *kayan_world, t: *kayan_table, row: u64) -> void = {266 if (row >= t*.count) {267 printfmt("kayan: row {} is past the {} rows in use\n", row, t*.count);268 exit(1);269 }270 last: u64 = t*.count - 1;271 if (row != last) {272 for (id: u32 = 0; id < w*.component_count; id += 1) {273 if (kayan_table_has(t, id)) {274 size: u64 = kayan_info(w, id)*.size;275 moved: []u8 in level = t*.columns[id][last * size..(last + 1) * size];276 slice<u8>copy(t*.columns[id][row * size..(row + 1) * size], moved);277 }278 }279 moved_entity: kayan_entity = t*.entities[last];280 t*.entities[row] = moved_entity;281 vec<kayan_slot>at(w*.slots, cast(moved_entity.index, u64))*.row = cast(row, u32);282 }283 t*.count = last;284}285 286// It takes `*const` because it only reads, so the caller's column slices stay good.287kayan_matches: proc(t: *const kayan_table, q: kayan_query) -> b32 = {288 return (t*.signature & q.all) == q.all && (t*.signature & q.none) == 0;289}290 291// T's column, cut to the rows in use. Asking for a column the table does not292// have is a bug in the caller, so it stops the program. The slice is a borrow of293// the table, good until a row is added or removed; the checker enforces that at294// the caller.295kayan_column: proc<T>(t: *const kayan_table) -> []T in level = {296 id: u32 = kayan_id_of<T>();297 if (!kayan_table_has(t, id)) {298 printfmt("kayan: table {} has no column for component {}\n", t*.signature, id);299 exit(1);300 }301 return kayan_view<T>(t*.columns[id], t*.count);302}303 304kayan_table_entities: proc(t: *const kayan_table) -> []const kayan_entity in level = {305 return t*.entities[0..t*.count];306}307 308// ---------------------------------------------------------------------------309// Entities310// ---------------------------------------------------------------------------311 312// It takes `*const`, as kayan_matches does.313kayan_alive: proc(w: *const kayan_world, e: kayan_entity) -> b32 = {314 if (cast(e.index, u64) >= w*.slots*.length) {315 return 0;316 }317 return vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.generation == e.generation;318}319 320// A new entity with the components in `signature`, all zeroed. A freed index is321// reused before a new one is made, with the generation its despawn left behind.322// It may grow a table, which moves columns; the signature says so by taking323// `*kayan_world`, so the caller's column slices are dead after it.324kayan_spawn: proc(w: *kayan_world, signature: u64) -> kayan_entity = {325 e: kayan_entity = {};326 if (!vec<u32>is_empty(w*.free)) {327 e.index = vec<u32>pop(w*.free);328 } else {329 e.index = cast(w*.slots*.length, u32);330 _ = vec<kayan_slot>push(w*.slots, { .generation = 1 });331 }332 generation: u32 = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.generation;333 e.generation = generation;334 335 table_index: u32 = kayan_table_for(w, signature);336 row: u64 = kayan_table_add_row(w, kayan_table_at(w, table_index), e);337 338 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));339 slot*.table = table_index;340 slot*.row = cast(row, u32);341 w*.live += 1;342 return e;343}344 345// A dead handle reports `dead`, so despawning twice is harmless.346kayan_despawn: proc(w: *kayan_world, e: kayan_entity) -> kayan_status = {347 if (!kayan_alive(w, e)) {348 return kayan_status.dead;349 }350 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));351 table: u32 = slot*.table;352 row: u64 = cast(slot*.row, u64);353 kayan_table_remove_row(w, kayan_table_at(w, table), row);354 slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));355 slot*.generation += 1;356 _ = vec<u32>push(w*.free, e.index);357 w*.live -= 1;358 return kayan_status.ok;359}360 361kayan_signature: proc(w: *const kayan_world, e: kayan_entity) -> u64 = {362 if (!kayan_alive(w, e)) {363 return 0;364 }365 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));366 return kayan_table_at(w, slot*.table)*.signature;367}368 369// Move `e` to the table for `signature`. Components both tables have are370// copied; components only the new table has start zeroed; components only the371// old table has are dropped.372kayan_move: proc(w: *kayan_world, e: kayan_entity, signature: u64) -> kayan_status = {373 if (!kayan_alive(w, e)) {374 return kayan_status.dead;375 }376 from_index: u32 = vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.table;377 from_row: u64 = cast(vec<kayan_slot>at(w*.slots, cast(e.index, u64))*.row, u64);378 if (kayan_table_at(w, from_index)*.signature == signature) {379 return kayan_status.ok;380 }381 // kayan_table_for may add a table, so the table pointers are taken after it.382 to_index: u32 = kayan_table_for(w, signature);383 to_row: u64 = kayan_table_add_row(w, kayan_table_at(w, to_index), e);384 from: *kayan_table in level = kayan_table_at(w, from_index);385 to: *kayan_table in level = kayan_table_at(w, to_index);386 for (id: u32 = 0; id < w*.component_count; id += 1) {387 if (kayan_table_has(from, id) && kayan_table_has(to, id)) {388 size: u64 = kayan_info(w, id)*.size;389 moved: []u8 in level = from*.columns[id][from_row * size..(from_row + 1) * size];390 slice<u8>copy(to*.columns[id][to_row * size..(to_row + 1) * size], moved);391 }392 }393 kayan_table_remove_row(w, from, from_row);394 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));395 slot*.table = to_index;396 slot*.row = cast(to_row, u32);397 return kayan_status.ok;398}399 400// e's T, or null if e is dead or has no T: a `option<*T>`, so the caller tests it.401// The row is checked against the rows in use, so a slot pointing past the end402// stops the program. The pointer is a borrow of the world: good until403// the next spawn, despawn, add or remove.404kayan_get: proc<T>(w: *const kayan_world, e: kayan_entity) -> option<*T in level> = {405 if (!kayan_alive(w, e)) {406 return null;407 }408 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));409 t: *kayan_table in level = kayan_table_at(w, slot*.table);410 if (!kayan_table_has(t, kayan_id_of<T>())) {411 return null;412 }413 rows: []T in level = kayan_column<T>(t);414 return rows[cast(slot*.row, u64)]&;415}416 417// Write e's T. Returns 0 if e is dead or has no T. Checked: it is kayan_get418// and a test, which is what every caller of kayan_get has to write anyway.419kayan_set: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> b32 = {420 p: option<*T> = kayan_get<T>(w, e);421 if (!p) {422 return 0;423 }424 p* = value;425 return 1;426}427 428// e's component `id` as its bytes, or an empty slice if e is dead or has no429// such component. For tools that work from reflection records rather than from430// types, like lesson 5's inspector.431kayan_get_bytes: proc(w: *const kayan_world, e: kayan_entity, id: u32) -> []u8 in level = {432 none: []u8 in level = {};433 if (!kayan_alive(w, e) || id >= w*.component_count) {434 return none;435 }436 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));437 t: *kayan_table in level = kayan_table_at(w, slot*.table);438 if (!kayan_table_has(t, id)) {439 return none;440 }441 size: u64 = kayan_info(w, id)*.size;442 column: []u8 in level = t*.columns[id][0..t*.count * size];443 row: u64 = cast(slot*.row, u64);444 return column[row * size..(row + 1) * size];445}446 447// Give e a T, moving it to a new table if it did not have one. Checked: it is448// written in terms of move and set.449kayan_add: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> kayan_status = {450 if (kayan_move(w, e, kayan_signature(w, e) | kayan_mask_of<T>()) == kayan_status.dead) {451 return kayan_status.dead;452 }453 _ = kayan_set<T>(w, e, value);454 return kayan_status.ok;455}456 457// Take e's T away, moving it to the table without it.458kayan_remove: proc<T>(w: *kayan_world, e: kayan_entity) -> kayan_status = {459 return kayan_move(w, e, kayan_signature(w, e) & ~kayan_mask_of<T>());460}461 462// Every row of every table must belong to a live entity whose slot points back463// at that table and row, and the rows must add up to the live count. Returns464// how many of those checks failed, printing each.465kayan_verify: proc(w: *const kayan_world) -> u64 = {466 problems: u64 = 0;467 rows: u64 = 0;468 for (ti: u64 = 0; ti < w*.tables*.length; ti += 1) {469 t: *kayan_table = vec<kayan_table>at(w*.tables, ti);470 ids: []const kayan_entity = kayan_table_entities(t);471 for (row: u64 = 0; row < ids.length; row += 1) {472 e: kayan_entity = ids[row];473 if (!kayan_alive(w, e)) {474 printfmt("kayan_verify: table {} row {} holds dead entity {}:{}\n",475 ti, row, e.index, e.generation);476 problems += 1;477 continue;478 }479 slot: *kayan_slot = vec<kayan_slot>at(w*.slots, cast(e.index, u64));480 if (cast(slot*.table, u64) != ti || cast(slot*.row, u64) != row) {481 printfmt("kayan_verify: {}:{} is in table {} row {}, its slot says table {} row {}\n",482 e.index, e.generation, ti, row, slot*.table, slot*.row);483 problems += 1;484 }485 }486 rows += t*.count;487 }488 if (rows != w*.live) {489 printfmt("kayan_verify: {} rows in tables, {} entities alive\n", rows, w*.live);490 problems += 1;491 }492 return problems;493}494 495// ---------------------------------------------------------------------------496// Printing497// ---------------------------------------------------------------------------498 499// It takes `*const` because it only prints, so a caller walking the tables keeps500// its pointers.501kayan_print_signature: proc(w: *const kayan_world, signature: u64) -> void = {502 printfmt("{");503 first: b32 = 1;504 for (id: u32 = 0; id < w*.component_count; id += 1) {505 if ((signature & kayan_bit(id)) != 0) {506 if (!first) {507 printfmt(" ");508 }509 printfmt("{}", kayan_component_name(w, id));510 first = 0;511 }512 }513 printfmt("}");514}515 516kayan_print_tables: proc(w: *const kayan_world) -> void = {517 for (ti: u64 = 0; ti < w*.tables*.length; ti += 1) {518 t: *kayan_table = vec<kayan_table>at(w*.tables, ti);519 printfmt(" table {} ", cast(ti, i32));520 kayan_print_signature(w, t*.signature);521 printfmt(":");522 ids: []const kayan_entity = kayan_table_entities(t);523 for (row: u64 = 0; row < ids.length; row += 1) {524 e: kayan_entity = ids[row];525 printfmt(" {}:{}", e.index, e.generation);526 }527 printfmt("\n");528 }529}Tooling
Build and check
tools/check.py runs every program and diffs its output
tools/check.py
1"""Build every lesson, run it, and compare what it prints with its .expected file.2 3 python tools/check.py every lesson4 python tools/check.py archetype only lessons whose path contains "archetype"5 python tools/check.py --bless rewrite the .expected files from this run6 python tools/check.py --sanitize also build with AddressSanitizer and UBSan7 8A lesson passes only if all four steps are clean:9 10 1. `rin compile` prints nothing but its "generated" line,11 2. the C compiler reports no warnings (-Wall -Wextra -Werror),12 3. the program exits with 0,13 4. its output matches the .expected file beside it, byte for byte.14 15The lessons are the list in build.rin, so there is one list to keep. The16dojo's playgrounds are in it too. Each dojo block is also checked on its own17(`python tools/check.py dojo` for only them): its solution.rin, a whole program18that imports the harness, must build and print its harness.expected;19level1.rin must build and fail, since it has a gap; level2.rin must build.20 21rin is found through RIN, then RIN_HOME, then PATH; its std is the folder beside22it, as `rin build` assumes. The C compiler is CC, or clang.23"""24 25from __future__ import annotations26 27import os28import re29import shutil30import subprocess31import sys32from pathlib import Path33 34ROOT = Path(__file__).resolve().parent.parent35OUT = ROOT / "build" / "check"36 37 38def find_rin() -> Path:39 names = ["rin.exe", "rin"] if os.name == "nt" else ["rin", "rin.exe"]40 if os.environ.get("RIN"):41 return Path(os.environ["RIN"])42 if os.environ.get("RIN_HOME"):43 for name in names:44 candidate = Path(os.environ["RIN_HOME"]) / name45 if candidate.exists():46 return candidate47 for name in names:48 found = shutil.which(name)49 if found:50 return Path(found)51 sys.exit("check: rin not found; set RIN or RIN_HOME, or put rin on PATH")52 53 54def list_from_build_rin(name: str) -> list[str] | None:55 text = (ROOT / "build.rin").read_text(encoding="utf-8")56 block = re.search(rf"^{name}:[^{{]*\{{(.*?)\}}", text, re.S | re.M)57 if not block:58 return None59 return re.findall(r'"([^"]+)"', block.group(1))60 61 62def lessons_from_build_rin() -> list[Path]:63 entries = list_from_build_rin("build_entries")64 if entries is None:65 sys.exit("check: no build_entries in build.rin")66 return [ROOT / entry for entry in entries]67 68 69def run(cmd: list[str], cwd: Path = ROOT,70 env: dict[str, str] | None = None) -> subprocess.CompletedProcess[str]:71 return subprocess.run(cmd, cwd=cwd, env=env, text=True, stdout=subprocess.PIPE,72 stderr=subprocess.STDOUT, timeout=300)73 74 75# On Windows a sanitized program loads clang_rt.asan_dynamic-x86_64.dll when it76# starts, and exits with 0xC0000135 before main if the DLL is not on PATH. It77# lives in the C compiler's own resource directory, so ask the compiler.78def sanitizer_env(cc: str) -> dict[str, str] | None:79 if os.name != "nt":80 return None81 resource = run([cc, "-print-resource-dir"]).stdout.strip()82 runtime = Path(resource) / "lib" / "windows"83 if not runtime.is_dir():84 return None85 env = dict(os.environ)86 env["PATH"] = str(runtime) + os.pathsep + env.get("PATH", "")87 return env88 89 90def check(rin: Path, cc: str, lesson: Path, bless: bool, sanitize: bool,91 env: dict[str, str] | None) -> tuple[bool, str]:92 stem = lesson.stem93 work = OUT / stem94 work.mkdir(parents=True, exist_ok=True)95 c_path = work / f"{stem}.c"96 exe = work / (f"{stem}.exe" if os.name == "nt" else stem)97 98 translate = run([str(rin), "compile", str(lesson), "-o", str(c_path),99 "--header", str(work / f"{stem}.h")])100 noise = [line for line in translate.stdout.splitlines()101 if line.strip() and not line.startswith("rin: generated")]102 if translate.returncode != 0 or noise:103 return False, "rin compile:\n" + translate.stdout104 105 flags = ["-std=c11", "-g", "-Wall", "-Wextra", "-Werror"]106 flags += [f"-D{define}" for define in list_from_build_rin("build_defines") or []]107 if sanitize:108 flags += ["-fsanitize=address,undefined", "-fno-sanitize-recover=all"]109 includes = ["-I", str(rin.parent), "-I", str(rin.parent / "std")]110 libs = [] if os.name == "nt" else ["-lm"]111 build = run([cc, *flags, *includes, str(c_path), "-o", str(exe), *libs])112 if build.returncode != 0 or build.stdout.strip():113 return False, f"{cc}:\n" + build.stdout114 115 ran = run([str(exe)], env=env)116 if ran.returncode != 0:117 return False, f"exit {ran.returncode}:\n{ran.stdout}"118 119 actual = ran.stdout.replace("\r\n", "\n")120 expected_path = lesson.with_suffix(".expected")121 if bless:122 expected_path.write_text(actual, encoding="utf-8", newline="\n")123 return True, "blessed"124 if not expected_path.exists():125 return False, f"no {expected_path.name}; run with --bless after reading the output:\n{actual}"126 expected = expected_path.read_text(encoding="utf-8").replace("\r\n", "\n")127 if actual != expected:128 got, want = actual.splitlines(), expected.splitlines()129 for i in range(max(len(got), len(want))):130 g = got[i] if i < len(got) else "<missing>"131 w = want[i] if i < len(want) else "<missing>"132 if g != w:133 return False, f"line {i + 1}\n got: {g}\n want: {w}"134 return True, ""135 136 137# ---------------------------------------------------------------- the dojo138 139DOJO = ROOT / "dojo"140# A kata in progress is full of unused parameters and locals.141KATA_QUIET = ["-Wno-unused-parameter", "-Wno-unused-variable", "-Wno-unused-but-set-variable",142 "-Wno-unused-function"]143 144 145def dojo_blocks() -> list[Path]:146 roots = [DOJO]147 blocks: list[Path] = []148 for root in roots:149 blocks += sorted((p for p in root.iterdir() if p.is_dir() and re.match(r"\d+\.", p.name)),150 key=lambda p: int(p.name.split(".")[0]))151 return blocks152 153 154# A kata file is a whole program: it imports what it uses and the harness,155# which has main, so each of the solution and the levels builds where it is.156def dojo_build(rin: Path, cc: str, block: Path, kata: Path, where: str, sanitize: bool,157 quiet: bool) -> tuple[Path | None, str]:158 folder = OUT / "dojo" / block.name159 folder.mkdir(parents=True, exist_ok=True)160 c_path = folder / f"{where}.c"161 exe = folder / (f"{where}.exe" if os.name == "nt" else where)162 translate = run([str(rin), "compile", str(kata), "-o", str(c_path),163 "--header", str(folder / f"{where}.h")])164 noise = [line for line in translate.stdout.splitlines()165 if line.strip() and not line.startswith("rin: generated")]166 if translate.returncode != 0 or noise:167 return None, "rin compile:\n" + translate.stdout168 flags = ["-std=c11", "-g", "-Wall", "-Wextra", "-Werror", *(KATA_QUIET if quiet else [])]169 flags += [f"-D{define}" for define in list_from_build_rin("build_defines") or []]170 if sanitize:171 flags += ["-fsanitize=address,undefined", "-fno-sanitize-recover=all"]172 includes = ["-I", str(rin.parent), "-I", str(rin.parent / "std")]173 libs = [] if os.name == "nt" else ["-lm"]174 build = run([cc, *flags, *includes, str(c_path), "-o", str(exe), *libs])175 if build.returncode != 0 or build.stdout.strip():176 return None, f"{cc}:\n" + build.stdout177 return exe, ""178 179 180def check_dojo(rin: Path, cc: str, block: Path, bless: bool, sanitize: bool,181 env: dict[str, str] | None) -> tuple[bool, str]:182 exe, err = dojo_build(rin, cc, block, block / "solution.rin", "solution", sanitize, False)183 if not exe:184 return False, "solution.rin, " + err185 ran = run([str(exe)], env=env)186 if ran.returncode != 0:187 return False, f"the solution fails its harness, exit {ran.returncode}:\n{ran.stdout}"188 actual = ran.stdout.replace("\r\n", "\n")189 expected_path = block / "harness.expected"190 detail = ""191 if bless:192 expected_path.write_text(actual, encoding="utf-8", newline="\n")193 detail = "blessed"194 elif not expected_path.exists():195 return False, f"no harness.expected; run with --bless after reading the output:\n{actual}"196 elif actual != expected_path.read_text(encoding="utf-8").replace("\r\n", "\n"):197 return False, "the harness's output with the solution is not harness.expected"198 for level in (1, 2):199 exe, err = dojo_build(rin, cc, block, block / f"level{level}.rin", f"level{level}", sanitize, True)200 if not exe:201 return False, f"level{level}.rin, " + err202 if level == 1 and run([str(exe)], env=env).returncode == 0:203 return False, "level 1 passes the harness as given; its gap is not a gap"204 return True, detail205 206 207def main() -> int:208 args = sys.argv[1:]209 bless = "--bless" in args210 sanitize = "--sanitize" in args211 filters = [a for a in args if not a.startswith("--")]212 213 rin = find_rin()214 cc = os.environ.get("CC", "clang")215 lessons = [l for l in lessons_from_build_rin()216 if not filters or any(f in str(l) for f in filters)]217 218 env = sanitizer_env(cc) if sanitize else None219 220 failed = 0221 for lesson in lessons:222 ok, detail = check(rin, cc, lesson, bless, sanitize, env)223 name = lesson.relative_to(ROOT).as_posix()224 print(f"{'ok ' if ok else 'FAIL'} {name}" + (f" ({detail})" if ok and detail else ""))225 if not ok:226 failed += 1227 print(" " + detail.replace("\n", "\n "))228 print(f"{len(lessons) - failed} of {len(lessons)} passed")229 230 blocks = [b for b in dojo_blocks()231 if not filters or any(f in b.relative_to(ROOT).as_posix() for f in filters)]232 dojo_failed = 0233 for block in blocks:234 ok, detail = check_dojo(rin, cc, block, bless, sanitize, env)235 print(f"{'ok ' if ok else 'FAIL'} {block.relative_to(ROOT).as_posix()}, the solution and the levels"236 + (f" ({detail})" if ok and detail else ""))237 if not ok:238 dojo_failed += 1239 print(" " + detail.replace("\n", "\n "))240 if blocks:241 print(f"{len(blocks) - dojo_failed} of {len(blocks)} dojo blocks passed")242 return 1 if failed or dojo_failed else 0243 244 245if __name__ == "__main__":246 sys.exit(main())tools/dojo.py
1"""The dojo's runner: start a kata, run it against its harness, keep your times.2 3 python tools/dojo.py the blocks, your reps, and what is due4 python tools/dojo.py start slot_map 2 a fresh kata.rin from level 2; the clock starts5 python tools/dojo.py slot_map build the harness with your kata.rin and run it6 python tools/dojo.py slot_map --seed 42 --stop 13377 the same fuzz run, stopping before operation 13378 python tools/dojo.py slot_map --sanitize with AddressSanitizer and UBSan9 python tools/dojo.py play slot_map the block's playground10 11A block can be named by its number, its name, or the start of its name:12`3`, `slot_map` and `slot` are the same block.13 14When a started kata passes, the time since `start` is added to dojo/log.md,15and the block is due again after 1, 3, 7, 14, then 30 days, counting its16passes. Spaced reps are what make a block stick, more than many in one day.17 18If the harness crashes rather than failing a check (an out-of-bounds index,19say), the runner runs it again with --trace to show the operation that was20running.21 22Your kata.rin files are yours: git ignores them, and `start` will not23overwrite one you have changed unless you add --force. A kata is a whole24program: it imports what it uses and the harness, which has main, so the25runner builds kata.rin, and the harness comes with it. rin and the C compiler26are found as tools/check.py finds them.27"""28 29from __future__ import annotations30 31import datetime as dt32import json33import os34import re35import shutil36import subprocess37import sys38from pathlib import Path39 40sys.path.insert(0, str(Path(__file__).resolve().parent))41from check import find_rin, list_from_build_rin # noqa: E40242 43ROOT = Path(__file__).resolve().parent.parent44DOJO = ROOT / "dojo"45OUT = ROOT / "build" / "dojo"46LOG = DOJO / "log.md"47STARTED = DOJO / ".started.json"48INTERVALS = [1, 3, 7, 14, 30]49 50 51def blocks() -> list[Path]:52 return sorted((p for p in DOJO.iterdir() if p.is_dir() and re.match(r"\d+\.", p.name)),53 key=lambda p: int(p.name.split(".")[0]))54 55 56def name_of(block: Path) -> str:57 return block.name.split(".", 1)[1]58 59 60def find(word: str) -> Path:61 found = [b for b in blocks() if word in (b.name, name_of(b), b.name.split(".")[0])]62 found = found or [b for b in blocks() if name_of(b).startswith(word)]63 if len(found) != 1:64 sys.exit(f"dojo: no single block called {word!r}; the blocks are "65 + ", ".join(name_of(b) for b in blocks()))66 return found[0]67 68 69# The target times, from the level table in each block's README:70# | 2 | every proc | 8:00 |71def targets(block: Path) -> dict[int, str]:72 readme = block / "README.md"73 text = readme.read_text(encoding="utf-8") if readme.exists() else ""74 return {int(m.group(1)): m.group(2) for m in re.finditer(r"^\| (\d) \|.*\| (\d+:\d\d) \|$", text, re.M)}75 76 77# rin writes C, and the C compiler does the rest, as tools/check.py does. A78# kata in progress is full of unused parameters and locals, so the warnings for79# those are off and warnings are not errors; tools/check.py builds the80# solutions with -Werror and every warning on.81def build(source: Path, exe: Path, sanitize: bool) -> None:82 exe.parent.mkdir(parents=True, exist_ok=True)83 rin = find_rin()84 c_path = exe.with_suffix(".c")85 translate = subprocess.run([str(rin), "compile", str(source), "-o", str(c_path),86 "--header", str(exe.with_suffix(".h"))], text=True,87 stdout=subprocess.PIPE, stderr=subprocess.STDOUT)88 noise = [l for l in translate.stdout.splitlines() if l.strip() and not l.startswith("rin: generated")]89 if translate.returncode != 0 or noise:90 print(translate.stdout, end="")91 sys.exit(1)92 cc = os.environ.get("CC", "clang")93 flags = ["-std=c11", "-g", "-Wall", "-Wextra", "-Wno-unused-parameter", "-Wno-unused-variable",94 "-Wno-unused-but-set-variable", "-Wno-unused-function"]95 flags += [f"-D{define}" for define in list_from_build_rin("build_defines") or []]96 if sanitize:97 flags += ["-fsanitize=address,undefined", "-fno-sanitize-recover=all"]98 includes = ["-I", str(rin.parent), "-I", str(rin.parent / "std")]99 libs = [] if os.name == "nt" else ["-lm"]100 if subprocess.run([cc, *flags, *includes, str(c_path), "-o", str(exe), *libs]).returncode != 0:101 sys.exit(1)102 103 104def load_started() -> dict:105 try:106 return json.loads(STARTED.read_text(encoding="utf-8"))107 except (OSError, ValueError):108 return {}109 110 111def save_started(started: dict) -> None:112 STARTED.write_text(json.dumps(started, indent=2) + "\n", encoding="utf-8")113 114 115def log_rows() -> list[list[str]]:116 if not LOG.exists():117 return []118 rows = []119 for line in LOG.read_text(encoding="utf-8").splitlines():120 cells = [c.strip() for c in line.strip().strip("|").split("|")]121 if len(cells) == 4 and re.match(r"\d{4}-\d\d-\d\d", cells[0]):122 rows.append(cells)123 return rows124 125 126def minutes(seconds: float) -> str:127 s = int(round(seconds))128 return f"{s // 60}:{s % 60:02d}"129 130 131def seconds(text: str) -> int:132 m, s = text.split(":")133 return int(m) * 60 + int(s)134 135 136def due(passes: list[dt.date]) -> dt.date | None:137 if not passes:138 return None139 return max(passes) + dt.timedelta(days=INTERVALS[min(len(passes), len(INTERVALS)) - 1])140 141 142def status() -> int:143 rows = log_rows()144 today = dt.date.today()145 started = load_started()146 print(f"{'block':<16}{'reps':>5} {'best time, level':<18}{'last':<12}next")147 for b in blocks():148 mine = [r for r in rows if r[1] == name_of(b)]149 passes = [dt.date.fromisoformat(r[0]) for r in mine]150 best = min(mine, key=lambda r: (-int(r[2]), seconds(r[3])), default=None)151 best_text = f"{best[3]}, level {best[2]}" if best else "-"152 next_day = due(passes)153 next_text = "now" if next_day is None or next_day <= today else next_day.isoformat()154 if name_of(b) in started:155 next_text = f"started, level {started[name_of(b)]['level']}"156 last = max(passes).isoformat() if passes else "-"157 print(f"{b.name:<16}{len(mine):>5} {best_text:<18}{last:<12}{next_text}")158 print()159 print("start one with: python tools/dojo.py start <block> <level 1 to 3>")160 return 0161 162 163def start(word: str, level: str, force: bool) -> int:164 b = find(word)165 source = b / f"level{level}.rin"166 if not source.exists():167 sys.exit(f"dojo: {name_of(b)} has no level {level}; the levels are "168 + ", ".join(p.stem[len("level"):] for p in sorted(b.glob("level*.rin"))))169 kata = b / "kata.rin"170 untouched = [p.read_text(encoding="utf-8") for p in b.glob("level*.rin")]171 if kata.exists() and kata.read_text(encoding="utf-8") not in untouched and not force:172 sys.exit(f"dojo: {kata.relative_to(ROOT)} has your work in it; add --force to replace it")173 shutil.copyfile(source, kata)174 started = load_started()175 started[name_of(b)] = {"level": int(level), "at": dt.datetime.now().isoformat(timespec="seconds")}176 save_started(started)177 target = targets(b).get(int(level))178 print(f"{kata.relative_to(ROOT)} is level {level}" + (f"; the target is {target}" if target else ""))179 print(f"the clock is running. Test it with: python tools/dojo.py {name_of(b)}")180 return 0181 182 183# Run a program, printing what it prints as it prints it, and keep the text to184# look for PASS in.185def stream(cmd: list[str]) -> tuple[int, str]:186 lines = []187 with subprocess.Popen(cmd, cwd=ROOT, text=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) as p:188 for line in p.stdout:189 print(line, end="", flush=True)190 lines.append(line)191 return p.returncode, "".join(lines)192 193 194def check(word: str, args: list[str]) -> int:195 b = find(word)196 if not (b / "kata.rin").exists():197 sys.exit(f"dojo: no kata.rin in {b.relative_to(ROOT)} yet; start one with: "198 f"python tools/dojo.py start {name_of(b)} 1")199 passthrough = [a for a in args if a != "--sanitize"]200 exe = OUT / name_of(b) / ("harness.exe" if os.name == "nt" else "harness")201 build(b / "kata.rin", exe, "--sanitize" in args)202 code, out = stream([str(exe), *passthrough])203 if code != 0 and not any(line.startswith("FAIL") for line in out.splitlines()):204 traced = subprocess.run([str(exe), *passthrough, "--trace", "1"], cwd=ROOT, text=True,205 stdout=subprocess.PIPE, stderr=subprocess.STDOUT)206 ops = [l for l in traced.stdout.splitlines() if l.startswith("operation ")]207 print("the harness crashed" + (f"; the last operation it started was\n {ops[-1]}" if ops208 else " in the scenario, after the last step printed above"))209 if code != 0 or "PASS" not in out.splitlines():210 return 1211 started = load_started()212 entry = started.pop(name_of(b), None)213 if entry and not passthrough:214 took = (dt.datetime.now() - dt.datetime.fromisoformat(entry["at"])).total_seconds()215 if not LOG.exists():216 LOG.write_text("# Dojo log\n\n| date | block | level | time |\n|---|---|---|---|\n", encoding="utf-8")217 with LOG.open("a", encoding="utf-8") as f:218 f.write(f"| {dt.date.today().isoformat()} | {name_of(b)} | {entry['level']} | {minutes(took)} |\n")219 save_started(started)220 target = targets(b).get(entry["level"])221 print(f"\nlevel {entry['level']} in {minutes(took)}" + (f", target {target}" if target else ""))222 passes = [dt.date.fromisoformat(r[0]) for r in log_rows() if r[1] == name_of(b)]223 print(f"logged in {LOG.relative_to(ROOT)}; {name_of(b)} is due again on {due(passes).isoformat()}")224 return 0225 226 227def play(word: str) -> int:228 b = find(word)229 exe = OUT / name_of(b) / ("playground.exe" if os.name == "nt" else "playground")230 build(b / f"play_{name_of(b)}.rin", exe, False)231 return subprocess.run([str(exe)], cwd=ROOT).returncode232 233 234def main() -> int:235 args = sys.argv[1:]236 if not args:237 return status()238 if args[0] in ("-h", "--help", "help"):239 print(__doc__)240 return 0241 if args[0] == "start":242 rest = [a for a in args[1:] if a != "--force"]243 if len(rest) != 2:244 sys.exit("dojo: start <block> <level>")245 return start(rest[0], rest[1], "--force" in args)246 if args[0] == "play":247 if len(args) != 2:248 sys.exit("dojo: play <block>")249 return play(args[1])250 return check(args[0], args[1:])251 252 253if __name__ == "__main__":254 sys.exit(main())dojo/dojo.rin
1// What every harness in the dojo shares: the run's settings, a random number2// generator that gives the same numbers everywhere, how a step is checked, and3// how a failure is reported. It also holds the two types the katas take as4// given: a handle, and a status.5//6// Every kata and harness here checks with rin's safety rules on. The handful of7// procs marked unsafe { } below read the command line and compare C strings, which8// is C's business and not the checker's.9//10// The harnesses import it, and so does every kata file that uses a handle or11// a status:12//13// import "../dojo.rin"14 15import "std/print.rin"16import "std/memops.rin"17import "std/vec.rin"18import "std/fmt.rin"19 20// The one region every kata lives in: the harness opens it in main and hands21// it to the scenario and the fuzz run, which make their kata in it.22//23// `frame` is one step shorter: a region a harness opens and closes many times24// over a run, once per game frame or fuzz step. Blocks 5 and 11 use it. It is declared here, not in the25// harness that uses it, because a nested region must be declared right after26// its enclosing one, and an imported file's declarations come after the27// importer's own.28level: region;29frame: region in level;30 31// A generational handle: an index, and which life of that index it was made32// for. Generations start at 1, so a zeroed handle, `{}`, is never alive and can33// mean "no handle".34dojo_handle: struct = {35 index: u32;36 generation: u32;37}38 39// What a change reports. `dead` means the handle no longer names anything40// alive, so nothing changed.41dojo_status: enum[must_use] = {42 ok,43 dead,44}45 46dojo_handle_eq: proc(a: dojo_handle, b: dojo_handle) -> b32 = {47 return a.index == b.index && a.generation == b.generation;48}49 50// Writes `index:generation`.51dojo_fmt_handle: proc(f: *fmt, h: dojo_handle) -> void = {52 fmt_u64(f, cast(h.index, u64));53 fmt_char(f, ':');54 fmt_u64(f, cast(h.generation, u64));55}56 57dojo_status_name: proc(s: dojo_status) -> ^const char = {58 return s == dojo_status.ok ? "ok" : "dead";59}60 61dojo_bool_name: proc(b: b32) -> ^const char = {62 return b ? "true" : "false";63}64 65// One harness run: which block, the fuzz seed, how many operations, where to66// stop for the debugger, and whether to print each operation. The command line67// sets them:68//69// harness --seed 42 --ops 5000 --stop 1337 --trace 170dojo_run: struct = {71 block: ^const char;72 seed: u64;73 ops: u64;74 stop: u64;75 trace: u64;76}77 78// Trusted: it walks a C string to its terminator.79dojo_parse_u64: proc(text: ^const c8, out: *u64) -> b32 = {80 unsafe {81 value: u64 = 0;82 i: u64 = 0;83 while (text[i] >= '0' && text[i] <= '9') {84 value = value * 10 + cast(text[i] - '0', u64);85 i += 1;86 }87 out* = value;88 return i > 0 && text[i] == 0;89 }90}91 92// Trusted: argv is C's array of C strings. The run lives in the region the93// harness opened, since it holds the block's name, a C string, and a struct94// with one of those is kept through a pointer in safe code.95dojo_run_init: proc(r: region R, argc: i32, argv: ^^c8, block: ^const char, title: ^const char) -> *dojo_run in R = {96 unsafe {97 run: *dojo_run in R = new<dojo_run>(r);98 run*.block = block;99 run*.seed = 1;100 run*.ops = 20000;101 for (i: i32 = 1; i < argc; i += 2) {102 value: u64 = 0;103 if (i + 1 >= argc || !dojo_parse_u64(argv[i + 1], value&)) {104 printfmt("usage: harness [--seed N] [--ops N] [--stop N] [--trace 1]\n");105 exit(2);106 }107 if (strcmp(argv[i], "--seed") == 0) {108 run*.seed = value;109 } else if (strcmp(argv[i], "--ops") == 0) {110 run*.ops = value;111 } else if (strcmp(argv[i], "--stop") == 0) {112 run*.stop = value;113 } else if (strcmp(argv[i], "--trace") == 0) {114 run*.trace = value;115 } else {116 printfmt("usage: harness [--seed N] [--ops N] [--stop N] [--trace 1]\n");117 exit(2);118 }119 }120 printfmt("dojo: {}\n", title);121 return run;122 }123}124 125// Are two C strings the same text? Safe code cannot call strcmp, which is126// C's; this is the one place that does, for the katas that keep names.127dojo_cstr_equal: proc(a: ^const char, b: ^const char) -> b32 = {128 unsafe {129 return strcmp(a, b) == 0;130 }131}132 133// A step of the scenario: what was done, what came back, and what should have.134// Printed when it matches; the end of the run when it does not. Trusted: it135// compares two C strings.136dojo_same: proc(what: ^const char, got: ^const char, want: ^const char) -> void = {137 unsafe {138 if (strcmp(got, want) != 0) {139 printfmt("FAIL in the scenario: {} gave {}, it should give {}\n", what, got, want);140 exit(1);141 }142 printfmt(" {}: {}\n", what, got);143 }144}145 146dojo_fuzz: proc(run: *dojo_run) -> void = {147 printfmt("fuzz: {} operations from seed {}, checked after each\n", run*.ops, run*.seed);148}149 150// Set a breakpoint here, and run with --stop N.151dojo_stopped_at: u64 = 0;152 153dojo_stop_here: proc(opcode: u64) -> void = {154 dojo_stopped_at = opcode;155}156 157// Call before operation `op`. It returns 1 when the operation should be158// described, and the caller prints the description and a newline: with159// --trace, every operation is, so a crash shows the last one that started;160// with --stop at this operation, it is, after dojo_stop_here has run, so a161// breakpoint there lands just before it.162dojo_before: proc(run: *dojo_run, opcode: u64) -> b32 = {163 if (run*.stop == opcode) {164 dojo_stop_here(opcode);165 printfmt("stopped before operation {}: ", opcode);166 return 1;167 }168 if (run*.trace != 0) {169 printfmt("operation {}: ", opcode);170 return 1;171 }172 return 0;173}174 175// A failure in the fuzz run, in three parts: dojo_fail starts the line, the176// caller prints what went wrong and a newline, and dojo_failed says how to get177// back here in the debugger and ends the run.178dojo_fail: proc(run: *dojo_run, opcode: u64) -> void = {179 printfmt("FAIL at operation {} (seed {}): ", opcode, run*.seed);180}181 182dojo_failed: proc(run: *dojo_run, opcode: u64) -> void = {183 printfmt(" debug it: build/dojo/{}/harness --seed {} --stop {}\n", run*.block, run*.seed, opcode);184 printfmt(" with a breakpoint on dojo_stop_here, which runs just before that operation\n");185 exit(1);186}187 188dojo_pass: proc() -> void = {189 printfmt("PASS\n");190}191 192// splitmix64: a few lines, fast, and the same numbers on every machine and in193// rust-dod's dojo, so a seed means the same run everywhere. u64 arithmetic194// wraps, as splitmix64 wants.195dojo_rng: struct = {196 state: u64;197}198 199dojo_rng_next: proc(r: *dojo_rng) -> u64 = {200 r*.state = r*.state + 11400714819323198485'u64;201 z: u64 = r*.state;202 z = (z ^ (z shr 30)) * 13787848793156543929'u64;203 z = (z ^ (z shr 27)) * 10723151780598845931'u64;204 return z ^ (z shr 31);205}206 207// A number in 0..n.208dojo_rng_below: proc(r: *dojo_rng, n: u64) -> u64 = {209 return dojo_rng_next(r) % n;210}build.rin
1// How the lessons are built. Read by `rin build`, not executed.2//3// Every lesson is its own program, named after its file. tools/check.py reads4// this same list, builds each lesson, runs it and compares its output with the5// .expected file beside it.6build_name: ^const char = "rin-dod";7build_dir: ^const char = "build";8build_generator: ^const char = "Ninja";9build_compiler: ^const char = "clang-cl";10build_type: ^const char = "Debug";11 12// std's string8 calls fopen, which the Windows SDK marks deprecated.13build_defines: [_]^const char = { "_CRT_SECURE_NO_WARNINGS" };14 15build_entries: [_]^const char = {16 // Build an ECS: start here, README.md17 "ecs/src/1.entities/1.1.ids/ids.rin",18 "ecs/src/1.entities/1.2.despawn/despawn.rin",19 "ecs/src/1.entities/1.3.generations/generations.rin",20 "ecs/src/1.entities/1.4.entity_index/entity_index.rin",21 "ecs/src/2.components/2.1.components/components.rin",22 "ecs/src/2.components/2.2.columns/columns.rin",23 "ecs/src/2.components/2.3.tables/tables.rin",24 "ecs/src/2.components/2.4.archetypes/archetypes.rin",25 "ecs/src/2.components/2.5.typed_access/typed_access.rin",26 "ecs/src/2.components/2.6.add_remove/add_remove.rin",27 "ecs/src/2.components/2.7.inspecting/inspecting.rin",28 "ecs/src/3.systems/3.1.queries/queries.rin",29 "ecs/src/3.systems/3.2.systems/systems.rin",30 "ecs/src/3.systems/3.3.commands/commands.rin",31 "ecs/src/3.systems/3.4.game/game.rin",32 33 // Advanced: layout and SIMD, docs/README.md34 "src/1.aos_vs_soa/aos_vs_soa.rin",35 "src/2.archetype_table/archetype_table.rin",36 "src/3.many_archetypes/many_archetypes.rin",37 "src/4.entity_handles/entity_handles.rin",38 "src/5.inspector/inspector.rin",39 "src/6.kayan/kayan.rin",40 "src/7.access_patterns/access_patterns.rin",41 42 // The dojo's playgrounds, dojo/README.md43 "dojo/1.free_list/play_free_list.rin",44 "dojo/2.swap_remove/play_swap_remove.rin",45 "dojo/3.slot_map/play_slot_map.rin",46 "dojo/4.column/play_column.rin",47 "dojo/5.commands/play_commands.rin",48 "dojo/6.bitset/play_bitset.rin",49 "dojo/7.arena/play_arena.rin",50 "dojo/8.index/play_index.rin",51 "dojo/9.interning/play_interning.rin",52 "dojo/10.edges/play_edges.rin",53 "dojo/11.reflect/play_reflect.rin",54 "dojo/12.watermark/play_watermark.rin",55 "dojo/13.dispatch/play_dispatch.rin",56};.gitignore
1build/2tools/__pycache__/3# The dojo: your katas, and the clock of the one you started.4dojo/*/kata.rin5dojo/.started.jsonLICENSE
1MIT License2 3Copyright (c) 2026 Ibrahim Hindawi4 5Permission is hereby granted, free of charge, to any person obtaining a copy6of this software and associated documentation files (the "Software"), to deal7in the Software without restriction, including without limitation the rights8to use, copy, modify, merge, publish, distribute, sublicense, and/or sell9copies of the Software, and to permit persons to whom the Software is10furnished to do so, subject to the following conditions:11 12The above copyright notice and this permission notice shall be included in all13copies or substantial portions of the Software.14 15THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR16IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,17FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE18AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER19LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,20OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE21SOFTWARE.rin · data-oriented design · build an ECS
rin-dod
Build an archetype entity component system from nothing, in fifteen chapters you read, compile and step through: from an entity as a number to a game running on it.
Data-oriented design in rin, in two courses and a dojo:
- Build an ECS. Start here. Fifteen chapters that build an archetype entity component system from nothing, in the order of its name: entities, then components, then systems. It ends with a game running on it. Each chapter is a program that compiles on its own, the output it must print, and a guide that builds it step by step.
- Advanced: layout and SIMD. Seven lessons, for after. They cover what a loop fetches from memory, aliasing and
noalias, what the compiler vectorizes and how to check it in the assembly, and which layout suits which access pattern. - The dojo. The thirteen building blocks an ECS is made of, taken out on their own: a playground to watch each one work and break, and a kata to write it yourself, timed, against a harness, until your hands know it.
There is no renderer, no window and no library: only data, and the loops that stream it.
Build an ECS
| chapter | about | |
|---|---|---|
| 1 | entities | |
| 1.1 | an entity is a number | ids, components as arrays indexed by them, a system as a loop |
| 1.2 | despawning | a free list, reused numbers, and the stale reference they cause |
| 1.3 | generations | handles that know when they are stale |
| 1.4 | the entity index | slots, packed rows, swap-remove, kayan_verify |
| 2 | components | |
| 2.1 | components and signatures | component ids, a registry of reflect records, 64-bit signatures |
| 2.2 | columns | one array of any component, as bytes, viewed as a typed slice |
| 2.3 | a table | the entity index and the columns, joined |
| 2.4 | archetypes | a table for every combination, found by signature |
| 2.5 | typed access | kayan_get<T>, kayan_set<T>; keep handles, borrow pointers |
| 2.6 | adding and removing | moving between tables, and the archetype graph |
| 2.7 | inspecting | any entity, field by field, from reflection |
| 3 | systems | |
| 3.1 | queries | the tables a system runs over, matched once and kept |
| 3.2 | systems | kernels, systems, a schedule, a frame |
| 3.3 | commands | why a system cannot change structure, and the buffer that can |
| 3.4 | a game | turrets, waves, bullets and fire, on everything above |
Read each chapter's guide first. It explains the new idea and builds its code step by step, and every sample in it is copied from the chapter's program. Then open the program: it holds everything built so far, with the chapter's new code near the end, so any chapter can be opened in a debugger and stepped through as a whole ECS. Run it and read the output against the code. Each guide ends with exercises, most of them "break this on purpose and watch what happens".
Each chapter's file starts from the previous chapter's, so diffing two neighbours shows exactly what a chapter changed:
git diff --no-index ecs/src/2.components/2.5.typed_access/typed_access.rin ecs/src/2.components/2.6.add_remove/add_remove.rin
The ECS is called kayan, after the C sketch it grew out of.
The dojo
| block | what it is | |
|---|---|---|
| 1 | free list and generations | handles that are safe to keep after what they name is gone |
| 2 | swap-remove and the back-reference | values packed with no holes, found by id: a sparse set |
| 3 | the slot map | 1 and 2 together: handles into packed values |
| 4 | a type-erased column | an array of any type as bytes, viewed as a typed slice |
| 5 | a command buffer | structural changes recorded now and applied later |
| 6 | bitsets | a set of components in a u64: has, all of, none of |
| 7 | an arena | push, push, clear: memory for things that die together |
| 8 | index, not pointer | links that stay right when their storage moves |
| 9 | interning | find or make, by key: a hash table that hands out dense ids |
| 10 | memoized edges | a slow lookup done once, then remembered on the edge |
| 11 | a reflection walk | reading, writing and comparing any value through its type's record |
| 12 | a watermark | looking only at what was added since last time |
| 13 | a dispatch table | behaviour as data: the schedule as a list of functions |
python tools/dojo.py your reps, and which blocks are due
python tools/dojo.py play slot_map watch it: the playground
python tools/dojo.py start slot_map 2 write it: a kata from level 2, timed
python tools/dojo.py slot_map check it against the harness
Each block has three levels (one proc, every proc, a blank page) and a few variations, and the runner keeps your times and spaces your reps out. dojo/README.md has the rest. Each course chapter says which blocks it is built from.
Advanced: layout and SIMD
| lesson | about | |
|---|---|---|
| 0 | from kayan | what was kept from the C kayan, what changed, and why |
| 1 | aos vs soa | the same particles in three layouts, what a loop fetches, and what SIMD needs |
| 2 | archetype table | one table: columns, rows, spawn, swap-remove, systems over slices |
| 3 | many archetypes | one table type for every archetype: ids, signatures, queries |
| 4 | entity handles | generational handles, slots, moving between archetypes |
| 5 | inspector | printing and editing any entity from reflection records |
| 6 | kayan | a small game, and deferring structural changes to a command buffer |
| 7 | access patterns | four layouts under streams and lookups, and how an ECS decides |
Lessons 1 and 7 are about memory, aliasing and the code the compiler makes. Lessons 2 to 6 build an ECS again, compressed into five steps, so read them as a recap of the course. docs/README.md has more.
Build and run
Needs the rin compiler with noalias (rin 5413d4e or later; rin.exe from rin-windows-x64, through RIN_HOME or PATH), CMake, Ninja, clang-cl and the Windows SDK.
rin build
build\cmake\ids.exe
Every chapter and lesson is its own program, named after its file. Or compile one by hand: rin writes C, and any C compiler does the rest, with rin's folder and its std on the include path:
rin compile ecs/src/2.components/2.4.archetypes/archetypes.rin -o archetypes.c
clang -std=c11 -g -I <rin> -I <rin>/std archetypes.c -o archetypes
python tools/check.py builds every program with clang and warnings as errors, runs it, and compares what it prints with its .expected file. --sanitize adds AddressSanitizer and UBSan, --bless rewrites the expected files after a deliberate change, and a word filters the programs. tools/check_snippets.py checks that every sample in the ECS guides is in its program:
python tools/check.py ecs
python tools/check.py kayan --sanitize
python tools/check_snippets.py
The checker also works on Linux. There it relaxes -Wformat only: rin's printfmt and std print write %llu for u64, which is unsigned long on 64-bit Linux.
Reading it in a browser
Open website/index.html. It has every chapter's and lesson's guide, its source with line numbers, and the output it must print, with an index on the left. It is generated from the files in this repository, so after changing a program or a guide run python tools/site.py to bring it up to date.
Layout
ecs/
src/
1.entities/
1.1.ids/
ids.rin the program
ids.expected what it prints
...
2.components/
3.systems/
docs/ a guide a chapter: steps, output, exercises
README.md the course on its own
src/ the advanced lessons, one folder each
includes/
kayan.rin lesson 4's world, used by lessons 5 and 6
docs/ the advanced lessons' guides
dojo/
1.free_list/ a block: README, playground, harness, levels, solution
...
dojo.rin what every harness shares
website/
index.html everything as one page: guide, code, output
template.html its layout and styles
tools/
check.py build, run and compare every program
check_snippets.py check every sample in the guides and dojo READMEs is in its program
site.py rebuild website/index.html from the repo
dojo.py start, run and time the dojo's katas
build.rin the program list, for `rin build`
License
MIT. See LICENSE.
Build an ECS · part 1, entities · 1.1
An entity is a number
ecs/src/1.entities/1.1.ids/ids.rin
An ECS has three parts: Entities, Components and Systems. This chapter builds all three in the simplest way that works, about a hundred lines, so you can see the whole shape before any of it gets clever. The chapters after this one keep the shape and fix its costs, one at a time.
The idea
A game is full of things: a turret, drones, bullets. An ECS splits each thing into three parts:
- An entity names the thing. It is just a number. It has no data and no behaviour of its own.
- Components are the thing's data: a position, a velocity, health. Each component is plain data, a small struct or a number.
- Systems are the behaviour: loops that run over every entity that has the components they need. Movement runs over everything with a position and a velocity.
A turret is entity 0 with a position and health. A drone is entity 1 with a position, a velocity and health. Nothing says "turret" or "drone" anywhere; what a thing is comes from which components it has.
Step 1: the entity
kayan_entity: struct = {
index: u32;
}
An entity is a number, so why the struct? Because a bare u32 can be mixed up with any other u32. A proc that takes a kayan_entity cannot be handed a count or a row index by mistake, and the struct costs nothing at run time: it compiles to the same 4 bytes. It also gives the number room to grow, and in 1.3 it does.
Step 2: the world
Where do components live? The simplest answer: one array per component, with one slot for every possible entity, indexed by the entity's number. Beside each array, a flag per entity saying whether it has that component.
kayan_world: struct = {
count: u32;
positions: [8]vec2;
has_position: [8]b32;
velocities: [8]vec2;
has_velocity: [8]b32;
healths: [8]f32;
has_health: [8]b32;
}
Look at the shape. This is lesson 1's struct of arrays: every position together, every velocity together. An entity's data is spread across the arrays, all at the same index. Entity 1's position is positions[1] and its health is healths[1].
Step 3: spawning
A new entity is the next number:
kayan_spawn: proc(w: *kayan_world) -> kayan_entity = {
if (w*.count == 8) {
printfmt("the world is full\n");
exit(1);
}
e: kayan_entity = { .index = w*.count };
w*.count += 1;
return e;
}
It has no components yet: every flag for its number is still false.
Step 4: giving it components
Giving an entity a component is writing the value at its number and raising the flag:
set_position: proc(w: *kayan_world, e: kayan_entity, p: vec2) -> void = {
w*.positions[e.index] = p;
w*.has_position[e.index] = true;
}
set_velocity and set_health are the same, one per component. Writing one proc per component type is tedious, and Part 2 replaces all of them with a single generic kayan_set<T>. For now, plain is better: you can see exactly which bytes get written.
Building a drone is spawning a number and attaching data to it:
drone_a: kayan_entity = kayan_spawn(w&);
set_position(w&, drone_a, { .x = 10.0, .y = 0.0 });
set_velocity(w&, drone_a, { .x = -1.0, .y = 0.5 });
set_health(w&, drone_a, 100.0);
Step 5: a system
A system is a loop. Movement runs over everything with a position and a velocity:
movement_system: proc(w: *kayan_world, dt: f32) -> u32 = {
moved: u32 = 0;
for (i: u32 = 0; i < w*.count; i += 1) {
if (w*.has_position[i] && w*.has_velocity[i]) {
w*.positions[i].x += w*.velocities[i].x * dt;
w*.positions[i].y += w*.velocities[i].y * dt;
moved += 1;
}
}
return moved;
}
It doesn't know about turrets, drones or bullets. It knows about positions and velocities, and anything that has both moves, including bullets that didn't exist when the system was written. That's the ECS idea in one proc: behaviour attaches to data, not to types of thing.
Run it
spawned turret 0, drones 1 and 2, bullet 3
entity 0 position ( 0.00, 0.00) velocity - health 500
entity 1 position (10.00, 0.00) velocity (-1.00, 0.50) health 100
entity 2 position ( 0.00, 10.00) velocity ( 0.50, -1.00) health 100
entity 3 position ( 0.00, 0.00) velocity ( 4.00, 4.00) health -
step 1: the movement system looked at 4 entities and moved 3
step 2: the movement system looked at 4 entities and moved 3
...
every component array has room for 8; 4 entities exist
positions hold 4 values, velocities 3, healths 3
The turret has no velocity, so movement skips it. The bullet has no health, and nothing cares.
What this costs
It works, and every later chapter keeps this shape: entities are numbers, components are arrays of plain data, systems are loops. But look at the last two lines of the output and at the loop:
- Every array is as long as the most entities the world can hold. Velocity has room for 8 and holds 3. A game with 10,000 entities and 50 component types would have 50 arrays of 10,000, mostly empty.
- Every system visits every entity and checks flags to skip the ones that don't apply. Movement looked at 4 and moved 3. With 10,000 entities of which 50 move, it would look at 10,000 to move 50.
- Nothing can be destroyed. There's no despawn, and numbers only go up.
The next chapter fixes 3. Chapter 1.4 packs the data so systems stop skipping, and Part 2 fixes 1 and 2 for good with tables.
Try
- Give the turret a velocity and run it. Nothing else changes: the turret moves because it now has the data movement wants.
- Add a
lifetime: [8]f32component with its flags, and anaging_systemthat counts it down. How many places did you have to touch? - Spawn a ninth entity. Read the error, then think about what a game would need instead of a fixed 8.
1// 1.1 An entity is a number2//3// An entity names one thing in the game: a drone, a bullet, the turret. It has4// no data of its own and no behaviour. It is a number, and everything about5// the thing it names is stored somewhere else, in arrays, at that number.6//7// This chapter builds the simplest ECS there is: one array per component, as8// long as the most entities the world can hold, and a flag beside each array9// saying which entities have that component. A system is a loop over every10// entity that checks the flags it needs.11//12// It works, and it already has the shape every later chapter keeps: entities13// are ids, components are plain data in arrays, systems are loops. It also14// wastes memory and time in ways the next chapters remove one at a time.15 16import "std/print.rin"17 18vec2: struct = {19 x: f32;20 y: f32;21}22 23// An entity is a number. Wrapping it in a struct costs nothing at run time and24// keeps it from being mixed up with any other u32: a proc that takes a25// kayan_entity cannot be handed a count or a row by mistake.26kayan_entity: struct = {27 index: u32;28}29 30// The world: how many entities exist, and for each component an array of31// values and an array of flags, both indexed by the entity's number. Every32// array has room for 8, the most entities this world can hold.33kayan_world: struct = {34 count: u32;35 positions: [8]vec2;36 has_position: [8]b32;37 velocities: [8]vec2;38 has_velocity: [8]b32;39 healths: [8]f32;40 has_health: [8]b32;41}42 43// A new entity is the next number. It has no components yet.44kayan_spawn: proc(w: *kayan_world) -> kayan_entity = {45 if (w*.count == 8) {46 printfmt("the world is full\n");47 exit(1);48 }49 e: kayan_entity = { .index = w*.count };50 w*.count += 1;51 return e;52}53 54// Giving an entity a component is writing the value at the entity's number and55// raising the flag beside it.56set_position: proc(w: *kayan_world, e: kayan_entity, p: vec2) -> void = {57 w*.positions[e.index] = p;58 w*.has_position[e.index] = true;59}60 61set_velocity: proc(w: *kayan_world, e: kayan_entity, v: vec2) -> void = {62 w*.velocities[e.index] = v;63 w*.has_velocity[e.index] = true;64}65 66set_health: proc(w: *kayan_world, e: kayan_entity, points: f32) -> void = {67 w*.healths[e.index] = points;68 w*.has_health[e.index] = true;69}70 71// A system: everything with a position and a velocity moves. It visits every72// entity in the world and skips the ones that lack either component. Returns73// how many it moved.74movement_system: proc(w: *kayan_world, dt: f32) -> u32 = {75 moved: u32 = 0;76 for (i: u32 = 0; i < w*.count; i += 1) {77 if (w*.has_position[i] && w*.has_velocity[i]) {78 w*.positions[i].x += w*.velocities[i].x * dt;79 w*.positions[i].y += w*.velocities[i].y * dt;80 moved += 1;81 }82 }83 return moved;84}85 86print_world: proc(w: *kayan_world) -> void = {87 for (i: u32 = 0; i < w*.count; i += 1) {88 printf(" entity %u", i);89 if (w*.has_position[i]) {90 printf(" position (%5.2f, %5.2f)", cast(w*.positions[i].x, f64), cast(w*.positions[i].y, f64));91 } else {92 printf(" position - ");93 }94 if (w*.has_velocity[i]) {95 printf(" velocity (%5.2f, %5.2f)", cast(w*.velocities[i].x, f64), cast(w*.velocities[i].y, f64));96 } else {97 printf(" velocity - ");98 }99 if (w*.has_health[i]) {100 printf(" health %.0f", cast(w*.healths[i], f64));101 } else {102 printf(" health -");103 }104 printf("\n");105 }106}107 108// How many of the 8 slots of one component's array hold a value.109used: proc(flags: [8]b32) -> u32 = {110 n: u32 = 0;111 for (i: u32 = 0; i < 8; i += 1) {112 if (flags[i]) {113 n += 1;114 }115 }116 return n;117}118 119main: proc() -> i32 = {120 w: kayan_world = {};121 122 turret: kayan_entity = kayan_spawn(w&);123 set_position(w&, turret, { .x = 0.0, .y = 0.0 });124 set_health(w&, turret, 500.0);125 126 drone_a: kayan_entity = kayan_spawn(w&);127 set_position(w&, drone_a, { .x = 10.0, .y = 0.0 });128 set_velocity(w&, drone_a, { .x = -1.0, .y = 0.5 });129 set_health(w&, drone_a, 100.0);130 131 drone_b: kayan_entity = kayan_spawn(w&);132 set_position(w&, drone_b, { .x = 0.0, .y = 10.0 });133 set_velocity(w&, drone_b, { .x = 0.5, .y = -1.0 });134 set_health(w&, drone_b, 100.0);135 136 bullet: kayan_entity = kayan_spawn(w&);137 set_position(w&, bullet, { .x = 0.0, .y = 0.0 });138 set_velocity(w&, bullet, { .x = 4.0, .y = 4.0 });139 140 printfmt("spawned turret {}, drones {} and {}, bullet {}\n",141 turret.index, drone_a.index, drone_b.index, bullet.index);142 print_world(w&);143 144 for (step: u32 = 1; step <= 2; step += 1) {145 moved: u32 = movement_system(w&, 0.5);146 printfmt("step {}: the movement system looked at {} entities and moved {}\n", step, w.count, moved);147 }148 print_world(w&);149 150 printfmt("every component array has room for 8; {} entities exist\n", w.count);151 printfmt(" positions hold {} values, velocities {}, healths {}\n",152 used(w.has_position), used(w.has_velocity), used(w.has_health));153 return 0;154}What ids prints. tools/check.py fails the chapter if a single byte differs.
spawned turret 0, drones 1 and 2, bullet 3
entity 0 position ( 0.00, 0.00) velocity - health 500
entity 1 position (10.00, 0.00) velocity (-1.00, 0.50) health 100
entity 2 position ( 0.00, 10.00) velocity ( 0.50, -1.00) health 100
entity 3 position ( 0.00, 0.00) velocity ( 4.00, 4.00) health -
step 1: the movement system looked at 4 entities and moved 3
step 2: the movement system looked at 4 entities and moved 3
entity 0 position ( 0.00, 0.00) velocity - health 500
entity 1 position ( 9.00, 0.50) velocity (-1.00, 0.50) health 100
entity 2 position ( 0.50, 9.00) velocity ( 0.50, -1.00) health 100
entity 3 position ( 4.00, 4.00) velocity ( 4.00, 4.00) health -
every component array has room for 8; 4 entities exist
positions hold 4 values, velocities 3, healths 3
Build an ECS · part 1, entities · 1.2
Despawning, and reusing numbers
ecs/src/1.entities/1.2.despawn/despawn.rin
In the dojo: 1. free list and generations, on its own, to watch and to practise.
Games destroy things constantly: a bullet hits, a drone dies, an effect fades. 1.1 could only add. This chapter adds kayan_despawn, and with it the first real problem in ECS design: what happens to an entity's number when the entity is gone?
Step 1: a free list
If despawned numbers were never reused, a game firing one bullet a frame would use up 8 numbers in 8 frames and 4 billion in a couple of years of play. More to the point, every array would keep growing. So despawned numbers go on a free list, and spawning takes from it first:
kayan_world: struct = {
count: u32;
alive: [8]b32;
free: [8]u32;
free_count: u32;
count is now how many numbers have ever been handed out, not how many entities exist; the numbers on the free list are the gaps. alive says which numbers are in use.
Step 2: spawning reuses
kayan_spawn: proc(w: *kayan_world) -> kayan_entity = {
e: kayan_entity = {};
if (w*.free_count > 0) {
w*.free_count -= 1;
e.index = w*.free[w*.free_count];
} else {
if (w*.count == 8) {
printfmt("the world is full\n");
exit(1);
}
e.index = w*.count;
w*.count += 1;
}
w*.alive[e.index] = true;
return e;
}
The free list is a stack: the last number freed is the first reused.
Step 3: despawning
kayan_despawn: proc(w: *kayan_world, e: kayan_entity) -> void = {
w*.alive[e.index] = false;
w*.has_position[e.index] = false;
w*.has_velocity[e.index] = false;
w*.has_health[e.index] = false;
w*.free[w*.free_count] = e.index;
w*.free_count += 1;
}
Despawning lowers every flag, so the entity's old components can't leak into whatever reuses the number. The values stay in the arrays, but with the flags down nobody reads them. Systems need no change: a despawned entity has no flags, so movement skips it without knowing anything about despawning.
Step 4: the bug
Now the problem. Things in a game refer to other things: a turret aims at a drone, a missile chases a target, a health bar follows a unit. With entities as numbers, a reference is a number:
target: kayan_entity = drone_a;
The program then destroys the drone, and the turret fires. The bullet is a new entity, and it takes the first free number, the drone's:
the turret targets entity 1, a drone
...
entity 1 is destroyed and despawned
entity 1 (free)
the turret fires: the bullet is entity 1
...
the turret aims at entity 1, at (2.00, 2.00), moving (4.00, 4.00)
that is the turret's own bullet, not the drone: the number was reused
Nothing told the turret its target died, and nothing could: references can be anywhere, in any number of places. The number in target is still a valid number, it just names a different entity now. No crash, no error, just a turret shooting at its own bullet. In a real game this is the bug where a homing missile turns around mid-flight, or a health bar jumps to another unit.
Run it
the turret targets entity 1, a drone
entity 0 position ( 0.00, 0.00) velocity - health 500
entity 1 position (10.00, 0.00) velocity (-1.00, 0.50) health 100
entity 2 position ( 0.00, 10.00) velocity ( 0.50, -1.00) health 100
entity 1 is destroyed and despawned
entity 0 position ( 0.00, 0.00) velocity - health 500
entity 1 (free)
entity 2 position ( 0.00, 10.00) velocity ( 0.50, -1.00) health 100
the turret fires: the bullet is entity 1
...
Why not just...
- ...never reuse numbers? The arrays would grow forever, and every system would walk an ever-longer list of dead numbers.
- ...find and clear every reference on despawn? You'd need to know where every reference lives. That's exactly what an ECS is trying to avoid: things refer to each other loosely, by number.
- **...check
alive[target.index]?** It saystrue: the number is alive. It's alive as a different entity, and nothing can tell the two apart.
The next chapter tells them apart with one extra number per entity.
Try
- Despawn the bullet too, then spawn two more entities. Which numbers do they get, and in what order? Why?
- Comment out the lines that lower the flags in
kayan_despawnand run it. What does the new bullet have that it shouldn't? - Despawn
drone_atwice. What does the free list look like after? Spawn twice more and see why that is a problem.
1// 1.2 Despawning, and reusing numbers2//3// Games destroy things all the time: a bullet hits, a drone dies. 1.1 could4// only add. Here an entity can be despawned, and its number goes on a free5// list, so the next spawn takes it instead of growing the world. Without that,6// a game that fires a bullet a frame would run out of numbers in seconds.7//8// Reuse has a cost, and this program walks straight into it. Something that9// kept an entity's number, the turret's target, still holds it after the10// entity is gone. When the number is handed to a new entity, the old11// reference silently names the new one. 1.3 fixes that.12 13import "std/print.rin"14 15vec2: struct = {16 x: f32;17 y: f32;18}19 20kayan_entity: struct = {21 index: u32;22}23 24// `alive` says which numbers are in use. `free` holds the numbers of despawned25// entities, `free_count` of them, ready to be handed out again.26kayan_world: struct = {27 count: u32;28 alive: [8]b32;29 free: [8]u32;30 free_count: u32;31 positions: [8]vec2;32 has_position: [8]b32;33 velocities: [8]vec2;34 has_velocity: [8]b32;35 healths: [8]f32;36 has_health: [8]b32;37}38 39// A free number is reused before a new one is made. A reused number must start40// with no components: whatever the last entity at this number had was cleared41// when it was despawned.42kayan_spawn: proc(w: *kayan_world) -> kayan_entity = {43 e: kayan_entity = {};44 if (w*.free_count > 0) {45 w*.free_count -= 1;46 e.index = w*.free[w*.free_count];47 } else {48 if (w*.count == 8) {49 printfmt("the world is full\n");50 exit(1);51 }52 e.index = w*.count;53 w*.count += 1;54 }55 w*.alive[e.index] = true;56 return e;57}58 59// The entity's components are dropped by lowering their flags, and its number60// goes on the free list. The values stay in the arrays, unread.61kayan_despawn: proc(w: *kayan_world, e: kayan_entity) -> void = {62 w*.alive[e.index] = false;63 w*.has_position[e.index] = false;64 w*.has_velocity[e.index] = false;65 w*.has_health[e.index] = false;66 w*.free[w*.free_count] = e.index;67 w*.free_count += 1;68}69 70set_position: proc(w: *kayan_world, e: kayan_entity, p: vec2) -> void = {71 w*.positions[e.index] = p;72 w*.has_position[e.index] = true;73}74 75set_velocity: proc(w: *kayan_world, e: kayan_entity, v: vec2) -> void = {76 w*.velocities[e.index] = v;77 w*.has_velocity[e.index] = true;78}79 80set_health: proc(w: *kayan_world, e: kayan_entity, points: f32) -> void = {81 w*.healths[e.index] = points;82 w*.has_health[e.index] = true;83}84 85// Despawned entities have no flags left, so the system skips them without86// knowing anything about despawning.87movement_system: proc(w: *kayan_world, dt: f32) -> u32 = {88 moved: u32 = 0;89 for (i: u32 = 0; i < w*.count; i += 1) {90 if (w*.has_position[i] && w*.has_velocity[i]) {91 w*.positions[i].x += w*.velocities[i].x * dt;92 w*.positions[i].y += w*.velocities[i].y * dt;93 moved += 1;94 }95 }96 return moved;97}98 99print_world: proc(w: *kayan_world) -> void = {100 for (i: u32 = 0; i < w*.count; i += 1) {101 if (!w*.alive[i]) {102 printf(" entity %u (free)\n", i);103 continue;104 }105 printf(" entity %u", i);106 if (w*.has_position[i]) {107 printf(" position (%5.2f, %5.2f)", cast(w*.positions[i].x, f64), cast(w*.positions[i].y, f64));108 } else {109 printf(" position - ");110 }111 if (w*.has_velocity[i]) {112 printf(" velocity (%5.2f, %5.2f)", cast(w*.velocities[i].x, f64), cast(w*.velocities[i].y, f64));113 } else {114 printf(" velocity - ");115 }116 if (w*.has_health[i]) {117 printf(" health %.0f", cast(w*.healths[i], f64));118 } else {119 printf(" health -");120 }121 printf("\n");122 }123}124 125main: proc() -> i32 = {126 w: kayan_world = {};127 128 turret: kayan_entity = kayan_spawn(w&);129 set_position(w&, turret, { .x = 0.0, .y = 0.0 });130 set_health(w&, turret, 500.0);131 132 drone_a: kayan_entity = kayan_spawn(w&);133 set_position(w&, drone_a, { .x = 10.0, .y = 0.0 });134 set_velocity(w&, drone_a, { .x = -1.0, .y = 0.5 });135 set_health(w&, drone_a, 100.0);136 137 drone_b: kayan_entity = kayan_spawn(w&);138 set_position(w&, drone_b, { .x = 0.0, .y = 10.0 });139 set_velocity(w&, drone_b, { .x = 0.5, .y = -1.0 });140 set_health(w&, drone_b, 100.0);141 142 // The turret remembers what it is aiming at, by number.143 target: kayan_entity = drone_a;144 printfmt("the turret targets entity {}, a drone\n", target.index);145 print_world(w&);146 147 printfmt("entity {} is destroyed and despawned\n", drone_a.index);148 kayan_despawn(w&, drone_a);149 print_world(w&);150 151 // The turret fires. The bullet takes the first free number.152 bullet: kayan_entity = kayan_spawn(w&);153 set_position(w&, bullet, { .x = 0.0, .y = 0.0 });154 set_velocity(w&, bullet, { .x = 4.0, .y = 4.0 });155 printfmt("the turret fires: the bullet is entity {}\n", bullet.index);156 _ = movement_system(w&, 0.5);157 print_world(w&);158 159 // Nothing told the turret its target died. Its number now names the bullet.160 printf("the turret aims at entity %u, at (%.2f, %.2f), moving (%.2f, %.2f)\n", target.index,161 cast(w.positions[target.index].x, f64), cast(w.positions[target.index].y, f64),162 cast(w.velocities[target.index].x, f64), cast(w.velocities[target.index].y, f64));163 printfmt(" that is the turret's own bullet, not the drone: the number was reused\n");164 return 0;165}What despawn prints. tools/check.py fails the chapter if a single byte differs.
the turret targets entity 1, a drone
entity 0 position ( 0.00, 0.00) velocity - health 500
entity 1 position (10.00, 0.00) velocity (-1.00, 0.50) health 100
entity 2 position ( 0.00, 10.00) velocity ( 0.50, -1.00) health 100
entity 1 is destroyed and despawned
entity 0 position ( 0.00, 0.00) velocity - health 500
entity 1 (free)
entity 2 position ( 0.00, 10.00) velocity ( 0.50, -1.00) health 100
the turret fires: the bullet is entity 1
entity 0 position ( 0.00, 0.00) velocity - health 500
entity 1 position ( 2.00, 2.00) velocity ( 4.00, 4.00) health -
entity 2 position ( 0.25, 9.50) velocity ( 0.50, -1.00) health 100
the turret aims at entity 1, at (2.00, 2.00), moving (4.00, 4.00)
that is the turret's own bullet, not the drone: the number was reused
Build an ECS · part 1, entities · 1.3
Generations
ecs/src/1.entities/1.3.generations/generations.rin
In the dojo: 1. free list and generations, on its own, to watch and to practise.
1.2 ended with a turret aiming at its own bullet. The number it remembered was reused, and nothing could tell the old entity from the new one. The fix is small and used by nearly every ECS: count each number's lives.
The idea
Give each number a generation. Entity 1 in its first life is 1:1. When it's despawned, number 1's generation goes up to 2, so the next entity to reuse it is 1:2. A reference carries both numbers:
kayan_entity: struct = {
index: u32;
generation: u32;
}
A reference with a generation is called a handle. The turret's target is the handle 1:1. After the drone dies and the bullet becomes 1:2, the target still says 1:1, and 1:1 != 1:2. One comparison catches the bug.
Step 1: the world keeps generations
kayan_world: struct = {
count: u32;
generations: [8]u32;
free: [8]u32;
free_count: u32;
generations[i] is the life number i is in now, or the life it'll be in next if i is free. 1.2's alive array is gone, because the generations already say it:
kayan_alive: proc(w: *kayan_world, e: kayan_entity) -> b32 = {
return e.index < w*.count && w*.generations[e.index] == e.generation;
}
A handle is alive when its generation is the one its number is in now.
Step 2: despawn bumps the generation
kayan_despawn: proc(w: *kayan_world, e: kayan_entity) -> kayan_status = {
if (!kayan_alive(w, e)) {
return kayan_status.dead;
}
w*.has_position[e.index] = false;
w*.has_velocity[e.index] = false;
w*.has_health[e.index] = false;
w*.generations[e.index] += 1;
w*.free[w*.free_count] = e.index;
w*.free_count += 1;
return kayan_status.ok;
}
The generation goes up at despawn, not at the next spawn. So a free number already holds a generation that no handle has been given yet, and no handle can match it. That's why there's no separate alive flag: "free" and "every existing handle is stale" are the same fact.
Despawning a dead handle now returns dead and changes nothing. In 1.2, despawning twice put the same number on the free list twice, and two later entities would have shared it.
Step 3: a status you can't ignore
kayan_status: enum[must_use] = {
ok,
dead,
}
enum[must_use] makes ignoring the result a compile error. When you mean to ignore it, you say so with _ = kayan_despawn(...). The program does that where it knows the handle is alive.
Step 4: generations start at 1
e.index = w*.count;
w*.count += 1;
w*.generations[e.index] = 1;
A brand-new number starts at generation 1, never 0. So the zeroed handle, 0:0, can never be alive, and {} is a free "no entity" value. A turret with nothing to aim at holds {}, and checking it needs no special case.
Step 5: every access checks
get_position: proc(w: *kayan_world, e: kayan_entity) -> *vec2 = {
if (!kayan_alive(w, e) || !w*.has_position[e.index]) {
return null;
}
return w*.positions[e.index]&;
}
Reading through a handle checks it first. A dead handle gives null instead of whatever data now sits at the number. The turret checks its target every time it aims, and picks a new one when the old one is gone.
Run it
the turret aims at 1:1, at (10.00, 0.00)
drone 1:1 is destroyed and despawned
the turret fires: the bullet is 1:2, the same number in its next life
entity 0:1 position ( 0.00, 0.00) velocity - health 500
entity 1:2 position ( 2.00, 2.00) velocity ( 4.00, 4.00) health -
entity 2:1 position ( 0.25, 9.50) velocity ( 0.50, -1.00) health 100
the turret's target 1:1 is dead; it picks 2:1 instead
the turret aims at 2:1, at (0.25, 9.50)
despawning 1:1 again: dead
the zeroed handle 0:0 is alive: no
The same moment that ended 1.2, and this time the handle knows.
Notes
- Cost: a handle is 8 bytes instead of 4, and checking one is one load and one compare. Cheap enough to do on every access.
- Wrapping: a
u32generation wraps after 4 billion reuses of the same number. A stale handle would have to survive that long to be fooled. Some ECSs use fewer bits for the generation, packing index and generation into oneu32oru64. The idea is the same. - What it doesn't fix: the arrays are still as long as the most entities, and systems still skip holes. That's next.
Try
- Keep a copy of the bullet's handle, despawn the bullet, spawn something new, and check the old handle with
kayan_alive. - Remove
_ =from one of thekayan_despawncalls and compile. Read the error. - Make
kayan_alivecompare only the index. Which line of the output goes wrong first?
1// 1.3 Generations2//3// 1.2's bug: a number outlives the entity it named, and when the number is4// reused the old reference names the new entity. The fix is to count lives.5// Each number gets a generation, and a reference carries both: entity 1 in its6// first life is 1:1, and the entity that reuses the number is 1:2. Despawning7// bumps the generation, so every reference to the dead entity stops matching8// at once, wherever it is kept.9//10// A reference with a generation is called a handle. Checking one is a single11// comparison, so every access can afford it.12 13import "std/print.rin"14 15vec2: struct = {16 x: f32;17 y: f32;18}19 20// A handle: which number, and which life of that number.21kayan_entity: struct = {22 index: u32;23 generation: u32;24}25 26// What despawning reports. `dead` means the handle no longer names a live27// entity, so nothing was changed. must_use makes ignoring it a compile error.28kayan_status: enum[must_use] = {29 ok,30 dead,31}32 33// `generations[i]` is the life number `i` is in, or will be in next if `i` is34// free. The `alive` array from 1.2 is gone: the generations say it.35kayan_world: struct = {36 count: u32;37 generations: [8]u32;38 free: [8]u32;39 free_count: u32;40 positions: [8]vec2;41 has_position: [8]b32;42 velocities: [8]vec2;43 has_velocity: [8]b32;44 healths: [8]f32;45 has_health: [8]b32;46}47 48// A handle is alive when its generation is the one its number is in now.49// Despawning bumps the number's generation straight away, so a free number50// already holds a generation no handle has been given yet, and no handle51// matches it.52kayan_alive: proc(w: *kayan_world, e: kayan_entity) -> b32 = {53 return e.index < w*.count && w*.generations[e.index] == e.generation;54}55 56// Generations start at 1. So the zeroed handle, 0:0, never names a live57// entity, and `{}` can stand for "no entity".58kayan_spawn: proc(w: *kayan_world) -> kayan_entity = {59 e: kayan_entity = {};60 if (w*.free_count > 0) {61 w*.free_count -= 1;62 e.index = w*.free[w*.free_count];63 } else {64 if (w*.count == 8) {65 printfmt("the world is full\n");66 exit(1);67 }68 e.index = w*.count;69 w*.count += 1;70 w*.generations[e.index] = 1;71 }72 e.generation = w*.generations[e.index];73 return e;74}75 76// A dead handle reports `dead` and changes nothing, so despawning twice is77// harmless.78kayan_despawn: proc(w: *kayan_world, e: kayan_entity) -> kayan_status = {79 if (!kayan_alive(w, e)) {80 return kayan_status.dead;81 }82 w*.has_position[e.index] = false;83 w*.has_velocity[e.index] = false;84 w*.has_health[e.index] = false;85 w*.generations[e.index] += 1;86 w*.free[w*.free_count] = e.index;87 w*.free_count += 1;88 return kayan_status.ok;89}90 91set_position: proc(w: *kayan_world, e: kayan_entity, p: vec2) -> void = {92 w*.positions[e.index] = p;93 w*.has_position[e.index] = true;94}95 96set_velocity: proc(w: *kayan_world, e: kayan_entity, v: vec2) -> void = {97 w*.velocities[e.index] = v;98 w*.has_velocity[e.index] = true;99}100 101set_health: proc(w: *kayan_world, e: kayan_entity, points: f32) -> void = {102 w*.healths[e.index] = points;103 w*.has_health[e.index] = true;104}105 106// Reading through a handle checks it first. A dead handle, or an entity107// without a position, gives null instead of whatever is at the number.108get_position: proc(w: *kayan_world, e: kayan_entity) -> *vec2 = {109 if (!kayan_alive(w, e) || !w*.has_position[e.index]) {110 return null;111 }112 return w*.positions[e.index]&;113}114 115movement_system: proc(w: *kayan_world, dt: f32) -> u32 = {116 moved: u32 = 0;117 for (i: u32 = 0; i < w*.count; i += 1) {118 if (w*.has_position[i] && w*.has_velocity[i]) {119 w*.positions[i].x += w*.velocities[i].x * dt;120 w*.positions[i].y += w*.velocities[i].y * dt;121 moved += 1;122 }123 }124 return moved;125}126 127print_world: proc(w: *kayan_world) -> void = {128 for (i: u32 = 0; i < w*.count; i += 1) {129 if (!w*.has_position[i] && !w*.has_velocity[i] && !w*.has_health[i]) {130 printf(" entity %u (free, next generation %u)\n", i, w*.generations[i]);131 continue;132 }133 printf(" entity %u:%u", i, w*.generations[i]);134 if (w*.has_position[i]) {135 printf(" position (%5.2f, %5.2f)", cast(w*.positions[i].x, f64), cast(w*.positions[i].y, f64));136 } else {137 printf(" position - ");138 }139 if (w*.has_velocity[i]) {140 printf(" velocity (%5.2f, %5.2f)", cast(w*.velocities[i].x, f64), cast(w*.velocities[i].y, f64));141 } else {142 printf(" velocity - ");143 }144 if (w*.has_health[i]) {145 printf(" health %.0f", cast(w*.healths[i], f64));146 } else {147 printf(" health -");148 }149 printf("\n");150 }151}152 153// The turret's aim: the target's position, or a new target if it is gone.154aim: proc(w: *kayan_world, target: *kayan_entity, fallback: kayan_entity) -> void = {155 p: *vec2 = get_position(w, target*);156 if (p == null) {157 printfmt("the turret's target {}:{} is dead; it picks {}:{} instead\n",158 target*.index, target*.generation, fallback.index, fallback.generation);159 target* = fallback;160 p = get_position(w, target*);161 }162 printf("the turret aims at %u:%u, at (%.2f, %.2f)\n", target*.index, target*.generation,163 cast(p*.x, f64), cast(p*.y, f64));164}165 166main: proc() -> i32 = {167 w: kayan_world = {};168 169 turret: kayan_entity = kayan_spawn(w&);170 set_position(w&, turret, { .x = 0.0, .y = 0.0 });171 set_health(w&, turret, 500.0);172 173 drone_a: kayan_entity = kayan_spawn(w&);174 set_position(w&, drone_a, { .x = 10.0, .y = 0.0 });175 set_velocity(w&, drone_a, { .x = -1.0, .y = 0.5 });176 set_health(w&, drone_a, 100.0);177 178 drone_b: kayan_entity = kayan_spawn(w&);179 set_position(w&, drone_b, { .x = 0.0, .y = 10.0 });180 set_velocity(w&, drone_b, { .x = 0.5, .y = -1.0 });181 set_health(w&, drone_b, 100.0);182 183 target: kayan_entity = drone_a;184 print_world(w&);185 aim(w&, target&, drone_b);186 187 printfmt("drone {}:{} is destroyed and despawned\n", drone_a.index, drone_a.generation);188 _ = kayan_despawn(w&, drone_a);189 190 bullet: kayan_entity = kayan_spawn(w&);191 set_position(w&, bullet, { .x = 0.0, .y = 0.0 });192 set_velocity(w&, bullet, { .x = 4.0, .y = 4.0 });193 printfmt("the turret fires: the bullet is {}:{}, the same number in its next life\n",194 bullet.index, bullet.generation);195 _ = movement_system(w&, 0.5);196 print_world(w&);197 198 // The same moment as the end of 1.2, and this time the handle knows.199 aim(w&, target&, drone_b);200 201 status: kayan_status = kayan_despawn(w&, drone_a);202 printfmt("despawning {}:{} again: {}\n", drone_a.index, drone_a.generation, status);203 204 none: kayan_entity = {};205 printfmt("the zeroed handle {}:{} is alive: {}\n", none.index, none.generation,206 kayan_alive(w&, none) ? "yes" : "no");207 return 0;208}What generations prints. tools/check.py fails the chapter if a single byte differs.
entity 0:1 position ( 0.00, 0.00) velocity - health 500
entity 1:1 position (10.00, 0.00) velocity (-1.00, 0.50) health 100
entity 2:1 position ( 0.00, 10.00) velocity ( 0.50, -1.00) health 100
the turret aims at 1:1, at (10.00, 0.00)
drone 1:1 is destroyed and despawned
the turret fires: the bullet is 1:2, the same number in its next life
entity 0:1 position ( 0.00, 0.00) velocity - health 500
entity 1:2 position ( 2.00, 2.00) velocity ( 4.00, 4.00) health -
entity 2:1 position ( 0.25, 9.50) velocity ( 0.50, -1.00) health 100
the turret's target 1:1 is dead; it picks 2:1 instead
the turret aims at 2:1, at (0.25, 9.50)
despawning 1:1 again: dead
the zeroed handle 0:0 is alive: no
Build an ECS · part 1, entities · 1.4
The entity index
ecs/src/1.entities/1.4.entity_index/entity_index.rin
In the dojo: 2. swap-remove and the back-reference, 3. the slot map, each on its own, to watch and to practise.
Handles fixed identity. Two costs from 1.1 remain: every array is as long as the most entities, and every system walks all of it, skipping holes. This chapter packs the data: live entities in rows 0 to count - 1, no holes, so a system is a straight loop over the rows. That's the stream lesson 1 measured as the fast case.
Packing has a consequence that the rest of the course is built around: an entity's data no longer sits at its number, and it moves.
To keep this chapter about the index, every entity is the same kind of thing: a particle with a position and a velocity. Part 2 brings back different kinds, each packed in a table of its own.
Step 1: two halves
kayan_slot: struct = {
generation: u32;
row: u32;
}
kayan_world: struct = {
slot_count: u32;
slots: [8]kayan_slot;
free: [8]u32;
free_count: u32;
count: u32;
entities: [8]kayan_entity;
positions: [8]vec2;
velocities: [8]vec2;
}
The world now has two halves:
- The slots, indexed by the entity's number. A number never moves, so this is where a handle looks first. A slot holds the generation from 1.3, and now also the row the entity's data is in.
- The rows, packed.
countof them, each with its components side by side in the arrays, andentities[row]saying which entity owns the row.
The two point at each other: slot to row, row to entity. That two-way link is called the entity index.
Step 2: spawning appends a row
row: u32 = w*.count;
w*.count += 1;
w*.entities[row] = e;
w*.positions[row] = p;
w*.velocities[row] = v;
w*.slots[e.index].row = row;
return e;
The number comes from the free list or the end, exactly as in 1.3. The data goes in the next row, which is always the end of the packed rows. The slot records where.
Step 3: despawning swap-removes
To keep the rows packed, a hole can't be left behind. The last row moves into it, called swap-remove:
kayan_despawn: proc(w: *kayan_world, e: kayan_entity) -> kayan_status = {
if (!kayan_alive(w, e)) {
return kayan_status.dead;
}
row: u32 = w*.slots[e.index].row;
last: u32 = w*.count - 1;
if (row != last) {
moved: kayan_entity = w*.entities[last];
w*.entities[row] = moved;
w*.positions[row] = w*.positions[last];
w*.velocities[row] = w*.velocities[last];
w*.slots[moved.index].row = row;
}
w*.count -= 1;
w*.slots[e.index].generation += 1;
w*.free[w*.free_count] = e.index;
w*.free_count += 1;
return kayan_status.ok;
}
The line that matters is this one:
w*.slots[moved.index].row = row;
The entity that owned the last row now lives somewhere else, and its slot must say so. entities[last] is how we know whose slot to fix: without that column, a moved row would be anonymous. Forget the line and that entity's handle still finds a row, just the wrong one, and nothing crashes.
Swap-remove costs one row copy, however many rows there are. Removing by shifting everything after the hole down would keep the order, but cost a copy of every later row. ECSs don't promise order, so they take the cheap one.
Step 4: reading through the index
get_position: proc(w: *kayan_world, e: kayan_entity) -> *vec2 = {
if (!kayan_alive(w, e)) {
return null;
}
return w*.positions[w*.slots[e.index].row]&;
}
Handle to slot to row. That's one extra lookup compared with 1.3, and it's the price of packing. Systems don't pay it, because they don't go through handles:
movement_system: proc(w: *kayan_world, dt: f32) -> void = {
for (row: u32 = 0; row < w*.count; row += 1) {
w*.positions[row].x += w*.velocities[row].x * dt;
w*.positions[row].y += w*.velocities[row].y * dt;
}
}
No flags, no holes, no handles. Every row is a live particle. This is the loop the rest of the course works to keep.
Step 5: checking the invariant
Rows move all the time, and one missed slot update corrupts the world silently. So the world can check itself:
kayan_verify: proc(w: *kayan_world) -> u32 = {
problems: u32 = 0;
for (row: u32 = 0; row < w*.count; row += 1) {
e: kayan_entity = w*.entities[row];
if (!kayan_alive(w, e)) {
printfmt("kayan_verify: row {} holds dead entity {}:{}\n", row, e.index, e.generation);
problems += 1;
} else if (w*.slots[e.index].row != row) {
printfmt("kayan_verify: {}:{} is in row {}, its slot says row {}\n",
e.index, e.generation, row, w*.slots[e.index].row);
problems += 1;
}
}
live: u32 = w*.slot_count - w*.free_count;
if (live != w*.count) {
printfmt("kayan_verify: {} rows, {} live entities\n", w*.count, live);
problems += 1;
}
return problems;
}
The invariant: every row belongs to a live entity whose slot points back at that row, and there are as many rows as live entities. Every structural change in this course has to keep it true. When something goes wrong later, run kayan_verify first.
Run it
spawned 5 particles
row 0 0:1 position ( 0.00, 0.00)
row 1 1:1 position ( 1.00, 0.00)
...
row 4 4:1 position ( 4.00, 0.00)
slots: 0 -> row 0 1 -> row 1 2 -> row 2 3 -> row 3 4 -> row 4
despawn 1:1, in row 1: the last row moves into the hole
row 0 0:1 position ( 0.00, 0.00)
row 1 4:1 position ( 4.00, 0.00)
row 2 2:1 position ( 2.00, 0.00)
row 3 3:1 position ( 3.00, 0.00)
slots: 0 -> row 0 1 free 2 -> row 2 3 -> row 3 4 -> row 1
despawn 4:1, which the last despawn moved: its handle still finds it
...
kayan_verify: 0 problems
Follow 4:1: it starts in row 4, moves to row 1 when 1:1 is removed, and its handle still finds it there because its slot was updated.
Part 1 in one picture
handle 4:1 ──> slots[4] { generation 1, row 1 } ──> rows[1] { owner 4:1, position, velocity }
- Handles never change: they're what the rest of the game keeps.
- Slots never move: one per number, holding the generation and the row.
- Rows move all the time, and stay packed so systems can stream them.
Part 2 keeps exactly this, and generalizes the one kind of row into tables, one per combination of components.
Try
- Delete the line
w*.slots[moved.index].row = row;, run it, and read whatkayan_verifyreports. Then look at whatget_positionreturns for the moved entity. - Despawn every particle, then spawn three. Which numbers and rows do they get?
- Add a
kayan_get_velocityand use it to stop one particle by handle. - Write a
print_rowsthat prints rows in handle order instead of row order. What does it have to do?
1// 1.4 The entity index2//3// So far every array has room for every number, and systems walk all of them,4// skipping holes. This chapter packs the data instead. Live entities sit in5// rows 0 to count-1 with no holes, and a system is a straight loop over the6// rows: the stream lesson 1 measured.7//8// Packing means an entity's data no longer sits at its number, and it moves.9// Removing a row moves the last row into the hole (swap-remove), so whatever10// was last changes row. Two things keep track:11//12// * each row records which entity owns it, `entities[row]`;13// * each number has a slot recording its generation and its current row.14//15// A handle finds its data through its slot. When a row moves, the moved16// entity's slot is updated. That one line is the invariant everything else in17// this course relies on, and `kayan_verify` checks it.18//19// To keep the chapter about the index, every entity here is the same kind of20// thing: a particle with a position and a velocity. Part 2 brings back21// different kinds, each packed in a table of its own.22 23import "std/print.rin"24 25vec2: struct = {26 x: f32;27 y: f32;28}29 30kayan_entity: struct = {31 index: u32;32 generation: u32;33}34 35kayan_status: enum[must_use] = {36 ok,37 dead,38}39 40// Where a number's entity is: its generation, and the row its data is in.41kayan_slot: struct = {42 generation: u32;43 row: u32;44}45 46// Two halves. The slots, indexed by the entity's number, which never moves. The47// rows, packed, indexed by row, which moves: `count` of them, each with its48// owner and its components side by side.49kayan_world: struct = {50 slot_count: u32;51 slots: [8]kayan_slot;52 free: [8]u32;53 free_count: u32;54 55 count: u32;56 entities: [8]kayan_entity;57 positions: [8]vec2;58 velocities: [8]vec2;59}60 61kayan_alive: proc(w: *kayan_world, e: kayan_entity) -> b32 = {62 return e.index < w*.slot_count && w*.slots[e.index].generation == e.generation;63}64 65// A number, as in 1.3, and the next row at the end of the packed rows.66kayan_spawn: proc(w: *kayan_world, p: vec2, v: vec2) -> kayan_entity = {67 e: kayan_entity = {};68 if (w*.free_count > 0) {69 w*.free_count -= 1;70 e.index = w*.free[w*.free_count];71 } else {72 if (w*.slot_count == 8) {73 printfmt("the world is full\n");74 exit(1);75 }76 e.index = w*.slot_count;77 w*.slot_count += 1;78 w*.slots[e.index].generation = 1;79 }80 e.generation = w*.slots[e.index].generation;81 82 row: u32 = w*.count;83 w*.count += 1;84 w*.entities[row] = e;85 w*.positions[row] = p;86 w*.velocities[row] = v;87 w*.slots[e.index].row = row;88 return e;89}90 91// Swap-remove: the last row moves into the hole, so the rows stay packed. The92// entity that owned the last row now lives at `row`, and its slot must say so.93kayan_despawn: proc(w: *kayan_world, e: kayan_entity) -> kayan_status = {94 if (!kayan_alive(w, e)) {95 return kayan_status.dead;96 }97 row: u32 = w*.slots[e.index].row;98 last: u32 = w*.count - 1;99 if (row != last) {100 moved: kayan_entity = w*.entities[last];101 w*.entities[row] = moved;102 w*.positions[row] = w*.positions[last];103 w*.velocities[row] = w*.velocities[last];104 w*.slots[moved.index].row = row;105 }106 w*.count -= 1;107 w*.slots[e.index].generation += 1;108 w*.free[w*.free_count] = e.index;109 w*.free_count += 1;110 return kayan_status.ok;111}112 113// Handle, then slot, then row: the data is wherever the row is now.114get_position: proc(w: *kayan_world, e: kayan_entity) -> *vec2 = {115 if (!kayan_alive(w, e)) {116 return null;117 }118 return w*.positions[w*.slots[e.index].row]&;119}120 121// No flags and no holes: every row is a live particle. This is a stream over122// two packed arrays.123movement_system: proc(w: *kayan_world, dt: f32) -> void = {124 for (row: u32 = 0; row < w*.count; row += 1) {125 w*.positions[row].x += w*.velocities[row].x * dt;126 w*.positions[row].y += w*.velocities[row].y * dt;127 }128}129 130// The invariant, checked: every row belongs to a live entity whose slot points131// back at that row, and there are as many rows as live entities. Returns how132// many checks failed, printing each one.133kayan_verify: proc(w: *kayan_world) -> u32 = {134 problems: u32 = 0;135 for (row: u32 = 0; row < w*.count; row += 1) {136 e: kayan_entity = w*.entities[row];137 if (!kayan_alive(w, e)) {138 printfmt("kayan_verify: row {} holds dead entity {}:{}\n", row, e.index, e.generation);139 problems += 1;140 } else if (w*.slots[e.index].row != row) {141 printfmt("kayan_verify: {}:{} is in row {}, its slot says row {}\n",142 e.index, e.generation, row, w*.slots[e.index].row);143 problems += 1;144 }145 }146 live: u32 = w*.slot_count - w*.free_count;147 if (live != w*.count) {148 printfmt("kayan_verify: {} rows, {} live entities\n", w*.count, live);149 problems += 1;150 }151 return problems;152}153 154print_rows: proc(w: *kayan_world) -> void = {155 for (row: u32 = 0; row < w*.count; row += 1) {156 e: kayan_entity = w*.entities[row];157 printf(" row %u %u:%u position (%5.2f, %5.2f)\n", row, e.index, e.generation,158 cast(w*.positions[row].x, f64), cast(w*.positions[row].y, f64));159 }160 // A slot is in use when the row it points at is owned by its number, in its161 // current generation. A free slot still holds the row it last had.162 printf(" slots:");163 for (i: u32 = 0; i < w*.slot_count; i += 1) {164 s: kayan_slot = w*.slots[i];165 owner: kayan_entity = w*.entities[s.row];166 if (s.row < w*.count && owner.index == i && owner.generation == s.generation) {167 printf(" %u -> row %u", i, s.row);168 } else {169 printf(" %u free", i);170 }171 }172 printf("\n");173}174 175main: proc() -> i32 = {176 w: kayan_world = {};177 p: [5]kayan_entity = {};178 for (i: u32 = 0; i < 5; i += 1) {179 p[i] = kayan_spawn(w&, { .x = cast(i, f32), .y = 0.0 }, { .x = 0.0, .y = 1.0 });180 }181 printfmt("spawned 5 particles\n");182 print_rows(w&);183 184 printfmt("despawn {}:{}, in row 1: the last row moves into the hole\n", p[1].index, p[1].generation);185 _ = kayan_despawn(w&, p[1]);186 print_rows(w&);187 188 printfmt("despawn {}:{}, which the last despawn moved: its handle still finds it\n",189 p[4].index, p[4].generation);190 _ = kayan_despawn(w&, p[4]);191 print_rows(w&);192 193 q: kayan_entity = kayan_spawn(w&, { .x = 9.0, .y = 0.0 }, { .x = 0.0, .y = 1.0 });194 printfmt("spawn {}:{}: it reuses a number and takes the next row\n", q.index, q.generation);195 print_rows(w&);196 197 movement_system(w&, 0.5);198 movement_system(w&, 0.5);199 printfmt("two steps of movement, a straight loop over {} rows\n", w.count);200 print_rows(w&);201 202 at: *vec2 = get_position(w&, p[2]);203 printf("%u:%u is at (%.2f, %.2f), wherever its row is\n", p[2].index, p[2].generation,204 cast(at*.x, f64), cast(at*.y, f64));205 printfmt("{}:{} has a position: {}\n", p[4].index, p[4].generation,206 get_position(w&, p[4]) != null ? "yes" : "no, it is dead");207 printfmt("kayan_verify: {} problems\n", kayan_verify(w&));208 return 0;209}What entity_index prints. tools/check.py fails the chapter if a single byte differs.
spawned 5 particles
row 0 0:1 position ( 0.00, 0.00)
row 1 1:1 position ( 1.00, 0.00)
row 2 2:1 position ( 2.00, 0.00)
row 3 3:1 position ( 3.00, 0.00)
row 4 4:1 position ( 4.00, 0.00)
slots: 0 -> row 0 1 -> row 1 2 -> row 2 3 -> row 3 4 -> row 4
despawn 1:1, in row 1: the last row moves into the hole
row 0 0:1 position ( 0.00, 0.00)
row 1 4:1 position ( 4.00, 0.00)
row 2 2:1 position ( 2.00, 0.00)
row 3 3:1 position ( 3.00, 0.00)
slots: 0 -> row 0 1 free 2 -> row 2 3 -> row 3 4 -> row 1
despawn 4:1, which the last despawn moved: its handle still finds it
row 0 0:1 position ( 0.00, 0.00)
row 1 3:1 position ( 3.00, 0.00)
row 2 2:1 position ( 2.00, 0.00)
slots: 0 -> row 0 1 free 2 -> row 2 3 -> row 1 4 free
spawn 4:2: it reuses a number and takes the next row
row 0 0:1 position ( 0.00, 0.00)
row 1 3:1 position ( 3.00, 0.00)
row 2 2:1 position ( 2.00, 0.00)
row 3 4:2 position ( 9.00, 0.00)
slots: 0 -> row 0 1 free 2 -> row 2 3 -> row 1 4 -> row 3
two steps of movement, a straight loop over 4 rows
row 0 0:1 position ( 0.00, 1.00)
row 1 3:1 position ( 3.00, 1.00)
row 2 2:1 position ( 2.00, 1.00)
row 3 4:2 position ( 9.00, 1.00)
slots: 0 -> row 0 1 free 2 -> row 2 3 -> row 1 4 -> row 3
2:1 is at (2.00, 1.00), wherever its row is
4:1 has a position: no, it is dead
kayan_verify: 0 problems
Build an ECS · part 2, components · 2.1
Components and signatures
ecs/src/2.components/2.1.components/components.rin
In the dojo: 6. bitsets, on its own, to watch and to practise.
Part 1 hard-coded its components: a positions array, a velocities array, a flag for each. Adding a component meant editing the world, kayan_spawn, kayan_despawn and every printer. Part 2 builds a world that works for components it has never heard of. This chapter lays the ground with four ideas:
- A component is a plain struct.
- Each component type has an id, a small number fixed at compile time.
- The world keeps a registry: per id, the component's size, alignment and name, which is all it needs to store bytes it doesn't understand.
- A set of components is a signature: one bit per id in a
u64.
There are no entities in this chapter. 2.3 puts the two halves together.
Step 1: components are plain structs
position: struct = { at: vec2; }
velocity: struct = { per_second: vec2; }
health: struct = { points: f32; max: f32; }
damage: struct = { amount: f32; }
lifetime: struct = { seconds: f32; }
burning: struct = { per_second: f32; }
turret: struct = { cooldown: f32; timer: f32; }
No procs, no pointers to other entities, no idea that a world exists. That's deliberate. Behaviour belongs to systems, and data that knows nothing can be copied, moved and stored as raw bytes, which the world is about to do.
Why position { at: vec2 } rather than using vec2 directly? Because a component's type is how the world tells components apart. Position and velocity are both two floats, but they're different things, and making them different types is what lets you ask for one and not the other.
Step 2: an id per component type
component: enum = {
position,
velocity,
health,
damage,
lifetime,
burning,
turret,
}
The enum gives each component a number. Then one line per type connects the type to its number:
component_of: proc<position>() -> component = { return component.position; }
component_of: proc<velocity>() -> component = { return component.velocity; }
Each line is a specialization of the generic proc component_of<T> for one type. component_of<health>() is resolved by the compiler to component.health. Nothing is looked up at run time; the id is a constant in the generated code.
Step 3: the world speaks ids, never types
The world can't be generic over component types. Which combinations exist is only known while the game runs (the "untyped core" from our earlier discussion). So the world works in ids and bits, and a small typed layer translates:
kayan_bit: proc(id: u32) -> u64 = {
return 1'u64 shl cast(id, u64);
}
// The typed layer: from a type to the id and bit the world works in.
kayan_id_of: proc<T>() -> u32 = {
return cast(component_of<T>(), u32);
}
kayan_mask_of: proc<T>() -> u64 = {
return kayan_bit(kayan_id_of<T>());
}
Game code says kayan_mask_of<velocity>(), and the world only ever sees 2, a bit.
Step 4: the registry
To store a component as bytes, the world needs three facts about it: how many bytes, what alignment, and a name for printing. rin's compiler already wrote all three down. position<> is the reflect record for position, data the compiler puts into the program. So the registry is an array of pointers to reflect records, indexed by id:
kayan_world: struct = {
infos: [64]*const reflect;
component_count: u32;
}
kayan_register: proc(w: *kayan_world, id: u32, info: *const reflect) -> void = {
if (id >= 64) {
printfmt("kayan: component id {} is past the 64 a signature can hold\n", id);
exit(1);
}
w*.infos[id] = info;
if (id + 1 > w*.component_count) {
w*.component_count = id + 1;
}
}
register_components: proc(w: *kayan_world) -> void = {
kayan_register(w, kayan_id_of<position>(), position<>&);
kayan_register(w, kayan_id_of<velocity>(), velocity<>&);
Each line names the type twice, once for the id and once for the record. A generic kayan_register_type<T> that took T<>& would be nicer, but rin doesn't yet allow T<> on a generic parameter, so registration names each type.
Step 5: signatures
A set of components is a u64 with a bit set per component in the set. What a thing is is a signature:
drone: u64 = kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<health>();
Every question you'd ask about sets is a bit operation:
kayan_contains: proc(signature: u64, wanted: u64) -> b32 = {
return (signature & wanted) == wanted;
}
"Does this drone have what movement wants?" is kayan_contains(drone, moves), one AND and one compare. Adding a component is |, and removing one is & ~:
put_out: u64 = burning_drone & ~kayan_mask_of<burning>();
64 bits means at most 64 component types per world, which is plenty for learning and for most games. Larger ECSs use wider bitsets or sorted id lists; the idea is the same.
Run it
the registry: 7 components
id name size align
0 position 8 4
1 velocity 8 4
2 health 8 4
3 damage 4 4
4 lifetime 4 4
5 burning 4 4
6 turret 8 4
signatures, one bit per component, highest id first:
drone 0000111 {position velocity health}
bullet 0011011 {position velocity damage lifetime}
turret 1000001 {position turret}
burning drone 0100111 {position velocity health burning}
movement wants {position velocity}:
drone yes, bullet yes, turret no, burning drone yes
a burning drone with burning removed is a drone again: yes
Read the bits as a table: each column is a component, each row a kind of thing. That table is the next three chapters. Each distinct row will become a table of its own.
Try
- Add a
shield: struct = { points: f32; }component. What do you have to touch? (Three lines.) Print a "shielded drone" signature. - Print
kayan_contains(moves, drone)(arguments swapped). Why is the answer different? - Reorder the
componentenum. Which outputs change, and which don't?
1// 2.1 Components and signatures2//3// Part 1 hard-coded its components: a positions array, a velocities array, a4// flag for each. Every new component meant editing the world. Part 2 makes the5// world work for components it has never heard of, and this chapter lays the6// ground: what a component is, how the world tells components apart without7// knowing their types, and how a set of components is written down.8//9// * A component is a plain struct.10// * Each component type gets an id, a small number, decided at compile time.11// * The world keeps a registry: for each id, the component's reflect record,12// which says its name, size and alignment. That is all the world needs to13// store, copy and move a component it knows nothing else about.14// * A set of components is a signature: a u64 with bit `id` set for each15// component in the set. Asking whether a set contains another is one AND.16//17// No entities in this chapter. 2.3 puts the two halves together.18 19import "std/print.rin"20 21vec2: struct = {22 x: f32;23 y: f32;24}25 26// The components. Plain data, nothing else: no procs, no pointers to other27// entities, no knowledge of the world.28position: struct = { at: vec2; }29velocity: struct = { per_second: vec2; }30health: struct = { points: f32; max: f32; }31damage: struct = { amount: f32; }32lifetime: struct = { seconds: f32; }33burning: struct = { per_second: f32; }34turret: struct = { cooldown: f32; timer: f32; }35 36// One id per component type: its position in this enum.37component: enum = {38 position,39 velocity,40 health,41 damage,42 lifetime,43 burning,44 turret,45}46 47// From a type to its id, at compile time. Each line is a specialization of the48// generic proc for one type; calling component_of<health>() is resolved by the49// compiler to `component.health`, with nothing looked up at run time.50component_of: proc<position>() -> component = { return component.position; }51component_of: proc<velocity>() -> component = { return component.velocity; }52component_of: proc<health>() -> component = { return component.health; }53component_of: proc<damage>() -> component = { return component.damage; }54component_of: proc<lifetime>() -> component = { return component.lifetime; }55component_of: proc<burning>() -> component = { return component.burning; }56component_of: proc<turret>() -> component = { return component.turret; }57 58// ---------------------------------------------------------------------------59// The world's side: ids and bits, never types60// ---------------------------------------------------------------------------61 62// The registry. The world stores components as bytes, so per id it needs to63// know how many bytes, at what alignment, and a name for printing. A reflect64// record has all three.65kayan_world: struct = {66 infos: [64]*const reflect;67 component_count: u32;68}69 70// One bit per component. A signature is a u64, so a world has at most 64.71kayan_bit: proc(id: u32) -> u64 = {72 return 1'u64 shl cast(id, u64);73}74 75// The typed layer: from a type to the id and bit the world works in.76kayan_id_of: proc<T>() -> u32 = {77 return cast(component_of<T>(), u32);78}79 80kayan_mask_of: proc<T>() -> u64 = {81 return kayan_bit(kayan_id_of<T>());82}83 84kayan_register: proc(w: *kayan_world, id: u32, info: *const reflect) -> void = {85 if (id >= 64) {86 printfmt("kayan: component id {} is past the 64 a signature can hold\n", id);87 exit(1);88 }89 w*.infos[id] = info;90 if (id + 1 > w*.component_count) {91 w*.component_count = id + 1;92 }93}94 95 96// A signature as names, `{position velocity}`.97kayan_print_signature: proc(w: *kayan_world, signature: u64) -> void = {98 printf("{");99 first: b32 = true;100 for (id: u32 = 0; id < w*.component_count; id += 1) {101 if ((signature & kayan_bit(id)) != 0) {102 printf(first ? "%s" : " %s", w*.infos[id]*.name);103 first = false;104 }105 }106 printf("}");107}108 109// A signature as bits, highest component first, so each column of the output110// is one component.111print_bits: proc(w: *kayan_world, signature: u64) -> void = {112 id: u32 = w*.component_count;113 while (id > 0) {114 id -= 1;115 printf((signature & kayan_bit(id)) != 0 ? "1" : "0");116 }117}118 119// Does a set contain every component of another? One AND and one compare.120kayan_contains: proc(signature: u64, wanted: u64) -> b32 = {121 return (signature & wanted) == wanted;122}123 124// `position<>` is the compiler's reflect record for position: its name, size125// and alignment, written into the program as data. Registering hands the world126// that record under position's id.127register_components: proc(w: *kayan_world) -> void = {128 kayan_register(w, kayan_id_of<position>(), position<>&);129 kayan_register(w, kayan_id_of<velocity>(), velocity<>&);130 kayan_register(w, kayan_id_of<health>(), health<>&);131 kayan_register(w, kayan_id_of<damage>(), damage<>&);132 kayan_register(w, kayan_id_of<lifetime>(), lifetime<>&);133 kayan_register(w, kayan_id_of<burning>(), burning<>&);134 kayan_register(w, kayan_id_of<turret>(), turret<>&);135}136 137main: proc() -> i32 = {138 w: kayan_world = {};139 register_components(w&);140 141 printfmt("the registry: {} components\n", w.component_count);142 printfmt(" id name size align\n");143 for (id: u32 = 0; id < w.component_count; id += 1) {144 info: *const reflect = w.infos[id];145 printf(" %2u %-9s %4u %5u\n", id, info*.name, cast(info*.size, u32), cast(info*.align, u32));146 }147 148 // What a thing is, written as the set of its components.149 drone: u64 = kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<health>();150 bullet: u64 = kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<damage>()151 | kayan_mask_of<lifetime>();152 gun: u64 = kayan_mask_of<position>() | kayan_mask_of<turret>();153 burning_drone: u64 = drone | kayan_mask_of<burning>();154 155 printfmt("signatures, one bit per component, highest id first:\n");156 printf(" drone "); print_bits(w&, drone); printf(" "); kayan_print_signature(w&, drone); printf("\n");157 printf(" bullet "); print_bits(w&, bullet); printf(" "); kayan_print_signature(w&, bullet); printf("\n");158 printf(" turret "); print_bits(w&, gun); printf(" "); kayan_print_signature(w&, gun); printf("\n");159 printf(" burning drone "); print_bits(w&, burning_drone); printf(" "); kayan_print_signature(w&, burning_drone); printf("\n");160 161 // Which of them would a movement system want? Everything that contains162 // {position velocity}.163 moves: u64 = kayan_mask_of<position>() | kayan_mask_of<velocity>();164 printf("movement wants "); kayan_print_signature(w&, moves); printf(":\n");165 printfmt(" drone {}, bullet {}, turret {}, burning drone {}\n",166 kayan_contains(drone, moves) ? "yes" : "no", kayan_contains(bullet, moves) ? "yes" : "no",167 kayan_contains(gun, moves) ? "yes" : "no", kayan_contains(burning_drone, moves) ? "yes" : "no");168 169 // Adding and removing a component is setting and clearing its bit.170 put_out: u64 = burning_drone & ~kayan_mask_of<burning>();171 printfmt("a burning drone with burning removed is a drone again: {}\n", put_out == drone ? "yes" : "no");172 return 0;173}What components prints. tools/check.py fails the chapter if a single byte differs.
the registry: 7 components
id name size align
0 position 8 4
1 velocity 8 4
2 health 8 4
3 damage 4 4
4 lifetime 4 4
5 burning 4 4
6 turret 8 4
signatures, one bit per component, highest id first:
drone 0000111 {position velocity health}
bullet 0011011 {position velocity damage lifetime}
turret 1000001 {position turret}
burning drone 0100111 {position velocity health burning}
movement wants {position velocity}:
drone yes, bullet yes, turret no, burning drone yes
a burning drone with burning removed is a drone again: yes
Build an ECS · part 2, components · 2.2
Columns
ecs/src/2.components/2.2.columns/columns.rin
In the dojo: 4. a type-erased column, 8. index, not pointer, each on its own, to watch and to practise.
1.4 kept components in typed arrays: positions: [8]vec2. A world that works for any component can't, because it doesn't know the types. It knows, from the registry, each component's size and alignment, and that's enough.
A column is one component's array, kept as bytes. Row r is the size bytes starting at r * size. Everything a table does with its data is a byte operation on columns: grow, copy one row over another, zero a new row. None of it depends on the type. The type comes back only at the edge, where game code reads a column as []T.
This chapter builds the column and exercises it by hand. 2.3 puts columns in a table.
Step 1: what a column is
kayan_column: struct = {
data: ^u8;
size: u64;
align: u64;
}
Bytes, plus how many bytes make a row, plus the alignment to allocate with. No count and no capacity: every column of a table has the same number of rows, so the table keeps those once (2.3).
A column for a component is made from its reflect record:
kayan_column_init: proc(info: *const reflect) -> kayan_column = {
c: kayan_column = {};
c.size = info*.size;
c.align = info*.align;
return c;
}
Step 2: rows are byte offsets
kayan_column_row: proc(c: *kayan_column, row: u64) -> *u8 = {
return c*.data[row * c*.size]&;
}
That's the whole addressing scheme: row * size. With it, copying and zeroing rows is memcpy and memset:
kayan_column_copy_row: proc(c: *kayan_column, to: u64, from: u64) -> void = {
memcpy(kayan_column_row(c, to), kayan_column_row(c, from), c*.size);
}
kayan_column_zero_row: proc(c: *kayan_column, row: u64) -> void = {
memset(kayan_column_row(c, row), 0, c*.size);
}
kayan_column_copy_row is swap-remove's copy from 1.4, for any component.
Step 3: growing
kayan_column_grow: proc(arena: *memops_arena, c: *kayan_column, count: u64, capacity: u64) -> void = {
data: ^u8 = cast(memops_arena_push_zero(arena, c*.size * capacity, c*.align), ^u8);
if (count > 0) {
memcpy(data, c*.data, c*.size * count);
}
c*.data = data;
}
A new block for capacity rows, with the rows in use copied over. The old block stays in the arena, unused: an arena frees everything at once, never one allocation. Doubling the capacity each time keeps the waste to at most the size of what's in use, and the copies to a few per table over a whole game.
This is also why pointers into a column go stale: after a grow, the data is somewhere else. 2.5 shows it happening.
Step 4: the typed view
kayan_column_slice: proc<T>(c: *kayan_column, count: u64) -> []T = {
if (c*.size != sizeof(T)) {
printfmt("kayan: a column of {}-byte rows read as a {}-byte type\n", c*.size, sizeof(T));
exit(1);
}
raw: ^T = cast(c*.data, ^T);
return raw[0..count];
}
This is the untyped/typed split in one proc. Below it, bytes. Above it, a []T that game code indexes like any slice, bounds checked in Debug. The size check catches the one mistake the byte level can't: reading a column as the wrong type.
Writing goes through the typed view too:
ps: []position = kayan_column_slice<position>(positions&, count);
hs: []health = kayan_column_slice<health>(healths&, count);
ps[row] = { .at = { .x = cast(i, f32), .y = 1.0 } };
hs[row] = { .points = 50.0 + 10.0 * cast(i, f32), .max = 100.0 };
(Writing kayan_column_slice<position>(...)[row] = ... in one line would be nicer. rin doesn't allow indexing a call's result directly yet, because the bounds check would evaluate the call twice. Hence the locals.)
Step 5: rows stay rows across columns
The program keeps two columns side by side, positions and healths, and moves them together:
last: u64 = count - 1;
kayan_column_copy_row(positions&, 1, last);
kayan_column_copy_row(healths&, 1, last);
count -= 1;
Same operation, each column using its own row size. Row 1 is still one entity's data across both columns. That's a table in all but name.
Run it
a position column has 8-byte rows, a health column 8-byte rows
grew to room for 2 rows
grew to room for 4 rows
grew to room for 8 rows
five rows, written through the typed view:
row 0 position ( 0.00, 1.00) health 50 / 100
row 1 position ( 1.00, 1.00) health 60 / 100
...
row 1's health as the column stores it, bytes: 00 00 70 42 00 00 c8 42
that is 60.0 and 100.0 as little-endian f32s; the column never knew
swap-removed row 1: row 4 moved into it, in both columns
The byte line is the point of the chapter. 00 00 70 42 is 60.0 as an f32, and the column stored it without knowing it was a float, or a health, or anything at all.
Try
- Read the health column as
[]damage(4-byte rows) and run it. Read the message. - Start the capacity at 1 instead of 2. How many grows does it take to fit 5 rows? What if it grew by 1 each time instead of doubling?
- Print the bytes of a position row. Which bytes are
xand which arey?
1// 2.2 Columns2//3// 1.4 kept its components in typed arrays, `positions: [8]vec2`. A world that4// works for any component cannot: it does not know the types. It knows, from5// the registry, each component's size and alignment, and that is enough. A6// column is an array of a component kept as bytes: row `r` is the `size` bytes7// at `r * size`.8//9// Everything a table needs to do with its data is a byte operation on columns:10// grow, copy a row over another (swap-remove), zero a new row. None of it11// depends on the type. The type comes back at the edge, where game code reads12// the column as `[]T`, and that typed view checks the size matches.13//14// This chapter builds the column and exercises it on its own, with the count15// and capacity kept beside it by hand. 2.3 puts columns in a table.16 17import "std/print.rin"18import "std/memops.rin"19 20vec2: struct = {21 x: f32;22 y: f32;23}24 25position: struct = { at: vec2; }26health: struct = { points: f32; max: f32; }27 28component: enum = {29 position,30 health,31}32 33component_of: proc<position>() -> component = { return component.position; }34component_of: proc<health>() -> component = { return component.health; }35 36// ---------------------------------------------------------------------------37// The column: bytes, and how many of them make one row38// ---------------------------------------------------------------------------39 40// A column does not know its count or capacity. A table keeps those once for41// all its columns, since every column of a table has the same number of rows.42kayan_column: struct = {43 data: ^u8;44 size: u64;45 align: u64;46}47 48// An empty column for one component. Its size and alignment come from the49// component's reflect record.50kayan_column_init: proc(info: *const reflect) -> kayan_column = {51 c: kayan_column = {};52 c.size = info*.size;53 c.align = info*.align;54 return c;55}56 57// Row `row`'s first byte.58kayan_column_row: proc(c: *kayan_column, row: u64) -> *u8 = {59 return c*.data[row * c*.size]&;60}61 62// A new block with room for `capacity` rows; the first `count` rows are copied63// over. The old block stays in the arena: an arena frees everything at once,64// never one allocation, and this course accepts the waste.65kayan_column_grow: proc(arena: *memops_arena, c: *kayan_column, count: u64, capacity: u64) -> void = {66 data: ^u8 = cast(memops_arena_push_zero(arena, c*.size * capacity, c*.align), ^u8);67 if (count > 0) {68 memcpy(data, c*.data, c*.size * count);69 }70 c*.data = data;71}72 73kayan_column_copy_row: proc(c: *kayan_column, to: u64, from: u64) -> void = {74 memcpy(kayan_column_row(c, to), kayan_column_row(c, from), c*.size);75}76 77kayan_column_zero_row: proc(c: *kayan_column, row: u64) -> void = {78 memset(kayan_column_row(c, row), 0, c*.size);79}80 81// The typed view: the column's first `count` rows as `[]T`. Reading a column82// as the wrong type would reinterpret its bytes, so the size is checked: the83// one mistake the byte level cannot catch is caught here, at the edge.84kayan_column_slice: proc<T>(c: *kayan_column, count: u64) -> []T = {85 if (c*.size != sizeof(T)) {86 printfmt("kayan: a column of {}-byte rows read as a {}-byte type\n", c*.size, sizeof(T));87 exit(1);88 }89 raw: ^T = cast(c*.data, ^T);90 return raw[0..count];91}92 93// ---------------------------------------------------------------------------94// Using it95// ---------------------------------------------------------------------------96 97print_bytes: proc(c: *kayan_column, row: u64) -> void = {98 bytes: *u8 = kayan_column_row(c, row);99 raw: ^u8 = cast(bytes, ^u8);100 for (i: u64 = 0; i < c*.size; i += 1) {101 printf(" %02x", cast(raw[i], u32));102 }103}104 105print_rows: proc(positions: *kayan_column, healths: *kayan_column, count: u64) -> void = {106 ps: []position = kayan_column_slice<position>(positions, count);107 hs: []health = kayan_column_slice<health>(healths, count);108 for (row: u64 = 0; row < count; row += 1) {109 printf(" row %u position (%5.2f, %5.2f) health %3.0f / %3.0f\n", cast(row, u32),110 cast(ps[row].at.x, f64), cast(ps[row].at.y, f64),111 cast(hs[row].points, f64), cast(hs[row].max, f64));112 }113}114 115main: proc() -> i32 = {116 arena: memops_arena = {};117 memops_arena_initialize(arena&);118 119 // Two columns, made from nothing but reflect records.120 positions: kayan_column = kayan_column_init(position<>&);121 healths: kayan_column = kayan_column_init(health<>&);122 printfmt("a position column has {}-byte rows, a health column {}-byte rows\n",123 positions.size, healths.size);124 125 // Count and capacity, kept by hand here. 2.3 moves them into a table.126 count: u64 = 0;127 capacity: u64 = 0;128 129 for (i: u64 = 0; i < 5; i += 1) {130 if (count == capacity) {131 capacity = capacity == 0 ? 2 : capacity * 2;132 kayan_column_grow(arena&, positions&, count, capacity);133 kayan_column_grow(arena&, healths&, count, capacity);134 printfmt(" grew to room for {} rows\n", capacity);135 }136 row: u64 = count;137 count += 1;138 kayan_column_zero_row(positions&, row);139 kayan_column_zero_row(healths&, row);140 // Back to types to write the values.141 ps: []position = kayan_column_slice<position>(positions&, count);142 hs: []health = kayan_column_slice<health>(healths&, count);143 ps[row] = { .at = { .x = cast(i, f32), .y = 1.0 } };144 hs[row] = { .points = 50.0 + 10.0 * cast(i, f32), .max = 100.0 };145 }146 printfmt("five rows, written through the typed view:\n");147 print_rows(positions&, healths&, count);148 149 printf("row 1's health as the column stores it, bytes:");150 print_bytes(healths&, 1);151 printf("\n that is 60.0 and 100.0 as little-endian f32s; the column never knew\n");152 153 // Swap-remove row 1 from both columns: the same byte copy for both, sized by154 // each column, so a row stays a row across columns.155 last: u64 = count - 1;156 kayan_column_copy_row(positions&, 1, last);157 kayan_column_copy_row(healths&, 1, last);158 count -= 1;159 printfmt("swap-removed row 1: row {} moved into it, in both columns\n", last);160 print_rows(positions&, healths&, count);161 return 0;162}What columns prints. tools/check.py fails the chapter if a single byte differs.
a position column has 8-byte rows, a health column 8-byte rows
grew to room for 2 rows
grew to room for 4 rows
grew to room for 8 rows
five rows, written through the typed view:
row 0 position ( 0.00, 1.00) health 50 / 100
row 1 position ( 1.00, 1.00) health 60 / 100
row 2 position ( 2.00, 1.00) health 70 / 100
row 3 position ( 3.00, 1.00) health 80 / 100
row 4 position ( 4.00, 1.00) health 90 / 100
row 1's health as the column stores it, bytes: 00 00 70 42 00 00 c8 42
that is 60.0 and 100.0 as little-endian f32s; the column never knew
swap-removed row 1: row 4 moved into it, in both columns
row 0 position ( 0.00, 1.00) health 50 / 100
row 1 position ( 4.00, 1.00) health 90 / 100
row 2 position ( 2.00, 1.00) health 70 / 100
row 3 position ( 3.00, 1.00) health 80 / 100
Build an ECS · part 2, components · 2.3
A table
ecs/src/2.components/2.3.tables/tables.rin
In the dojo: 2. swap-remove and the back-reference, 4. a type-erased column, each on its own, to watch and to practise.
The two halves meet. Part 1 built the entity index: handles, slots, packed rows, swap-remove. 2.1 and 2.2 built the component side: ids, a registry, columns of bytes. A table is 1.4's packed rows with the typed arrays replaced by columns, one per component in its signature:
table {position velocity health}
entities [ 0:1 | 3:1 | 2:1 | ... ] who owns each row
columns position [ ... ] velocity [ ... ] health [ ... ]
count, capacity shared by every column
Row r is entity entities[r], and its components are row r of every column. This world still has a single table, so every entity has the same components; 2.4 gives each combination its own.
Step 1: the table
kayan_table: struct = {
signature: u64;
count: u64;
capacity: u64;
entities: ^kayan_entity;
columns: [64]kayan_column;
}
signaturesays which components the table has.columns[id]is in use when the signature has bitid. The rest stay zeroed. 64 slots indexed by id means finding a column is an index, never a search. kayan in C looked columns up by name in a hash map, which is the thing this replaces.countandcapacitybelong to the table, shared by every column and byentities, so they can't disagree.
A table is made from a signature and the registry:
kayan_table_init: proc(w: *kayan_world, signature: u64) -> kayan_table = {
t: kayan_table = {};
t.signature = signature;
for (id: u32 = 0; id < 64; id += 1) {
if ((signature & kayan_bit(id)) != 0) {
if (id >= w*.component_count || w*.infos[id] == null) {
printfmt("kayan: signature {} uses component {}, which was never registered\n", signature, id);
exit(1);
}
t.columns[id] = kayan_column_init(w*.infos[id]);
}
}
return t;
}
The signature is the column list. In your C kayan, the signature and the columns were written separately and could disagree (archetype D had position columns its signature didn't mention). Here one is made from the other.
Step 2: rows, for every column at once
Adding a row grows every column together when the table is full, then zeroes the new row in each:
kayan_table_add_row: proc(w: *kayan_world, t: *kayan_table, e: kayan_entity) -> u64 = {
if (t*.count == t*.capacity) {
kayan_table_grow(w, t);
}
row: u64 = t*.count;
t*.count += 1;
t*.entities[row] = e;
for (id: u32 = 0; id < w*.component_count; id += 1) {
if (kayan_table_has(t, id)) {
kayan_column_zero_row(t*.columns[id]&, row);
}
}
return row;
}
Removing a row is 1.4's swap-remove, for every column:
kayan_table_remove_row: proc(w: *kayan_world, t: *kayan_table, row: u64) -> void = {
last: u64 = t*.count - 1;
if (row != last) {
for (id: u32 = 0; id < w*.component_count; id += 1) {
if (kayan_table_has(t, id)) {
kayan_column_copy_row(t*.columns[id]&, row, last);
}
}
moved: kayan_entity = t*.entities[last];
t*.entities[row] = moved;
vec<kayan_slot>at(w*.slots&, cast(moved.index, u64))*.row = cast(row, u32);
}
t*.count = last;
}
Compare it with 1.4's kayan_despawn: the same moves and the same slot fix, but over columns of bytes instead of named arrays. This proc works for any table that will ever exist.
Step 3: the typed view of a table
kayan_slice: proc<T>(t: *kayan_table) -> []T = {
id: u32 = kayan_id_of<T>();
if (!kayan_table_has(t, id)) {
printfmt("kayan: the table has no column for component {}\n", id);
exit(1);
}
return kayan_column_slice<T>(t*.columns[id]&, t*.count);
}
kayan_slice<position>(t) is the table's position column as a []position, cut to the rows in use. Systems will live on this in Part 3: get the slices, loop over rows.
Step 4: the world
kayan_world: struct = {
arena: *memops_arena;
infos: [64]*const reflect;
component_count: u32;
slots: vec<kayan_slot>;
free: vec<u32>;
live: u64;
table: kayan_table;
}
The registry from 2.1, the slots and free list from Part 1 (as vecs now, so the 8-entity limit is gone), and one table. Spawning is Part 1's spawn with the row coming from the table:
e.generation = slot*.generation;
slot*.row = cast(kayan_table_add_row(w, w*.table&, e), u32);
w*.live += 1;
return e;
and despawning is Part 1's despawn calling kayan_table_remove_row.
Step 5: writing a new entity's data
There's no typed get or set yet (that's 2.5), so the program finds the row and writes through the column slices:
row: u64 = kayan_row(w&, drones[i]);
ps: []position = kayan_slice<position>(w.table&);
vs: []velocity = kayan_slice<velocity>(w.table&);
hs: []health = kayan_slice<health>(w.table&);
ps[row] = { .at = { .x = 10.0 * cast(i, f32), .y = 0.0 } };
Run it
spawned 6 drones into one table; it grew from 4 rows to 8
6 rows, room for 8
row 0 0:1 position ( 0.00, 0.00) health 100
...
despawn 0:1 and 3:1
4 rows, room for 8
row 0 5:1 position (50.00, 0.00) health 100
row 1 1:1 position (10.00, 0.00) health 100
row 2 2:1 position (20.00, 0.00) health 100
row 3 4:1 position (40.00, 0.00) health 100
spawn 3:2: a reused number, the next row, zeroed except what we set
...
kayan_verify: 0 problems
Row 0 went to 5:1 (the last row moved into the hole), then row 3 went to 4:1 the same way. Every column moved together, and the slots followed.
Try
- Remove
velocityfrom the table's signature and run it. Which line fails, and why is failing there the right behaviour? - Make
kayan_table_growstart at 1 row. Count the grows for 6 drones. - Delete the slot fix in
kayan_table_remove_rowand runkayan_verify.
1// 2.3 A table2//3// Two halves meet. Part 1 built the entity index: handles, slots, packed rows,4// swap-remove. 2.1 and 2.2 built the component side: ids, a registry, columns5// of bytes. A table is the packed rows of 1.4 with their typed arrays replaced6// by columns, one per component in the table's signature:7//8// table {position velocity health}9// entities [ 0:1 | 3:1 | 2:1 | ... ] who owns each row10// columns position [ ... ] velocity [ ... ] health [ ... ]11// count, capacity shared by every column12//13// Row r is entity entities[r], and its components are row r of every column.14// Adding a row grows every column together, and swap-remove copies the last15// row over the hole in every column together, so a row stays one entity.16//17// This world still has a single table, so every entity has the same18// components. 2.4 gives each combination its own.19 20import "std/print.rin"21import "std/memops.rin"22import "std/vec.rin"23 24vec2: struct = {25 x: f32;26 y: f32;27}28 29position: struct = { at: vec2; }30velocity: struct = { per_second: vec2; }31health: struct = { points: f32; max: f32; }32 33component: enum = {34 position,35 velocity,36 health,37}38 39component_of: proc<position>() -> component = { return component.position; }40component_of: proc<velocity>() -> component = { return component.velocity; }41component_of: proc<health>() -> component = { return component.health; }42 43// ---------------------------------------------------------------------------44// Entities (Part 1)45// ---------------------------------------------------------------------------46 47kayan_entity: struct = {48 index: u32;49 generation: u32;50}51 52kayan_status: enum[must_use] = {53 ok,54 dead,55}56 57kayan_slot: struct = {58 generation: u32;59 row: u32;60}61 62// ---------------------------------------------------------------------------63// Columns (2.2)64// ---------------------------------------------------------------------------65 66kayan_column: struct = {67 data: ^u8;68 size: u64;69 align: u64;70}71 72kayan_column_init: proc(info: *const reflect) -> kayan_column = {73 c: kayan_column = {};74 c.size = info*.size;75 c.align = info*.align;76 return c;77}78 79kayan_column_row: proc(c: *kayan_column, row: u64) -> *u8 = {80 return c*.data[row * c*.size]&;81}82 83kayan_column_grow: proc(arena: *memops_arena, c: *kayan_column, count: u64, capacity: u64) -> void = {84 data: ^u8 = cast(memops_arena_push_zero(arena, c*.size * capacity, c*.align), ^u8);85 if (count > 0) {86 memcpy(data, c*.data, c*.size * count);87 }88 c*.data = data;89}90 91kayan_column_copy_row: proc(c: *kayan_column, to: u64, from: u64) -> void = {92 memcpy(kayan_column_row(c, to), kayan_column_row(c, from), c*.size);93}94 95kayan_column_zero_row: proc(c: *kayan_column, row: u64) -> void = {96 memset(kayan_column_row(c, row), 0, c*.size);97}98 99kayan_column_slice: proc<T>(c: *kayan_column, count: u64) -> []T = {100 if (c*.size != sizeof(T)) {101 printfmt("kayan: a column of {}-byte rows read as a {}-byte type\n", c*.size, sizeof(T));102 exit(1);103 }104 raw: ^T = cast(c*.data, ^T);105 return raw[0..count];106}107 108// ---------------------------------------------------------------------------109// The table110// ---------------------------------------------------------------------------111 112// `columns[id]` is in use when the signature has bit `id`; the others stay113// zeroed. Count and capacity are the table's, shared by its columns and by114// `entities`, so they can never disagree.115kayan_table: struct = {116 signature: u64;117 count: u64;118 capacity: u64;119 entities: ^kayan_entity;120 columns: [64]kayan_column;121}122 123// The world: the registry (2.1), the slots and free list (Part 1), and for now124// a single table. Slots and the free list are vecs now, so the 8-entity limit125// of Part 1 is gone; the arena holds everything.126kayan_world: struct = {127 arena: *memops_arena;128 infos: [64]*const reflect;129 component_count: u32;130 slots: vec<kayan_slot>;131 free: vec<u32>;132 live: u64;133 table: kayan_table;134}135 136kayan_bit: proc(id: u32) -> u64 = {137 return 1'u64 shl cast(id, u64);138}139 140kayan_id_of: proc<T>() -> u32 = {141 return cast(component_of<T>(), u32);142}143 144kayan_mask_of: proc<T>() -> u64 = {145 return kayan_bit(kayan_id_of<T>());146}147 148kayan_register: proc(w: *kayan_world, id: u32, info: *const reflect) -> void = {149 if (id >= 64) {150 printfmt("kayan: component id {} is past the 64 a signature can hold\n", id);151 exit(1);152 }153 w*.infos[id] = info;154 if (id + 1 > w*.component_count) {155 w*.component_count = id + 1;156 }157}158 159kayan_table_has: proc(t: *kayan_table, id: u32) -> b32 = {160 return (t*.signature & kayan_bit(id)) != 0;161}162 163// A table for a signature: one empty column per component in it, sized from164// the registry. Every component in the signature must be registered.165kayan_table_init: proc(w: *kayan_world, signature: u64) -> kayan_table = {166 t: kayan_table = {};167 t.signature = signature;168 for (id: u32 = 0; id < 64; id += 1) {169 if ((signature & kayan_bit(id)) != 0) {170 if (id >= w*.component_count || w*.infos[id] == null) {171 printfmt("kayan: signature {} uses component {}, which was never registered\n", signature, id);172 exit(1);173 }174 t.columns[id] = kayan_column_init(w*.infos[id]);175 }176 }177 return t;178}179 180// Double the room in `entities` and in every column at once.181kayan_table_grow: proc(w: *kayan_world, t: *kayan_table) -> void = {182 capacity: u64 = t*.capacity == 0 ? 4 : t*.capacity * 2;183 entities: ^kayan_entity = cast(memops_arena_push_zero(w*.arena,184 sizeof(kayan_entity) * capacity, alignof(kayan_entity)), ^kayan_entity);185 if (t*.count > 0) {186 memcpy(entities, t*.entities, sizeof(kayan_entity) * t*.count);187 }188 t*.entities = entities;189 for (id: u32 = 0; id < w*.component_count; id += 1) {190 if (kayan_table_has(t, id)) {191 kayan_column_grow(w*.arena, t*.columns[id]&, t*.count, capacity);192 }193 }194 t*.capacity = capacity;195}196 197// A new row at the end, owned by `e`, with every component zeroed.198kayan_table_add_row: proc(w: *kayan_world, t: *kayan_table, e: kayan_entity) -> u64 = {199 if (t*.count == t*.capacity) {200 kayan_table_grow(w, t);201 }202 row: u64 = t*.count;203 t*.count += 1;204 t*.entities[row] = e;205 for (id: u32 = 0; id < w*.component_count; id += 1) {206 if (kayan_table_has(t, id)) {207 kayan_column_zero_row(t*.columns[id]&, row);208 }209 }210 return row;211}212 213// 1.4's swap-remove, for every column: the last row is copied over `row`, and214// the slot of the entity that owned the last row is told where it went.215kayan_table_remove_row: proc(w: *kayan_world, t: *kayan_table, row: u64) -> void = {216 last: u64 = t*.count - 1;217 if (row != last) {218 for (id: u32 = 0; id < w*.component_count; id += 1) {219 if (kayan_table_has(t, id)) {220 kayan_column_copy_row(t*.columns[id]&, row, last);221 }222 }223 moved: kayan_entity = t*.entities[last];224 t*.entities[row] = moved;225 vec<kayan_slot>at(w*.slots&, cast(moved.index, u64))*.row = cast(row, u32);226 }227 t*.count = last;228}229 230// The typed view of one of the table's columns: T's rows, all of them in use.231// Asking for a component the table does not have is a bug, so it stops.232kayan_slice: proc<T>(t: *kayan_table) -> []T = {233 id: u32 = kayan_id_of<T>();234 if (!kayan_table_has(t, id)) {235 printfmt("kayan: the table has no column for component {}\n", id);236 exit(1);237 }238 return kayan_column_slice<T>(t*.columns[id]&, t*.count);239}240 241// ---------------------------------------------------------------------------242// The world243// ---------------------------------------------------------------------------244 245kayan_world_init: proc(w: *kayan_world, arena: *memops_arena) -> void = {246 w* = {};247 w*.arena = arena;248}249 250kayan_alive: proc(w: *kayan_world, e: kayan_entity) -> b32 = {251 if (cast(e.index, u64) >= w*.slots.length) {252 return false;253 }254 return vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.generation == e.generation;255}256 257// Part 1's spawn, with the row coming from the table.258kayan_spawn: proc(w: *kayan_world) -> kayan_entity = {259 e: kayan_entity = {};260 if (!vec<u32>is_empty(w*.free&)) {261 e.index = vec<u32>pop(w*.free&);262 } else {263 e.index = cast(w*.slots.length, u32);264 _ = vec<kayan_slot>append(w*.arena, w*.slots&, { .generation = 1 });265 }266 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));267 e.generation = slot*.generation;268 slot*.row = cast(kayan_table_add_row(w, w*.table&, e), u32);269 w*.live += 1;270 return e;271}272 273kayan_despawn: proc(w: *kayan_world, e: kayan_entity) -> kayan_status = {274 if (!kayan_alive(w, e)) {275 return kayan_status.dead;276 }277 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));278 kayan_table_remove_row(w, w*.table&, cast(slot*.row, u64));279 slot*.generation += 1;280 _ = vec<u32>append(w*.arena, w*.free&, e.index);281 w*.live -= 1;282 return kayan_status.ok;283}284 285// The row a live entity is in. Typed get and set arrive in 2.5; until then,286// game code finds the row and indexes the table's columns.287kayan_row: proc(w: *kayan_world, e: kayan_entity) -> u64 = {288 return cast(vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.row, u64);289}290 291// 1.4's invariant, unchanged: every row belongs to a live entity whose slot292// points back at it, and the rows add up to the live count.293kayan_verify: proc(w: *kayan_world) -> u64 = {294 problems: u64 = 0;295 t: *kayan_table = w*.table&;296 for (row: u64 = 0; row < t*.count; row += 1) {297 e: kayan_entity = t*.entities[row];298 if (!kayan_alive(w, e)) {299 printfmt("kayan_verify: row {} holds dead entity {}:{}\n", row, e.index, e.generation);300 problems += 1;301 } else if (kayan_row(w, e) != row) {302 printfmt("kayan_verify: {}:{} is in row {}, its slot says row {}\n",303 e.index, e.generation, row, kayan_row(w, e));304 problems += 1;305 }306 }307 if (t*.count != w*.live) {308 printfmt("kayan_verify: {} rows, {} live entities\n", t*.count, w*.live);309 problems += 1;310 }311 return problems;312}313 314// ---------------------------------------------------------------------------315// The program316// ---------------------------------------------------------------------------317 318print_table: proc(w: *kayan_world) -> void = {319 t: *kayan_table = w*.table&;320 ps: []position = kayan_slice<position>(t);321 hs: []health = kayan_slice<health>(t);322 printfmt(" {} rows, room for {}\n", t*.count, t*.capacity);323 for (row: u64 = 0; row < t*.count; row += 1) {324 e: kayan_entity = t*.entities[row];325 printf(" row %u %u:%u position (%5.2f, %5.2f) health %3.0f\n", cast(row, u32), e.index, e.generation,326 cast(ps[row].at.x, f64), cast(ps[row].at.y, f64), cast(hs[row].points, f64));327 }328}329 330main: proc() -> i32 = {331 arena: memops_arena = {};332 memops_arena_initialize(arena&);333 w: kayan_world = {};334 kayan_world_init(w&, arena&);335 kayan_register(w&, kayan_id_of<position>(), position<>&);336 kayan_register(w&, kayan_id_of<velocity>(), velocity<>&);337 kayan_register(w&, kayan_id_of<health>(), health<>&);338 339 // One table, so one kind of entity: every entity is a drone.340 w.table = kayan_table_init(w&, kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<health>());341 342 drones: [6]kayan_entity = {};343 for (i: u32 = 0; i < 6; i += 1) {344 drones[i] = kayan_spawn(w&);345 // Write the new row through the typed views of its columns.346 row: u64 = kayan_row(w&, drones[i]);347 ps: []position = kayan_slice<position>(w.table&);348 vs: []velocity = kayan_slice<velocity>(w.table&);349 hs: []health = kayan_slice<health>(w.table&);350 ps[row] = { .at = { .x = 10.0 * cast(i, f32), .y = 0.0 } };351 vs[row] = { .per_second = { .x = 0.0, .y = 1.0 } };352 hs[row] = { .points = 100.0, .max = 100.0 };353 }354 printfmt("spawned 6 drones into one table; it grew from 4 rows to 8\n");355 print_table(w&);356 357 printfmt("despawn {}:{} and {}:{}\n", drones[0].index, drones[0].generation, drones[3].index, drones[3].generation);358 _ = kayan_despawn(w&, drones[0]);359 _ = kayan_despawn(w&, drones[3]);360 print_table(w&);361 362 again: kayan_entity = kayan_spawn(w&);363 hs: []health = kayan_slice<health>(w.table&);364 hs[kayan_row(w&, again)] = { .points = 40.0, .max = 100.0 };365 printfmt("spawn {}:{}: a reused number, the next row, zeroed except what we set\n",366 again.index, again.generation);367 print_table(w&);368 369 printfmt("kayan_verify: {} problems\n", kayan_verify(w&));370 return 0;371}What tables prints. tools/check.py fails the chapter if a single byte differs.
spawned 6 drones into one table; it grew from 4 rows to 8
6 rows, room for 8
row 0 0:1 position ( 0.00, 0.00) health 100
row 1 1:1 position (10.00, 0.00) health 100
row 2 2:1 position (20.00, 0.00) health 100
row 3 3:1 position (30.00, 0.00) health 100
row 4 4:1 position (40.00, 0.00) health 100
row 5 5:1 position (50.00, 0.00) health 100
despawn 0:1 and 3:1
4 rows, room for 8
row 0 5:1 position (50.00, 0.00) health 100
row 1 1:1 position (10.00, 0.00) health 100
row 2 2:1 position (20.00, 0.00) health 100
row 3 4:1 position (40.00, 0.00) health 100
spawn 3:2: a reused number, the next row, zeroed except what we set
5 rows, room for 8
row 0 5:1 position (50.00, 0.00) health 100
row 1 1:1 position (10.00, 0.00) health 100
row 2 2:1 position (20.00, 0.00) health 100
row 3 4:1 position (40.00, 0.00) health 100
row 4 3:2 position ( 0.00, 0.00) health 40
kayan_verify: 0 problems
Build an ECS · part 2, components · 2.4
Many tables: archetypes
ecs/src/2.components/2.4.archetypes/archetypes.rin
In the dojo: 3. the slot map, 8. index, not pointer, 9. interning, each on its own, to watch and to practise.
2.3's world had one table, so every entity had the same components. Real games mix: a turret has a position and a turret component, a drone has a position, velocity and health, a bullet has a position, velocity, damage and lifetime. Each combination gets a table of its own. Every row of a table has every component in its signature, so there are no holes and no flags. A table for one combination is called an archetype.
table 0 {position turret} 0:1
table 1 {position velocity health} 1:1 2:1 3:1
table 2 {position velocity damage lifetime} 4:1 5:1
Nobody declares the archetypes. The first time something spawns with a new combination, the world makes its table and remembers it by signature, so the next spawn finds it.
Step 1: slots say which table
kayan_slot: struct = {
generation: u32;
table: u32;
row: u32;
}
Handle to slot to (table, row). That's the whole entity index from here on.
Step 2: the world keeps every table
kayan_world: struct = {
arena: *memops_arena;
infos: [64]*const reflect;
component_count: u32;
tables: vec<kayan_table>;
by_signature: hashmap<u64, u32>;
slots: vec<kayan_slot>;
free: vec<u32>;
live: u64;
}
tables holds them all, and by_signature maps a signature to its table's index. Tables are referred to by index, never by pointer. Making a new table can grow the tables vec and move every table in memory, and an index survives that. A pointer is good until the next table is made:
kayan_table_at: proc(w: *kayan_world, index: u32) -> *kayan_table = {
return vec<kayan_table>at(w*.tables&, cast(index, u64));
}
Step 3: where archetypes come from
kayan_table_for: proc(w: *kayan_world, signature: u64) -> u32 = {
found: *u32 = hashmap<u64, u32>get(w*.by_signature&, signature);
if (found != null) {
return found*;
}
index: u32 = cast(w*.tables.length, u32);
_ = vec<kayan_table>append(w*.arena, w*.tables&, kayan_table_init(w, signature));
_ = hashmap<u64, u32>put(w*.arena, w*.by_signature&, signature, index);
return index;
}
Found, or made. A game has tens of archetypes, maybe a few hundred, so a linear search would do too. The hashmap costs two lines with rin's std, and 2.6 makes the lookup rare anyway.
Step 4: spawning takes a signature
kayan_spawn: proc(w: *kayan_world, signature: u64) -> kayan_entity = {
and puts the new entity in that signature's table:
table_index: u32 = kayan_table_for(w, signature);
t: *kayan_table = kayan_table_at(w, table_index);
slot*.table = table_index;
slot*.row = cast(kayan_table_add_row(w, t, e), u32);
Note the order: the table's pointer is taken after kayan_table_for, because finding the table can make one and move the rest. Getting this order wrong is the classic archetype-ECS bug. It works until the first new archetype appears mid-game.
Despawning finds the table through the slot:
kayan_table_remove_row(w, kayan_table_at(w, slot*.table), cast(slot*.row, u64));
Step 5: the invariant, over every table
kayan_verify now walks every table: each row belongs to a live entity whose slot points back at that table and row, and the rows of all tables add up to the live count. Same invariant, one more field.
Step 6: one column, many tables
The end of the program reads every position in the world:
for (ti: u32 = 0; ti < cast(w.tables.length, u32); ti += 1) {
t: *kayan_table = kayan_table_at(w&, ti);
ps: []position = kayan_slice<position>(t);
The position column of each table, in turn. Positions are no longer one array; they're one array per archetype that has them. This is exactly what a query does in Part 3: visit the tables that have the right components, and stream their columns.
Run it
spawned the turret: the world made table 0 for it
spawned 3 drones: the first made a table, the others found it
spawned 2 bullets
6 entities in 3 tables:
table 0 {position turret}: 0:1
table 1 {position velocity health}: 1:1 2:1 3:1
table 2 {position velocity damage lifetime}: 4:1 5:1
despawn drone 1:1: a swap-remove inside its own table
table 0 {position turret}: 0:1
table 1 {position velocity health}: 3:1 2:1
table 2 {position velocity damage lifetime}: 4:1 5:1
every position, table by table:
...
kayan_verify: 0 problems
Try
- Spawn a "drone with damage" (position, velocity, health, damage). How many tables now?
- Spawn 100 entities with 100 different signatures (use the bits of a counter). What does that do to a system that visits tables? This is called archetype fragmentation.
- Take the table pointer before
kayan_table_forinkayan_spawn, then spawn something with a new signature and print the tables. Where did the row go? Nothing crashes, and the sanitizers stay quiet too: the old tables are still valid arena memory, which is what makes this bug so quiet.
1// 2.4 Many tables: archetypes2//3// 2.3's world had one table, so every entity had the same components. Real4// games mix: a turret has a position and a turret component, a drone a5// position, velocity and health, a bullet a position, velocity, damage and a6// lifetime. Each combination gets a table of its own, and every row of a table7// has every component in its signature, so there are no holes and no flags.8// A table for one combination is called an archetype.9//10// table 0 {position turret} 0:111// table 1 {position velocity health} 1:1 2:1 3:112// table 2 {position velocity damage lifetime} 4:1 5:113//14// Nobody declares the archetypes. The first time something spawns with a new15// combination, the world makes its table, and remembers it by signature so the16// next spawn finds it. A slot now says which table an entity is in as well as17// which row.18 19import "std/print.rin"20import "std/memops.rin"21import "std/vec.rin"22import "std/hashmap.rin"23 24vec2: struct = {25 x: f32;26 y: f32;27}28 29position: struct = { at: vec2; }30velocity: struct = { per_second: vec2; }31health: struct = { points: f32; max: f32; }32damage: struct = { amount: f32; }33lifetime: struct = { seconds: f32; }34turret: struct = { cooldown: f32; timer: f32; }35 36component: enum = {37 position,38 velocity,39 health,40 damage,41 lifetime,42 turret,43}44 45component_of: proc<position>() -> component = { return component.position; }46component_of: proc<velocity>() -> component = { return component.velocity; }47component_of: proc<health>() -> component = { return component.health; }48component_of: proc<damage>() -> component = { return component.damage; }49component_of: proc<lifetime>() -> component = { return component.lifetime; }50component_of: proc<turret>() -> component = { return component.turret; }51 52// ---------------------------------------------------------------------------53// Entities (Part 1)54// ---------------------------------------------------------------------------55 56kayan_entity: struct = {57 index: u32;58 generation: u32;59}60 61kayan_status: enum[must_use] = {62 ok,63 dead,64}65 66// Where an entity lives: which table, and which row of it.67kayan_slot: struct = {68 generation: u32;69 table: u32;70 row: u32;71}72 73// ---------------------------------------------------------------------------74// Columns (2.2)75// ---------------------------------------------------------------------------76 77kayan_column: struct = {78 data: ^u8;79 size: u64;80 align: u64;81}82 83kayan_column_init: proc(info: *const reflect) -> kayan_column = {84 c: kayan_column = {};85 c.size = info*.size;86 c.align = info*.align;87 return c;88}89 90kayan_column_row: proc(c: *kayan_column, row: u64) -> *u8 = {91 return c*.data[row * c*.size]&;92}93 94kayan_column_grow: proc(arena: *memops_arena, c: *kayan_column, count: u64, capacity: u64) -> void = {95 data: ^u8 = cast(memops_arena_push_zero(arena, c*.size * capacity, c*.align), ^u8);96 if (count > 0) {97 memcpy(data, c*.data, c*.size * count);98 }99 c*.data = data;100}101 102kayan_column_copy_row: proc(c: *kayan_column, to: u64, from: u64) -> void = {103 memcpy(kayan_column_row(c, to), kayan_column_row(c, from), c*.size);104}105 106kayan_column_zero_row: proc(c: *kayan_column, row: u64) -> void = {107 memset(kayan_column_row(c, row), 0, c*.size);108}109 110kayan_column_slice: proc<T>(c: *kayan_column, count: u64) -> []T = {111 if (c*.size != sizeof(T)) {112 printfmt("kayan: a column of {}-byte rows read as a {}-byte type\n", c*.size, sizeof(T));113 exit(1);114 }115 raw: ^T = cast(c*.data, ^T);116 return raw[0..count];117}118 119// ---------------------------------------------------------------------------120// The table121// ---------------------------------------------------------------------------122 123// `columns[id]` is in use when the signature has bit `id`; the others stay124// zeroed. Count and capacity are the table's, shared by its columns and by125// `entities`, so they can never disagree.126kayan_table: struct = {127 signature: u64;128 count: u64;129 capacity: u64;130 entities: ^kayan_entity;131 columns: [64]kayan_column;132}133 134// The world: the registry, the slots and free list, and every table, with a135// map from signature to table index so finding one is a lookup, not a search.136// Tables are referred to by index, never by pointer: making a table can grow137// `tables` and move all of them, and an index survives that.138kayan_world: struct = {139 arena: *memops_arena;140 infos: [64]*const reflect;141 component_count: u32;142 tables: vec<kayan_table>;143 by_signature: hashmap<u64, u32>;144 slots: vec<kayan_slot>;145 free: vec<u32>;146 live: u64;147}148 149kayan_bit: proc(id: u32) -> u64 = {150 return 1'u64 shl cast(id, u64);151}152 153kayan_id_of: proc<T>() -> u32 = {154 return cast(component_of<T>(), u32);155}156 157kayan_mask_of: proc<T>() -> u64 = {158 return kayan_bit(kayan_id_of<T>());159}160 161kayan_register: proc(w: *kayan_world, id: u32, info: *const reflect) -> void = {162 if (id >= 64) {163 printfmt("kayan: component id {} is past the 64 a signature can hold\n", id);164 exit(1);165 }166 w*.infos[id] = info;167 if (id + 1 > w*.component_count) {168 w*.component_count = id + 1;169 }170}171 172kayan_table_has: proc(t: *kayan_table, id: u32) -> b32 = {173 return (t*.signature & kayan_bit(id)) != 0;174}175 176// A table for a signature: one empty column per component in it, sized from177// the registry. Every component in the signature must be registered.178kayan_table_init: proc(w: *kayan_world, signature: u64) -> kayan_table = {179 t: kayan_table = {};180 t.signature = signature;181 for (id: u32 = 0; id < 64; id += 1) {182 if ((signature & kayan_bit(id)) != 0) {183 if (id >= w*.component_count || w*.infos[id] == null) {184 printfmt("kayan: signature {} uses component {}, which was never registered\n", signature, id);185 exit(1);186 }187 t.columns[id] = kayan_column_init(w*.infos[id]);188 }189 }190 return t;191}192 193// Double the room in `entities` and in every column at once.194kayan_table_grow: proc(w: *kayan_world, t: *kayan_table) -> void = {195 capacity: u64 = t*.capacity == 0 ? 4 : t*.capacity * 2;196 entities: ^kayan_entity = cast(memops_arena_push_zero(w*.arena,197 sizeof(kayan_entity) * capacity, alignof(kayan_entity)), ^kayan_entity);198 if (t*.count > 0) {199 memcpy(entities, t*.entities, sizeof(kayan_entity) * t*.count);200 }201 t*.entities = entities;202 for (id: u32 = 0; id < w*.component_count; id += 1) {203 if (kayan_table_has(t, id)) {204 kayan_column_grow(w*.arena, t*.columns[id]&, t*.count, capacity);205 }206 }207 t*.capacity = capacity;208}209 210// A new row at the end, owned by `e`, with every component zeroed.211kayan_table_add_row: proc(w: *kayan_world, t: *kayan_table, e: kayan_entity) -> u64 = {212 if (t*.count == t*.capacity) {213 kayan_table_grow(w, t);214 }215 row: u64 = t*.count;216 t*.count += 1;217 t*.entities[row] = e;218 for (id: u32 = 0; id < w*.component_count; id += 1) {219 if (kayan_table_has(t, id)) {220 kayan_column_zero_row(t*.columns[id]&, row);221 }222 }223 return row;224}225 226// 1.4's swap-remove, for every column: the last row is copied over `row`, and227// the slot of the entity that owned the last row is told where it went.228kayan_table_remove_row: proc(w: *kayan_world, t: *kayan_table, row: u64) -> void = {229 last: u64 = t*.count - 1;230 if (row != last) {231 for (id: u32 = 0; id < w*.component_count; id += 1) {232 if (kayan_table_has(t, id)) {233 kayan_column_copy_row(t*.columns[id]&, row, last);234 }235 }236 moved: kayan_entity = t*.entities[last];237 t*.entities[row] = moved;238 vec<kayan_slot>at(w*.slots&, cast(moved.index, u64))*.row = cast(row, u32);239 }240 t*.count = last;241}242 243// The typed view of one of the table's columns: T's rows, all of them in use.244// Asking for a component the table does not have is a bug, so it stops.245kayan_slice: proc<T>(t: *kayan_table) -> []T = {246 id: u32 = kayan_id_of<T>();247 if (!kayan_table_has(t, id)) {248 printfmt("kayan: the table has no column for component {}\n", id);249 exit(1);250 }251 return kayan_column_slice<T>(t*.columns[id]&, t*.count);252}253 254// A pointer to a table is good until the next table is made.255kayan_table_at: proc(w: *kayan_world, index: u32) -> *kayan_table = {256 return vec<kayan_table>at(w*.tables&, cast(index, u64));257}258 259// The table for a signature: found if it exists, made if it does not. This is260// where archetypes come from.261kayan_table_for: proc(w: *kayan_world, signature: u64) -> u32 = {262 found: *u32 = hashmap<u64, u32>get(w*.by_signature&, signature);263 if (found != null) {264 return found*;265 }266 index: u32 = cast(w*.tables.length, u32);267 _ = vec<kayan_table>append(w*.arena, w*.tables&, kayan_table_init(w, signature));268 _ = hashmap<u64, u32>put(w*.arena, w*.by_signature&, signature, index);269 return index;270}271 272// ---------------------------------------------------------------------------273// The world274// ---------------------------------------------------------------------------275 276kayan_world_init: proc(w: *kayan_world, arena: *memops_arena) -> void = {277 w* = {};278 w*.arena = arena;279}280 281kayan_alive: proc(w: *kayan_world, e: kayan_entity) -> b32 = {282 if (cast(e.index, u64) >= w*.slots.length) {283 return false;284 }285 return vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.generation == e.generation;286}287 288// A new entity with the components in `signature`, all zeroed, in the table289// for that signature.290kayan_spawn: proc(w: *kayan_world, signature: u64) -> kayan_entity = {291 e: kayan_entity = {};292 if (!vec<u32>is_empty(w*.free&)) {293 e.index = vec<u32>pop(w*.free&);294 } else {295 e.index = cast(w*.slots.length, u32);296 _ = vec<kayan_slot>append(w*.arena, w*.slots&, { .generation = 1 });297 }298 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));299 e.generation = slot*.generation;300 // kayan_table_for can make a table and move the others, so the table's301 // pointer is taken after it.302 table_index: u32 = kayan_table_for(w, signature);303 t: *kayan_table = kayan_table_at(w, table_index);304 slot*.table = table_index;305 slot*.row = cast(kayan_table_add_row(w, t, e), u32);306 w*.live += 1;307 return e;308}309 310kayan_despawn: proc(w: *kayan_world, e: kayan_entity) -> kayan_status = {311 if (!kayan_alive(w, e)) {312 return kayan_status.dead;313 }314 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));315 kayan_table_remove_row(w, kayan_table_at(w, slot*.table), cast(slot*.row, u64));316 slot*.generation += 1;317 _ = vec<u32>append(w*.arena, w*.free&, e.index);318 w*.live -= 1;319 return kayan_status.ok;320}321 322// The table and row a live entity is in. Typed get and set arrive in 2.5;323// until then, game code finds the table and row and indexes the columns.324kayan_table_of: proc(w: *kayan_world, e: kayan_entity) -> *kayan_table = {325 return kayan_table_at(w, vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.table);326}327 328kayan_row: proc(w: *kayan_world, e: kayan_entity) -> u64 = {329 return cast(vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.row, u64);330}331 332// The invariant, over every table: each row belongs to a live entity whose333// slot points back at that table and row, and the rows of all tables add up to334// the live count.335kayan_verify: proc(w: *kayan_world) -> u64 = {336 problems: u64 = 0;337 rows: u64 = 0;338 for (ti: u64 = 0; ti < w*.tables.length; ti += 1) {339 t: *kayan_table = kayan_table_at(w, cast(ti, u32));340 for (row: u64 = 0; row < t*.count; row += 1) {341 e: kayan_entity = t*.entities[row];342 if (!kayan_alive(w, e)) {343 printfmt("kayan_verify: table {} row {} holds dead entity {}:{}\n", ti, row, e.index, e.generation);344 problems += 1;345 continue;346 }347 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));348 if (cast(slot*.table, u64) != ti || cast(slot*.row, u64) != row) {349 printfmt("kayan_verify: {}:{} is in table {} row {}, its slot says table {} row {}\n",350 e.index, e.generation, ti, row, slot*.table, slot*.row);351 problems += 1;352 }353 }354 rows += t*.count;355 }356 if (rows != w*.live) {357 printfmt("kayan_verify: {} rows in tables, {} live entities\n", rows, w*.live);358 problems += 1;359 }360 return problems;361}362 363kayan_print_signature: proc(w: *kayan_world, signature: u64) -> void = {364 printf("{");365 first: b32 = true;366 for (id: u32 = 0; id < w*.component_count; id += 1) {367 if ((signature & kayan_bit(id)) != 0) {368 printf(first ? "%s" : " %s", w*.infos[id]*.name);369 first = false;370 }371 }372 printf("}");373}374 375kayan_print_tables: proc(w: *kayan_world) -> void = {376 for (ti: u64 = 0; ti < w*.tables.length; ti += 1) {377 t: *kayan_table = kayan_table_at(w, cast(ti, u32));378 printf(" table %u ", cast(ti, u32));379 kayan_print_signature(w, t*.signature);380 printf(":");381 for (row: u64 = 0; row < t*.count; row += 1) {382 printf(" %u:%u", t*.entities[row].index, t*.entities[row].generation);383 }384 printf("\n");385 }386}387 388// ---------------------------------------------------------------------------389// The program390// ---------------------------------------------------------------------------391 392// Until 2.5's typed get, writing a component is: find the entity's table and393// row, take the typed view of the column, write the row.394place: proc(w: *kayan_world, e: kayan_entity, x: f32, y: f32) -> void = {395 ps: []position = kayan_slice<position>(kayan_table_of(w, e));396 ps[kayan_row(w, e)] = { .at = { .x = x, .y = y } };397}398 399main: proc() -> i32 = {400 arena: memops_arena = {};401 memops_arena_initialize(arena&);402 w: kayan_world = {};403 kayan_world_init(w&, arena&);404 kayan_register(w&, kayan_id_of<position>(), position<>&);405 kayan_register(w&, kayan_id_of<velocity>(), velocity<>&);406 kayan_register(w&, kayan_id_of<health>(), health<>&);407 kayan_register(w&, kayan_id_of<damage>(), damage<>&);408 kayan_register(w&, kayan_id_of<lifetime>(), lifetime<>&);409 kayan_register(w&, kayan_id_of<turret>(), turret<>&);410 411 // What each kind of thing is, as a set of components.412 gun: u64 = kayan_mask_of<position>() | kayan_mask_of<turret>();413 drone: u64 = kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<health>();414 bullet: u64 = kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<damage>()415 | kayan_mask_of<lifetime>();416 417 t0: kayan_entity = kayan_spawn(w&, gun);418 place(w&, t0, 0.0, 0.0);419 printfmt("spawned the turret: the world made table {} for it\n", w.tables.length - 1);420 421 drones: [3]kayan_entity = {};422 for (i: u32 = 0; i < 3; i += 1) {423 drones[i] = kayan_spawn(w&, drone);424 place(w&, drones[i], 10.0 * cast(i + 1, f32), 5.0);425 }426 printfmt("spawned 3 drones: the first made a table, the others found it\n");427 428 b0: kayan_entity = kayan_spawn(w&, bullet);429 b1: kayan_entity = kayan_spawn(w&, bullet);430 place(w&, b0, 0.0, 1.0);431 place(w&, b1, 1.0, 0.0);432 printfmt("spawned 2 bullets\n");433 printfmt("{} entities in {} tables:\n", w.live, w.tables.length);434 kayan_print_tables(w&);435 436 printfmt("despawn drone {}:{}: a swap-remove inside its own table\n", drones[0].index, drones[0].generation);437 _ = kayan_despawn(w&, drones[0]);438 kayan_print_tables(w&);439 440 // The positions of everything, table by table: the same column, in each441 // table that has it.442 printfmt("every position, table by table:\n");443 for (ti: u32 = 0; ti < cast(w.tables.length, u32); ti += 1) {444 t: *kayan_table = kayan_table_at(w&, ti);445 ps: []position = kayan_slice<position>(t);446 for (row: u64 = 0; row < t*.count; row += 1) {447 printf(" table %u row %u %u:%u at (%5.2f, %5.2f)\n", ti, cast(row, u32),448 t*.entities[row].index, t*.entities[row].generation,449 cast(ps[row].at.x, f64), cast(ps[row].at.y, f64));450 }451 }452 printfmt("kayan_verify: {} problems\n", kayan_verify(w&));453 return 0;454}What archetypes prints. tools/check.py fails the chapter if a single byte differs.
spawned the turret: the world made table 0 for it
spawned 3 drones: the first made a table, the others found it
spawned 2 bullets
6 entities in 3 tables:
table 0 {position turret}: 0:1
table 1 {position velocity health}: 1:1 2:1 3:1
table 2 {position velocity damage lifetime}: 4:1 5:1
despawn drone 1:1: a swap-remove inside its own table
table 0 {position turret}: 0:1
table 1 {position velocity health}: 3:1 2:1
table 2 {position velocity damage lifetime}: 4:1 5:1
every position, table by table:
table 0 row 0 0:1 at ( 0.00, 0.00)
table 1 row 0 3:1 at (30.00, 5.00)
table 1 row 1 2:1 at (20.00, 5.00)
table 2 row 0 4:1 at ( 0.00, 1.00)
table 2 row 1 5:1 at ( 1.00, 0.00)
kayan_verify: 0 problems
Build an ECS · part 2, components · 2.5
Typed access
ecs/src/2.components/2.5.typed_access/typed_access.rin
In the dojo: 4. a type-erased column, 8. index, not pointer, each on its own, to watch and to practise.
The world stores any component in any combination, as bytes. Game code should never see the bytes. This chapter adds the typed layer on top:
kayan_get<position>(w, e) -> *position, or null
kayan_has<velocity>(w, e) -> b32
kayan_set<health>(w, e, value) -> ok, dead, or missing
Each is a few lines that turn the type into an id, find the entity's table and row through its slot, and hand the row back as a T. The byte-level core underneath doesn't change at all. This is the split we talked about: C-style byte fiddling below, type-safe generic procs on top, each written once.
Step 1: get
kayan_get: proc<T>(w: *kayan_world, e: kayan_entity) -> *T = {
if (!kayan_alive(w, e)) {
return null;
}
slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));
t: *kayan_table = kayan_table_at(w, slot*.table);
if (!kayan_table_has(t, kayan_id_of<T>())) {
return null;
}
rows: []T = kayan_slice<T>(t);
return rows[cast(slot*.row, u64)]&;
}
Handle, slot, table, row, typed view. It's the same path as 1.4's get_position, for any component. Two ways to get null: the handle is dead, or the entity has no T. The caller checks once and then has a real pointer.
Step 2: has and set
kayan_has: proc<T>(w: *kayan_world, e: kayan_entity) -> b32 = {
return (kayan_signature(w, e) & kayan_mask_of<T>()) != 0;
}
kayan_has doesn't touch the row at all. An entity's components are its table's signature, one AND away.
kayan_set: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> kayan_status = {
if (!kayan_alive(w, e)) {
return kayan_status.dead;
}
p: *T = kayan_get<T>(w, e);
if (p == null) {
return kayan_status.missing;
}
p* = value;
return kayan_status.ok;
}
kayan_set writes a component the entity already has. Giving it a component it doesn't have changes which table it belongs in, and that's 2.6. So the status gains a third value:
kayan_status: enum[must_use] = {
ok,
dead,
missing,
}
Step 3: building things
With kayan_set, game code builds entities with small procs. Engines call these prefabs or archetype constructors:
spawn_drone: proc(w: *kayan_world, x: f32, y: f32) -> kayan_entity = {
e: kayan_entity = kayan_spawn(w, kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<health>());
_ = kayan_set<position>(w, e, { .at = { .x = x, .y = y } });
_ = kayan_set<velocity>(w, e, { .per_second = { .x = -1.0, .y = 0.0 } });
_ = kayan_set<health>(w, e, { .points = 100.0, .max = 100.0 });
return e;
}
The _ = says "I know this returns a status and I'm ignoring it on purpose". Here that's safe: the entity was just spawned with exactly these components.
Reading and changing is kayan_get:
h: *health = kayan_get<health>(w&, a);
h*.points -= 25.0;
Step 4: keep handles, borrow pointers
The pointer kayan_get returns points into a column. 2.2 showed what columns do when their table fills up: they move. So a pointer is good until the next structural change: a spawn, a despawn, or (from 2.6) an add or remove. The program proves it:
kept: *health = kayan_get<health>(w&, a);
for (i: u32 = 0; i < 8; i += 1) {
_ = spawn_drone(w&, 40.0 + cast(i, f32), 0.0);
}
kept*.points = 1.0;
kept a pointer to 0:1's health, spawned 8 drones (the table grew), wrote 1 through it:
0:1 at (20.00, 5.00), health 75
the write went to the column's old block; the handle still finds the real one
The health is still 75. The write landed in the column's old block, which the arena still owns, so nothing crashes and no sanitizer complains. It's just lost. Hence the rule every ECS has in some form:
Keep handles. Borrow pointers: get, use, forget.
Run it
two drones and a bullet, built with kayan_set:
0:1 at (20.00, 5.00), health 100
1:1 at (30.00, -5.00), health 100
drone 0:1 is hit through kayan_get<health>:
0:1 at (20.00, 5.00), health 75
the bullet has a velocity: yes; a health: no
kayan_get<health> on the bullet is null: yes
kayan_set<health> on the bullet: missing
despawned 1:1; kayan_set<health> on it now: dead
1:1 is not a live drone
...
kayan_verify: 0 problems
Try
- Write
spawn_turretandspawn_bullet-style procs for your own kinds of thing. - Move the
keptpointer'skayan_getto after the spawning loop. Now the write sticks. Why? - Make
kayan_setadd the component when it's missing, instead of returningmissing. What would it need from 2.6?
1// 2.5 Typed access2//3// The world now stores any component in any combination, as bytes. Game code4// should never see the bytes. This chapter adds the typed layer on top:5//6// kayan_get<position>(w, e) -> *position, or null7// kayan_has<velocity>(w, e) -> b328// kayan_set<health>(w, e, value) -> ok, dead, or missing9//10// Each is a few lines that turn the type into an id (kayan_id_of<T>), find the11// entity's table and row through its slot, and hand back the row as a T. The12// byte-level core underneath does not change at all.13//14// One rule comes with it, and the program shows why: keep handles, borrow15// pointers. A pointer from kayan_get points into a column, and the column16// moves when its table grows. A handle never goes stale silently.17 18import "std/print.rin"19import "std/memops.rin"20import "std/vec.rin"21import "std/hashmap.rin"22 23vec2: struct = {24 x: f32;25 y: f32;26}27 28position: struct = { at: vec2; }29velocity: struct = { per_second: vec2; }30health: struct = { points: f32; max: f32; }31damage: struct = { amount: f32; }32lifetime: struct = { seconds: f32; }33turret: struct = { cooldown: f32; timer: f32; }34 35component: enum = {36 position,37 velocity,38 health,39 damage,40 lifetime,41 turret,42}43 44component_of: proc<position>() -> component = { return component.position; }45component_of: proc<velocity>() -> component = { return component.velocity; }46component_of: proc<health>() -> component = { return component.health; }47component_of: proc<damage>() -> component = { return component.damage; }48component_of: proc<lifetime>() -> component = { return component.lifetime; }49component_of: proc<turret>() -> component = { return component.turret; }50 51// ---------------------------------------------------------------------------52// Entities (Part 1)53// ---------------------------------------------------------------------------54 55kayan_entity: struct = {56 index: u32;57 generation: u32;58}59 60// `missing`: the entity is alive but has no such component.61kayan_status: enum[must_use] = {62 ok,63 dead,64 missing,65}66 67// Where an entity lives: which table, and which row of it.68kayan_slot: struct = {69 generation: u32;70 table: u32;71 row: u32;72}73 74// ---------------------------------------------------------------------------75// Columns (2.2)76// ---------------------------------------------------------------------------77 78kayan_column: struct = {79 data: ^u8;80 size: u64;81 align: u64;82}83 84kayan_column_init: proc(info: *const reflect) -> kayan_column = {85 c: kayan_column = {};86 c.size = info*.size;87 c.align = info*.align;88 return c;89}90 91kayan_column_row: proc(c: *kayan_column, row: u64) -> *u8 = {92 return c*.data[row * c*.size]&;93}94 95kayan_column_grow: proc(arena: *memops_arena, c: *kayan_column, count: u64, capacity: u64) -> void = {96 data: ^u8 = cast(memops_arena_push_zero(arena, c*.size * capacity, c*.align), ^u8);97 if (count > 0) {98 memcpy(data, c*.data, c*.size * count);99 }100 c*.data = data;101}102 103kayan_column_copy_row: proc(c: *kayan_column, to: u64, from: u64) -> void = {104 memcpy(kayan_column_row(c, to), kayan_column_row(c, from), c*.size);105}106 107kayan_column_zero_row: proc(c: *kayan_column, row: u64) -> void = {108 memset(kayan_column_row(c, row), 0, c*.size);109}110 111kayan_column_slice: proc<T>(c: *kayan_column, count: u64) -> []T = {112 if (c*.size != sizeof(T)) {113 printfmt("kayan: a column of {}-byte rows read as a {}-byte type\n", c*.size, sizeof(T));114 exit(1);115 }116 raw: ^T = cast(c*.data, ^T);117 return raw[0..count];118}119 120// ---------------------------------------------------------------------------121// The table122// ---------------------------------------------------------------------------123 124// `columns[id]` is in use when the signature has bit `id`; the others stay125// zeroed. Count and capacity are the table's, shared by its columns and by126// `entities`, so they can never disagree.127kayan_table: struct = {128 signature: u64;129 count: u64;130 capacity: u64;131 entities: ^kayan_entity;132 columns: [64]kayan_column;133}134 135// The world: the registry, the slots and free list, and every table, with a136// map from signature to table index so finding one is a lookup, not a search.137// Tables are referred to by index, never by pointer: making a table can grow138// `tables` and move all of them, and an index survives that.139kayan_world: struct = {140 arena: *memops_arena;141 infos: [64]*const reflect;142 component_count: u32;143 tables: vec<kayan_table>;144 by_signature: hashmap<u64, u32>;145 slots: vec<kayan_slot>;146 free: vec<u32>;147 live: u64;148}149 150kayan_bit: proc(id: u32) -> u64 = {151 return 1'u64 shl cast(id, u64);152}153 154kayan_id_of: proc<T>() -> u32 = {155 return cast(component_of<T>(), u32);156}157 158kayan_mask_of: proc<T>() -> u64 = {159 return kayan_bit(kayan_id_of<T>());160}161 162kayan_register: proc(w: *kayan_world, id: u32, info: *const reflect) -> void = {163 if (id >= 64) {164 printfmt("kayan: component id {} is past the 64 a signature can hold\n", id);165 exit(1);166 }167 w*.infos[id] = info;168 if (id + 1 > w*.component_count) {169 w*.component_count = id + 1;170 }171}172 173kayan_table_has: proc(t: *kayan_table, id: u32) -> b32 = {174 return (t*.signature & kayan_bit(id)) != 0;175}176 177// A table for a signature: one empty column per component in it, sized from178// the registry. Every component in the signature must be registered.179kayan_table_init: proc(w: *kayan_world, signature: u64) -> kayan_table = {180 t: kayan_table = {};181 t.signature = signature;182 for (id: u32 = 0; id < 64; id += 1) {183 if ((signature & kayan_bit(id)) != 0) {184 if (id >= w*.component_count || w*.infos[id] == null) {185 printfmt("kayan: signature {} uses component {}, which was never registered\n", signature, id);186 exit(1);187 }188 t.columns[id] = kayan_column_init(w*.infos[id]);189 }190 }191 return t;192}193 194// Double the room in `entities` and in every column at once.195kayan_table_grow: proc(w: *kayan_world, t: *kayan_table) -> void = {196 capacity: u64 = t*.capacity == 0 ? 4 : t*.capacity * 2;197 entities: ^kayan_entity = cast(memops_arena_push_zero(w*.arena,198 sizeof(kayan_entity) * capacity, alignof(kayan_entity)), ^kayan_entity);199 if (t*.count > 0) {200 memcpy(entities, t*.entities, sizeof(kayan_entity) * t*.count);201 }202 t*.entities = entities;203 for (id: u32 = 0; id < w*.component_count; id += 1) {204 if (kayan_table_has(t, id)) {205 kayan_column_grow(w*.arena, t*.columns[id]&, t*.count, capacity);206 }207 }208 t*.capacity = capacity;209}210 211// A new row at the end, owned by `e`, with every component zeroed.212kayan_table_add_row: proc(w: *kayan_world, t: *kayan_table, e: kayan_entity) -> u64 = {213 if (t*.count == t*.capacity) {214 kayan_table_grow(w, t);215 }216 row: u64 = t*.count;217 t*.count += 1;218 t*.entities[row] = e;219 for (id: u32 = 0; id < w*.component_count; id += 1) {220 if (kayan_table_has(t, id)) {221 kayan_column_zero_row(t*.columns[id]&, row);222 }223 }224 return row;225}226 227// 1.4's swap-remove, for every column: the last row is copied over `row`, and228// the slot of the entity that owned the last row is told where it went.229kayan_table_remove_row: proc(w: *kayan_world, t: *kayan_table, row: u64) -> void = {230 last: u64 = t*.count - 1;231 if (row != last) {232 for (id: u32 = 0; id < w*.component_count; id += 1) {233 if (kayan_table_has(t, id)) {234 kayan_column_copy_row(t*.columns[id]&, row, last);235 }236 }237 moved: kayan_entity = t*.entities[last];238 t*.entities[row] = moved;239 vec<kayan_slot>at(w*.slots&, cast(moved.index, u64))*.row = cast(row, u32);240 }241 t*.count = last;242}243 244// The typed view of one of the table's columns: T's rows, all of them in use.245// Asking for a component the table does not have is a bug, so it stops.246kayan_slice: proc<T>(t: *kayan_table) -> []T = {247 id: u32 = kayan_id_of<T>();248 if (!kayan_table_has(t, id)) {249 printfmt("kayan: the table has no column for component {}\n", id);250 exit(1);251 }252 return kayan_column_slice<T>(t*.columns[id]&, t*.count);253}254 255// A pointer to a table is good until the next table is made.256kayan_table_at: proc(w: *kayan_world, index: u32) -> *kayan_table = {257 return vec<kayan_table>at(w*.tables&, cast(index, u64));258}259 260// The table for a signature: found if it exists, made if it does not. This is261// where archetypes come from.262kayan_table_for: proc(w: *kayan_world, signature: u64) -> u32 = {263 found: *u32 = hashmap<u64, u32>get(w*.by_signature&, signature);264 if (found != null) {265 return found*;266 }267 index: u32 = cast(w*.tables.length, u32);268 _ = vec<kayan_table>append(w*.arena, w*.tables&, kayan_table_init(w, signature));269 _ = hashmap<u64, u32>put(w*.arena, w*.by_signature&, signature, index);270 return index;271}272 273// ---------------------------------------------------------------------------274// The world275// ---------------------------------------------------------------------------276 277kayan_world_init: proc(w: *kayan_world, arena: *memops_arena) -> void = {278 w* = {};279 w*.arena = arena;280}281 282kayan_alive: proc(w: *kayan_world, e: kayan_entity) -> b32 = {283 if (cast(e.index, u64) >= w*.slots.length) {284 return false;285 }286 return vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.generation == e.generation;287}288 289// A new entity with the components in `signature`, all zeroed, in the table290// for that signature.291kayan_spawn: proc(w: *kayan_world, signature: u64) -> kayan_entity = {292 e: kayan_entity = {};293 if (!vec<u32>is_empty(w*.free&)) {294 e.index = vec<u32>pop(w*.free&);295 } else {296 e.index = cast(w*.slots.length, u32);297 _ = vec<kayan_slot>append(w*.arena, w*.slots&, { .generation = 1 });298 }299 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));300 e.generation = slot*.generation;301 // kayan_table_for can make a table and move the others, so the table's302 // pointer is taken after it.303 table_index: u32 = kayan_table_for(w, signature);304 t: *kayan_table = kayan_table_at(w, table_index);305 slot*.table = table_index;306 slot*.row = cast(kayan_table_add_row(w, t, e), u32);307 w*.live += 1;308 return e;309}310 311kayan_despawn: proc(w: *kayan_world, e: kayan_entity) -> kayan_status = {312 if (!kayan_alive(w, e)) {313 return kayan_status.dead;314 }315 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));316 kayan_table_remove_row(w, kayan_table_at(w, slot*.table), cast(slot*.row, u64));317 slot*.generation += 1;318 _ = vec<u32>append(w*.arena, w*.free&, e.index);319 w*.live -= 1;320 return kayan_status.ok;321}322 323// The components a live entity has: its table's signature. 0 for a dead one.324kayan_signature: proc(w: *kayan_world, e: kayan_entity) -> u64 = {325 if (!kayan_alive(w, e)) {326 return 0;327 }328 return kayan_table_at(w, vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.table)*.signature;329}330 331// ---------------------------------------------------------------------------332// The typed layer333// ---------------------------------------------------------------------------334 335// e's T, or null if e is dead or has no T. Handle, slot, table, row, typed336// view: the same path as 1.4's get_position, for any component. The pointer is337// good until the next structural change, so use it now and keep the handle.338kayan_get: proc<T>(w: *kayan_world, e: kayan_entity) -> *T = {339 if (!kayan_alive(w, e)) {340 return null;341 }342 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));343 t: *kayan_table = kayan_table_at(w, slot*.table);344 if (!kayan_table_has(t, kayan_id_of<T>())) {345 return null;346 }347 rows: []T = kayan_slice<T>(t);348 return rows[cast(slot*.row, u64)]&;349}350 351kayan_has: proc<T>(w: *kayan_world, e: kayan_entity) -> b32 = {352 return (kayan_signature(w, e) & kayan_mask_of<T>()) != 0;353}354 355// Write e's T. It must already have one: giving an entity a component it does356// not have changes its table, which is 2.6.357kayan_set: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> kayan_status = {358 if (!kayan_alive(w, e)) {359 return kayan_status.dead;360 }361 p: *T = kayan_get<T>(w, e);362 if (p == null) {363 return kayan_status.missing;364 }365 p* = value;366 return kayan_status.ok;367}368 369// The invariant, over every table: each row belongs to a live entity whose370// slot points back at that table and row, and the rows of all tables add up to371// the live count.372kayan_verify: proc(w: *kayan_world) -> u64 = {373 problems: u64 = 0;374 rows: u64 = 0;375 for (ti: u64 = 0; ti < w*.tables.length; ti += 1) {376 t: *kayan_table = kayan_table_at(w, cast(ti, u32));377 for (row: u64 = 0; row < t*.count; row += 1) {378 e: kayan_entity = t*.entities[row];379 if (!kayan_alive(w, e)) {380 printfmt("kayan_verify: table {} row {} holds dead entity {}:{}\n", ti, row, e.index, e.generation);381 problems += 1;382 continue;383 }384 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));385 if (cast(slot*.table, u64) != ti || cast(slot*.row, u64) != row) {386 printfmt("kayan_verify: {}:{} is in table {} row {}, its slot says table {} row {}\n",387 e.index, e.generation, ti, row, slot*.table, slot*.row);388 problems += 1;389 }390 }391 rows += t*.count;392 }393 if (rows != w*.live) {394 printfmt("kayan_verify: {} rows in tables, {} live entities\n", rows, w*.live);395 problems += 1;396 }397 return problems;398}399 400kayan_print_signature: proc(w: *kayan_world, signature: u64) -> void = {401 printf("{");402 first: b32 = true;403 for (id: u32 = 0; id < w*.component_count; id += 1) {404 if ((signature & kayan_bit(id)) != 0) {405 printf(first ? "%s" : " %s", w*.infos[id]*.name);406 first = false;407 }408 }409 printf("}");410}411 412kayan_print_tables: proc(w: *kayan_world) -> void = {413 for (ti: u64 = 0; ti < w*.tables.length; ti += 1) {414 t: *kayan_table = kayan_table_at(w, cast(ti, u32));415 printf(" table %u ", cast(ti, u32));416 kayan_print_signature(w, t*.signature);417 printf(":");418 for (row: u64 = 0; row < t*.count; row += 1) {419 printf(" %u:%u", t*.entities[row].index, t*.entities[row].generation);420 }421 printf("\n");422 }423}424 425// ---------------------------------------------------------------------------426// The program427// ---------------------------------------------------------------------------428 429// Game code builds things with procs like these: spawn with a signature, set430// each component. The statuses are ignored on purpose, because the entity was431// just spawned with exactly these components.432spawn_drone: proc(w: *kayan_world, x: f32, y: f32) -> kayan_entity = {433 e: kayan_entity = kayan_spawn(w, kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<health>());434 _ = kayan_set<position>(w, e, { .at = { .x = x, .y = y } });435 _ = kayan_set<velocity>(w, e, { .per_second = { .x = -1.0, .y = 0.0 } });436 _ = kayan_set<health>(w, e, { .points = 100.0, .max = 100.0 });437 return e;438}439 440spawn_bullet: proc(w: *kayan_world, x: f32, y: f32) -> kayan_entity = {441 e: kayan_entity = kayan_spawn(w, kayan_mask_of<position>() | kayan_mask_of<velocity>()442 | kayan_mask_of<damage>() | kayan_mask_of<lifetime>());443 _ = kayan_set<position>(w, e, { .at = { .x = x, .y = y } });444 _ = kayan_set<velocity>(w, e, { .per_second = { .x = 8.0, .y = 0.0 } });445 _ = kayan_set<damage>(w, e, { .amount = 25.0 });446 _ = kayan_set<lifetime>(w, e, { .seconds = 2.0 });447 return e;448}449 450print_drone: proc(w: *kayan_world, e: kayan_entity) -> void = {451 p: *position = kayan_get<position>(w, e);452 h: *health = kayan_get<health>(w, e);453 if (p == null || h == null) {454 printfmt(" {}:{} is not a live drone\n", e.index, e.generation);455 return;456 }457 printf(" %u:%u at (%5.2f, %5.2f), health %.0f\n", e.index, e.generation,458 cast(p*.at.x, f64), cast(p*.at.y, f64), cast(h*.points, f64));459}460 461main: proc() -> i32 = {462 arena: memops_arena = {};463 memops_arena_initialize(arena&);464 w: kayan_world = {};465 kayan_world_init(w&, arena&);466 kayan_register(w&, kayan_id_of<position>(), position<>&);467 kayan_register(w&, kayan_id_of<velocity>(), velocity<>&);468 kayan_register(w&, kayan_id_of<health>(), health<>&);469 kayan_register(w&, kayan_id_of<damage>(), damage<>&);470 kayan_register(w&, kayan_id_of<lifetime>(), lifetime<>&);471 kayan_register(w&, kayan_id_of<turret>(), turret<>&);472 473 a: kayan_entity = spawn_drone(w&, 20.0, 5.0);474 b: kayan_entity = spawn_drone(w&, 30.0, -5.0);475 shot: kayan_entity = spawn_bullet(w&, 0.0, 0.0);476 printfmt("two drones and a bullet, built with kayan_set:\n");477 print_drone(w&, a);478 print_drone(w&, b);479 480 // Read, change, done: the pointer is used right away.481 h: *health = kayan_get<health>(w&, a);482 h*.points -= 25.0;483 printfmt("drone {}:{} is hit through kayan_get<health>:\n", a.index, a.generation);484 print_drone(w&, a);485 486 printfmt("the bullet has a velocity: {}; a health: {}\n",487 kayan_has<velocity>(w&, shot) ? "yes" : "no", kayan_has<health>(w&, shot) ? "yes" : "no");488 printfmt("kayan_get<health> on the bullet is null: {}\n", kayan_get<health>(w&, shot) == null ? "yes" : "no");489 printfmt("kayan_set<health> on the bullet: {}\n", kayan_set<health>(w&, shot, { .points = 1.0, .max = 1.0 }));490 491 _ = kayan_despawn(w&, b);492 printfmt("despawned {}:{}; kayan_set<health> on it now: {}\n", b.index, b.generation,493 kayan_set<health>(w&, b, { .points = 1.0, .max = 1.0 }));494 print_drone(w&, b);495 496 // Keep handles, borrow pointers. A pointer kept across spawns points into497 // a column that moves when the table grows.498 kept: *health = kayan_get<health>(w&, a);499 for (i: u32 = 0; i < 8; i += 1) {500 _ = spawn_drone(w&, 40.0 + cast(i, f32), 0.0);501 }502 kept*.points = 1.0;503 printfmt("kept a pointer to {}:{}'s health, spawned 8 drones (the table grew), wrote 1 through it:\n",504 a.index, a.generation);505 print_drone(w&, a);506 printfmt(" the write went to the column's old block; the handle still finds the real one\n");507 printfmt("kayan_verify: {} problems\n", kayan_verify(w&));508 return 0;509}What typed_access prints. tools/check.py fails the chapter if a single byte differs.
two drones and a bullet, built with kayan_set:
0:1 at (20.00, 5.00), health 100
1:1 at (30.00, -5.00), health 100
drone 0:1 is hit through kayan_get<health>:
0:1 at (20.00, 5.00), health 75
the bullet has a velocity: yes; a health: no
kayan_get<health> on the bullet is null: yes
kayan_set<health> on the bullet: missing
despawned 1:1; kayan_set<health> on it now: dead
1:1 is not a live drone
kept a pointer to 0:1's health, spawned 8 drones (the table grew), wrote 1 through it:
0:1 at (20.00, 5.00), health 75
the write went to the column's old block; the handle still finds the real one
kayan_verify: 0 problems
Build an ECS · part 2, components · 2.6
Adding and removing components
ecs/src/2.components/2.6.add_remove/add_remove.rin
In the dojo: 2. swap-remove and the back-reference, 10. memoized edges, each on its own, to watch and to practise.
A drone catches fire: it gains a burning component. The fire goes out: it loses it. Changing what an entity is, while the game runs, is the thing your C kayan couldn't do, because its archetypes were structs fixed at compile time. In an archetype ECS a table holds exactly one combination, so a drone that gains a component no longer belongs in its table. It moves:
- find the table for its signature plus (or minus) the component;
- add a row there, and copy every component the two tables share;
- swap-remove its old row, fixing whoever moves into the hole;
- point its slot at the new table and row.
Its handle doesn't change. Everything else about it is where it was, one table over.
Step 1: the move
kayan_move: proc(w: *kayan_world, e: kayan_entity, to_index: u32) -> void = {
slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));
from_index: u32 = slot*.table;
if (from_index == to_index) {
return;
}
from: *kayan_table = kayan_table_at(w, from_index);
to: *kayan_table = kayan_table_at(w, to_index);
from_row: u64 = cast(slot*.row, u64);
to_row: u64 = kayan_table_add_row(w, to, e);
for (id: u32 = 0; id < w*.component_count; id += 1) {
if (kayan_table_has(from, id) && kayan_table_has(to, id)) {
memcpy(kayan_column_row(to*.columns[id]&, to_row),
kayan_column_row(from*.columns[id]&, from_row), from*.columns[id].size);
}
}
kayan_table_remove_row(w, from, from_row);
slot*.table = to_index;
slot*.row = cast(to_row, u32);
}
Every piece is something you've already built: kayan_table_add_row (2.3), a byte copy per shared column (2.2), kayan_table_remove_row with its slot fix (1.4). Components only the new table has start zeroed (add_row zeroes them). Components only the old table has disappear with the old row.
A move costs one row copy in and one row copy out. That's cheap, but not free, and it's why components that flip on and off every frame are sometimes better as a field (a bool on_fire inside health) than as a component.
Step 2: add and remove, typed
kayan_add: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> kayan_status = {
if (!kayan_alive(w, e)) {
return kayan_status.dead;
}
if (!kayan_has<T>(w, e)) {
from: u32 = vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.table;
kayan_move(w, e, kayan_table_with(w, from, kayan_id_of<T>()));
}
return kayan_set<T>(w, e, value);
}
If the entity already has a T, adding is just setting. Otherwise it moves to the table with T, then sets it. kayan_remove<T> is the mirror image: missing if it has no T, otherwise a move to the table without T.
Step 3: the archetype graph
Which table is "the table with T"? Its signature is the old one plus T's bit, so it could be looked up in by_signature every time. But a game toggling components does that constantly, always between the same few tables. So each table remembers its neighbours:
kayan_table: struct = {
signature: u64;
count: u64;
capacity: u64;
entities: ^kayan_entity;
columns: [64]kayan_column;
add_edges: [64]u32;
remove_edges: [64]u32;
}
add_edges[id] is where this table's entities go when they gain component id. remove_edges[id] is where they go when they lose it. Each is stored as index + 1, so the zero a new table starts with means "not known yet", with no initialization pass.
kayan_table_with: proc(w: *kayan_world, from: u32, id: u32) -> u32 = {
edge: u32 = kayan_table_at(w, from)*.add_edges[id];
if (edge != 0) {
w*.edge_hits += 1;
return edge - 1;
}
w*.lookups += 1;
to: u32 = kayan_table_for(w, kayan_table_at(w, from)*.signature | kayan_bit(id));
// kayan_table_for can move the tables, so both pointers are taken after it.
kayan_table_at(w, from)*.add_edges[id] = to + 1;
kayan_table_at(w, to)*.remove_edges[id] = from + 1;
return to;
}
The first move from {position velocity health} with burning looks the table up (and makes it). It then writes the answer on both tables: an add edge one way and a remove edge back. After that, catching fire and putting it out both follow edges. The tables and their edges form a graph, the archetype graph, and flecs is built around it.
Run it
4 drones:
table 0 {position velocity health}: 0:1 1:1 2:1 3:1
frame 1
0:1 catches fire: ok
2:1 catches fire: ok
table 0 {position velocity health}: 3:1 1:1
table 1 {position velocity health burning}: 0:1 2:1
frame 2
0:1 is put out: ok
1:1 catches fire: ok
2:1 catches fire: ok
...
frame 3
1:1 is put out: ok
2:1 is put out: ok
3:1 is put out: missing
3:1 catches fire: ok
...
2:1 moved tables twice and kept its data: at (30.00, 0.00), health 100
moves that looked up a table by signature: 1; that followed an edge: 6
kayan_verify: 0 problems
Watch row order shuffle as entities move: 3:1 jumps to row 0 of table 0 when 0:1 leaves. Order is never promised, and swap-remove is why. And one lookup for seven moves: that's the graph working.
Try
- Add a
stunnedcomponent and a second edge: stun a burning drone. How many tables exist now, and which edges were written? - Print every table's known edges after frame 3.
- Make
kayan_add<burning>on a drone that's already burning add to the rate instead of replacing it. Where does that logic belong: in kayan or in the game?
1// 2.6 Adding and removing components2//3// A drone catches fire: it gains a `burning` component. The fire goes out: it4// loses it. In an archetype ECS a table holds exactly one combination, so a5// drone that gains a component no longer belongs in its table. It moves:6//7// 1. find the table for its signature plus (or minus) the component;8// 2. add a row there, and copy every component the two tables share;9// 3. swap-remove its old row, fixing whoever moved into the hole;10// 4. point its slot at the new table and row.11//12// Its handle does not change. Everything else about it is where it was, one13// table over.14//15// Step 1 is a hashmap lookup by signature, and a game toggling components16// does it constantly, always between the same few tables. So each table17// remembers its neighbours: "add burning: table 2", "remove burning: table 1".18// These edges make the tables a graph, the archetype graph, and after the19// first move between two tables every later one follows an edge instead of20// looking anything up.21 22import "std/print.rin"23import "std/memops.rin"24import "std/vec.rin"25import "std/hashmap.rin"26 27vec2: struct = {28 x: f32;29 y: f32;30}31 32position: struct = { at: vec2; }33velocity: struct = { per_second: vec2; }34health: struct = { points: f32; max: f32; }35damage: struct = { amount: f32; }36lifetime: struct = { seconds: f32; }37turret: struct = { cooldown: f32; timer: f32; }38burning: struct = { per_second: f32; }39 40component: enum = {41 position,42 velocity,43 health,44 damage,45 lifetime,46 turret,47 burning,48}49 50component_of: proc<position>() -> component = { return component.position; }51component_of: proc<velocity>() -> component = { return component.velocity; }52component_of: proc<health>() -> component = { return component.health; }53component_of: proc<damage>() -> component = { return component.damage; }54component_of: proc<lifetime>() -> component = { return component.lifetime; }55component_of: proc<turret>() -> component = { return component.turret; }56component_of: proc<burning>() -> component = { return component.burning; }57 58// ---------------------------------------------------------------------------59// Entities (Part 1)60// ---------------------------------------------------------------------------61 62kayan_entity: struct = {63 index: u32;64 generation: u32;65}66 67// `missing`: the entity is alive but has no such component.68kayan_status: enum[must_use] = {69 ok,70 dead,71 missing,72}73 74// Where an entity lives: which table, and which row of it.75kayan_slot: struct = {76 generation: u32;77 table: u32;78 row: u32;79}80 81// ---------------------------------------------------------------------------82// Columns (2.2)83// ---------------------------------------------------------------------------84 85kayan_column: struct = {86 data: ^u8;87 size: u64;88 align: u64;89}90 91kayan_column_init: proc(info: *const reflect) -> kayan_column = {92 c: kayan_column = {};93 c.size = info*.size;94 c.align = info*.align;95 return c;96}97 98kayan_column_row: proc(c: *kayan_column, row: u64) -> *u8 = {99 return c*.data[row * c*.size]&;100}101 102kayan_column_grow: proc(arena: *memops_arena, c: *kayan_column, count: u64, capacity: u64) -> void = {103 data: ^u8 = cast(memops_arena_push_zero(arena, c*.size * capacity, c*.align), ^u8);104 if (count > 0) {105 memcpy(data, c*.data, c*.size * count);106 }107 c*.data = data;108}109 110kayan_column_copy_row: proc(c: *kayan_column, to: u64, from: u64) -> void = {111 memcpy(kayan_column_row(c, to), kayan_column_row(c, from), c*.size);112}113 114kayan_column_zero_row: proc(c: *kayan_column, row: u64) -> void = {115 memset(kayan_column_row(c, row), 0, c*.size);116}117 118kayan_column_slice: proc<T>(c: *kayan_column, count: u64) -> []T = {119 if (c*.size != sizeof(T)) {120 printfmt("kayan: a column of {}-byte rows read as a {}-byte type\n", c*.size, sizeof(T));121 exit(1);122 }123 raw: ^T = cast(c*.data, ^T);124 return raw[0..count];125}126 127// ---------------------------------------------------------------------------128// The table129// ---------------------------------------------------------------------------130 131// `columns[id]` is in use when the signature has bit `id`; the others stay132// zeroed. Count and capacity are the table's, shared by its columns and by133// `entities`, so they can never disagree.134//135// The edges: `add_edges[id]` is the table this one's entities move to when136// they gain component `id`, and `remove_edges[id]` when they lose it. Each is137// the table's index plus one, so 0, which a new table starts with, means "not138// known yet".139kayan_table: struct = {140 signature: u64;141 count: u64;142 capacity: u64;143 entities: ^kayan_entity;144 columns: [64]kayan_column;145 add_edges: [64]u32;146 remove_edges: [64]u32;147}148 149// The world: the registry, the slots and free list, and every table, with a150// map from signature to table index so finding one is a lookup, not a search.151// Tables are referred to by index, never by pointer: making a table can grow152// `tables` and move all of them, and an index survives that.153kayan_world: struct = {154 arena: *memops_arena;155 infos: [64]*const reflect;156 component_count: u32;157 tables: vec<kayan_table>;158 by_signature: hashmap<u64, u32>;159 slots: vec<kayan_slot>;160 free: vec<u32>;161 live: u64;162 // How table moves found their destination, for this chapter's output.163 lookups: u64;164 edge_hits: u64;165}166 167kayan_bit: proc(id: u32) -> u64 = {168 return 1'u64 shl cast(id, u64);169}170 171kayan_id_of: proc<T>() -> u32 = {172 return cast(component_of<T>(), u32);173}174 175kayan_mask_of: proc<T>() -> u64 = {176 return kayan_bit(kayan_id_of<T>());177}178 179kayan_register: proc(w: *kayan_world, id: u32, info: *const reflect) -> void = {180 if (id >= 64) {181 printfmt("kayan: component id {} is past the 64 a signature can hold\n", id);182 exit(1);183 }184 w*.infos[id] = info;185 if (id + 1 > w*.component_count) {186 w*.component_count = id + 1;187 }188}189 190kayan_table_has: proc(t: *kayan_table, id: u32) -> b32 = {191 return (t*.signature & kayan_bit(id)) != 0;192}193 194// A table for a signature: one empty column per component in it, sized from195// the registry. Every component in the signature must be registered.196kayan_table_init: proc(w: *kayan_world, signature: u64) -> kayan_table = {197 t: kayan_table = {};198 t.signature = signature;199 for (id: u32 = 0; id < 64; id += 1) {200 if ((signature & kayan_bit(id)) != 0) {201 if (id >= w*.component_count || w*.infos[id] == null) {202 printfmt("kayan: signature {} uses component {}, which was never registered\n", signature, id);203 exit(1);204 }205 t.columns[id] = kayan_column_init(w*.infos[id]);206 }207 }208 return t;209}210 211// Double the room in `entities` and in every column at once.212kayan_table_grow: proc(w: *kayan_world, t: *kayan_table) -> void = {213 capacity: u64 = t*.capacity == 0 ? 4 : t*.capacity * 2;214 entities: ^kayan_entity = cast(memops_arena_push_zero(w*.arena,215 sizeof(kayan_entity) * capacity, alignof(kayan_entity)), ^kayan_entity);216 if (t*.count > 0) {217 memcpy(entities, t*.entities, sizeof(kayan_entity) * t*.count);218 }219 t*.entities = entities;220 for (id: u32 = 0; id < w*.component_count; id += 1) {221 if (kayan_table_has(t, id)) {222 kayan_column_grow(w*.arena, t*.columns[id]&, t*.count, capacity);223 }224 }225 t*.capacity = capacity;226}227 228// A new row at the end, owned by `e`, with every component zeroed.229kayan_table_add_row: proc(w: *kayan_world, t: *kayan_table, e: kayan_entity) -> u64 = {230 if (t*.count == t*.capacity) {231 kayan_table_grow(w, t);232 }233 row: u64 = t*.count;234 t*.count += 1;235 t*.entities[row] = e;236 for (id: u32 = 0; id < w*.component_count; id += 1) {237 if (kayan_table_has(t, id)) {238 kayan_column_zero_row(t*.columns[id]&, row);239 }240 }241 return row;242}243 244// 1.4's swap-remove, for every column: the last row is copied over `row`, and245// the slot of the entity that owned the last row is told where it went.246kayan_table_remove_row: proc(w: *kayan_world, t: *kayan_table, row: u64) -> void = {247 last: u64 = t*.count - 1;248 if (row != last) {249 for (id: u32 = 0; id < w*.component_count; id += 1) {250 if (kayan_table_has(t, id)) {251 kayan_column_copy_row(t*.columns[id]&, row, last);252 }253 }254 moved: kayan_entity = t*.entities[last];255 t*.entities[row] = moved;256 vec<kayan_slot>at(w*.slots&, cast(moved.index, u64))*.row = cast(row, u32);257 }258 t*.count = last;259}260 261// The typed view of one of the table's columns: T's rows, all of them in use.262// Asking for a component the table does not have is a bug, so it stops.263kayan_slice: proc<T>(t: *kayan_table) -> []T = {264 id: u32 = kayan_id_of<T>();265 if (!kayan_table_has(t, id)) {266 printfmt("kayan: the table has no column for component {}\n", id);267 exit(1);268 }269 return kayan_column_slice<T>(t*.columns[id]&, t*.count);270}271 272// A pointer to a table is good until the next table is made.273kayan_table_at: proc(w: *kayan_world, index: u32) -> *kayan_table = {274 return vec<kayan_table>at(w*.tables&, cast(index, u64));275}276 277// The table for a signature: found if it exists, made if it does not. This is278// where archetypes come from.279kayan_table_for: proc(w: *kayan_world, signature: u64) -> u32 = {280 found: *u32 = hashmap<u64, u32>get(w*.by_signature&, signature);281 if (found != null) {282 return found*;283 }284 index: u32 = cast(w*.tables.length, u32);285 _ = vec<kayan_table>append(w*.arena, w*.tables&, kayan_table_init(w, signature));286 _ = hashmap<u64, u32>put(w*.arena, w*.by_signature&, signature, index);287 return index;288}289 290// ---------------------------------------------------------------------------291// The world292// ---------------------------------------------------------------------------293 294kayan_world_init: proc(w: *kayan_world, arena: *memops_arena) -> void = {295 w* = {};296 w*.arena = arena;297}298 299kayan_alive: proc(w: *kayan_world, e: kayan_entity) -> b32 = {300 if (cast(e.index, u64) >= w*.slots.length) {301 return false;302 }303 return vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.generation == e.generation;304}305 306// A new entity with the components in `signature`, all zeroed, in the table307// for that signature.308kayan_spawn: proc(w: *kayan_world, signature: u64) -> kayan_entity = {309 e: kayan_entity = {};310 if (!vec<u32>is_empty(w*.free&)) {311 e.index = vec<u32>pop(w*.free&);312 } else {313 e.index = cast(w*.slots.length, u32);314 _ = vec<kayan_slot>append(w*.arena, w*.slots&, { .generation = 1 });315 }316 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));317 e.generation = slot*.generation;318 // kayan_table_for can make a table and move the others, so the table's319 // pointer is taken after it.320 table_index: u32 = kayan_table_for(w, signature);321 t: *kayan_table = kayan_table_at(w, table_index);322 slot*.table = table_index;323 slot*.row = cast(kayan_table_add_row(w, t, e), u32);324 w*.live += 1;325 return e;326}327 328kayan_despawn: proc(w: *kayan_world, e: kayan_entity) -> kayan_status = {329 if (!kayan_alive(w, e)) {330 return kayan_status.dead;331 }332 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));333 kayan_table_remove_row(w, kayan_table_at(w, slot*.table), cast(slot*.row, u64));334 slot*.generation += 1;335 _ = vec<u32>append(w*.arena, w*.free&, e.index);336 w*.live -= 1;337 return kayan_status.ok;338}339 340// The components a live entity has: its table's signature. 0 for a dead one.341kayan_signature: proc(w: *kayan_world, e: kayan_entity) -> u64 = {342 if (!kayan_alive(w, e)) {343 return 0;344 }345 return kayan_table_at(w, vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.table)*.signature;346}347 348// ---------------------------------------------------------------------------349// Moving between tables350// ---------------------------------------------------------------------------351 352// The table an entity of table `from` moves to when it gains component `id`.353// The first time, it is found by signature, which can make the table; the354// answer is written on both tables, as from's add edge and the destination's355// remove edge, so neither direction is looked up again.356kayan_table_with: proc(w: *kayan_world, from: u32, id: u32) -> u32 = {357 edge: u32 = kayan_table_at(w, from)*.add_edges[id];358 if (edge != 0) {359 w*.edge_hits += 1;360 return edge - 1;361 }362 w*.lookups += 1;363 to: u32 = kayan_table_for(w, kayan_table_at(w, from)*.signature | kayan_bit(id));364 // kayan_table_for can move the tables, so both pointers are taken after it.365 kayan_table_at(w, from)*.add_edges[id] = to + 1;366 kayan_table_at(w, to)*.remove_edges[id] = from + 1;367 return to;368}369 370// The same for losing component `id`.371kayan_table_without: proc(w: *kayan_world, from: u32, id: u32) -> u32 = {372 edge: u32 = kayan_table_at(w, from)*.remove_edges[id];373 if (edge != 0) {374 w*.edge_hits += 1;375 return edge - 1;376 }377 w*.lookups += 1;378 to: u32 = kayan_table_for(w, kayan_table_at(w, from)*.signature & ~kayan_bit(id));379 kayan_table_at(w, from)*.remove_edges[id] = to + 1;380 kayan_table_at(w, to)*.add_edges[id] = from + 1;381 return to;382}383 384// Move a live entity to table `to_index`. Components both tables have are385// copied, byte for byte; components only the new table has start zeroed;386// components only the old table has are dropped with its old row.387kayan_move: proc(w: *kayan_world, e: kayan_entity, to_index: u32) -> void = {388 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));389 from_index: u32 = slot*.table;390 if (from_index == to_index) {391 return;392 }393 from: *kayan_table = kayan_table_at(w, from_index);394 to: *kayan_table = kayan_table_at(w, to_index);395 from_row: u64 = cast(slot*.row, u64);396 to_row: u64 = kayan_table_add_row(w, to, e);397 for (id: u32 = 0; id < w*.component_count; id += 1) {398 if (kayan_table_has(from, id) && kayan_table_has(to, id)) {399 memcpy(kayan_column_row(to*.columns[id]&, to_row),400 kayan_column_row(from*.columns[id]&, from_row), from*.columns[id].size);401 }402 }403 kayan_table_remove_row(w, from, from_row);404 slot*.table = to_index;405 slot*.row = cast(to_row, u32);406}407 408// ---------------------------------------------------------------------------409// The typed layer410// ---------------------------------------------------------------------------411 412// e's T, or null if e is dead or has no T. Handle, slot, table, row, typed413// view: the same path as 1.4's get_position, for any component. The pointer is414// good until the next structural change, so use it now and keep the handle.415kayan_get: proc<T>(w: *kayan_world, e: kayan_entity) -> *T = {416 if (!kayan_alive(w, e)) {417 return null;418 }419 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));420 t: *kayan_table = kayan_table_at(w, slot*.table);421 if (!kayan_table_has(t, kayan_id_of<T>())) {422 return null;423 }424 rows: []T = kayan_slice<T>(t);425 return rows[cast(slot*.row, u64)]&;426}427 428kayan_has: proc<T>(w: *kayan_world, e: kayan_entity) -> b32 = {429 return (kayan_signature(w, e) & kayan_mask_of<T>()) != 0;430}431 432// Write e's T. It must already have one: giving an entity a component it does433// not have changes its table, which is 2.6.434kayan_set: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> kayan_status = {435 if (!kayan_alive(w, e)) {436 return kayan_status.dead;437 }438 p: *T = kayan_get<T>(w, e);439 if (p == null) {440 return kayan_status.missing;441 }442 p* = value;443 return kayan_status.ok;444}445 446// Give e a T. If it already has one, this is kayan_set; if not, it moves to447// the table with T and then sets it.448kayan_add: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> kayan_status = {449 if (!kayan_alive(w, e)) {450 return kayan_status.dead;451 }452 if (!kayan_has<T>(w, e)) {453 from: u32 = vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.table;454 kayan_move(w, e, kayan_table_with(w, from, kayan_id_of<T>()));455 }456 return kayan_set<T>(w, e, value);457}458 459// Take e's T away, moving it to the table without T.460kayan_remove: proc<T>(w: *kayan_world, e: kayan_entity) -> kayan_status = {461 if (!kayan_alive(w, e)) {462 return kayan_status.dead;463 }464 if (!kayan_has<T>(w, e)) {465 return kayan_status.missing;466 }467 from: u32 = vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.table;468 kayan_move(w, e, kayan_table_without(w, from, kayan_id_of<T>()));469 return kayan_status.ok;470}471 472// The invariant, over every table: each row belongs to a live entity whose473// slot points back at that table and row, and the rows of all tables add up to474// the live count.475kayan_verify: proc(w: *kayan_world) -> u64 = {476 problems: u64 = 0;477 rows: u64 = 0;478 for (ti: u64 = 0; ti < w*.tables.length; ti += 1) {479 t: *kayan_table = kayan_table_at(w, cast(ti, u32));480 for (row: u64 = 0; row < t*.count; row += 1) {481 e: kayan_entity = t*.entities[row];482 if (!kayan_alive(w, e)) {483 printfmt("kayan_verify: table {} row {} holds dead entity {}:{}\n", ti, row, e.index, e.generation);484 problems += 1;485 continue;486 }487 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));488 if (cast(slot*.table, u64) != ti || cast(slot*.row, u64) != row) {489 printfmt("kayan_verify: {}:{} is in table {} row {}, its slot says table {} row {}\n",490 e.index, e.generation, ti, row, slot*.table, slot*.row);491 problems += 1;492 }493 }494 rows += t*.count;495 }496 if (rows != w*.live) {497 printfmt("kayan_verify: {} rows in tables, {} live entities\n", rows, w*.live);498 problems += 1;499 }500 return problems;501}502 503kayan_print_signature: proc(w: *kayan_world, signature: u64) -> void = {504 printf("{");505 first: b32 = true;506 for (id: u32 = 0; id < w*.component_count; id += 1) {507 if ((signature & kayan_bit(id)) != 0) {508 printf(first ? "%s" : " %s", w*.infos[id]*.name);509 first = false;510 }511 }512 printf("}");513}514 515kayan_print_tables: proc(w: *kayan_world) -> void = {516 for (ti: u64 = 0; ti < w*.tables.length; ti += 1) {517 t: *kayan_table = kayan_table_at(w, cast(ti, u32));518 printf(" table %u ", cast(ti, u32));519 kayan_print_signature(w, t*.signature);520 printf(":");521 for (row: u64 = 0; row < t*.count; row += 1) {522 printf(" %u:%u", t*.entities[row].index, t*.entities[row].generation);523 }524 printf("\n");525 }526}527 528// ---------------------------------------------------------------------------529// The program530// ---------------------------------------------------------------------------531 532spawn_drone: proc(w: *kayan_world, x: f32, y: f32) -> kayan_entity = {533 e: kayan_entity = kayan_spawn(w, kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<health>());534 _ = kayan_set<position>(w, e, { .at = { .x = x, .y = y } });535 _ = kayan_set<velocity>(w, e, { .per_second = { .x = -1.0, .y = 0.0 } });536 _ = kayan_set<health>(w, e, { .points = 100.0, .max = 100.0 });537 return e;538}539 540ignite: proc(w: *kayan_world, e: kayan_entity) -> void = {541 printfmt(" {}:{} catches fire: {}\n", e.index, e.generation,542 kayan_add<burning>(w, e, { .per_second = 10.0 }));543}544 545put_out: proc(w: *kayan_world, e: kayan_entity) -> void = {546 printfmt(" {}:{} is put out: {}\n", e.index, e.generation, kayan_remove<burning>(w, e));547}548 549main: proc() -> i32 = {550 arena: memops_arena = {};551 memops_arena_initialize(arena&);552 w: kayan_world = {};553 kayan_world_init(w&, arena&);554 kayan_register(w&, kayan_id_of<position>(), position<>&);555 kayan_register(w&, kayan_id_of<velocity>(), velocity<>&);556 kayan_register(w&, kayan_id_of<health>(), health<>&);557 kayan_register(w&, kayan_id_of<damage>(), damage<>&);558 kayan_register(w&, kayan_id_of<lifetime>(), lifetime<>&);559 kayan_register(w&, kayan_id_of<turret>(), turret<>&);560 kayan_register(w&, kayan_id_of<burning>(), burning<>&);561 562 d: [4]kayan_entity = {};563 for (i: u32 = 0; i < 4; i += 1) {564 d[i] = spawn_drone(w&, 10.0 * cast(i + 1, f32), 0.0);565 }566 printfmt("4 drones:\n");567 kayan_print_tables(w&);568 569 printfmt("frame 1\n");570 ignite(w&, d[0]);571 ignite(w&, d[2]);572 kayan_print_tables(w&);573 574 printfmt("frame 2\n");575 put_out(w&, d[0]);576 ignite(w&, d[1]);577 ignite(w&, d[2]);578 kayan_print_tables(w&);579 580 printfmt("frame 3\n");581 put_out(w&, d[1]);582 put_out(w&, d[2]);583 put_out(w&, d[3]);584 ignite(w&, d[3]);585 kayan_print_tables(w&);586 587 p: *position = kayan_get<position>(w&, d[2]);588 printf("%u:%u moved tables twice and kept its data: at (%.2f, %.2f), health %.0f\n", d[2].index, d[2].generation,589 cast(p*.at.x, f64), cast(p*.at.y, f64), cast(kayan_get<health>(w&, d[2])*.points, f64));590 printfmt("moves that looked up a table by signature: {}; that followed an edge: {}\n", w.lookups, w.edge_hits);591 printfmt("kayan_verify: {} problems\n", kayan_verify(w&));592 return 0;593}What add_remove prints. tools/check.py fails the chapter if a single byte differs.
4 drones:
table 0 {position velocity health}: 0:1 1:1 2:1 3:1
frame 1
0:1 catches fire: ok
2:1 catches fire: ok
table 0 {position velocity health}: 3:1 1:1
table 1 {position velocity health burning}: 0:1 2:1
frame 2
0:1 is put out: ok
1:1 catches fire: ok
2:1 catches fire: ok
table 0 {position velocity health}: 3:1 0:1
table 1 {position velocity health burning}: 2:1 1:1
frame 3
1:1 is put out: ok
2:1 is put out: ok
3:1 is put out: missing
3:1 catches fire: ok
table 0 {position velocity health}: 2:1 0:1 1:1
table 1 {position velocity health burning}: 3:1
2:1 moved tables twice and kept its data: at (30.00, 0.00), health 100
moves that looked up a table by signature: 1; that followed an edge: 6
kayan_verify: 0 problems
Build an ECS · part 2, components · 2.7
Inspecting the world
ecs/src/2.components/2.7.inspecting/inspecting.rin
In the dojo: 11. a reflection walk, on its own, to watch and to practise.
The world can store, copy and move any component, but it can't show you one. When something goes wrong at frame 3,000, you want to see an entity: every component it has, every field, with names. In C that means writing a printer per component and keeping each in step with its struct. In rin, the compiler already wrote down everything a printer needs. Each component's reflect record lists its fields with their names, types and byte offsets. The world registered those records back in 2.1 to learn component sizes. Here they pay off a second time.
kayan_inspect(w, e) every component of e, field by field
kayan_print_layout(w) every table, its columns, and their memory
Neither names a single component type. Add a component tomorrow and both print it.
Step 1: a component as bytes
Tools work from reflect records, not types, so they need the byte view:
kayan_get_bytes: proc(w: *kayan_world, e: kayan_entity, id: u32) -> []u8 = {
none: []u8 = {};
if (!kayan_alive(w, e) || id >= w*.component_count) {
return none;
}
slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));
t: *kayan_table = kayan_table_at(w, slot*.table);
if (!kayan_table_has(t, id)) {
return none;
}
c: *kayan_column = t*.columns[id]&;
row: u64 = cast(slot*.row, u64);
all: []u8 = c*.data[0..t*.count * c*.size];
return all[row * c*.size..(row + 1) * c*.size];
}
It's kayan_get<T>'s path, stopping one step early: the row's bytes as a slice, sized by the column. Game code uses the typed layer; tools use this.
Step 2: walking fields
kayan_print_fields: proc(type: *const reflect, base: []const u8, depth: u32) -> void = {
fields: ^const reflect_field = reflect_fields(type);
for (i: u64 = 0; i < type*.count; i += 1) {
f: *const reflect_field = fields[i]&;
if (reflect_field_has_attr(f, "hidden")) {
continue;
}
at: []const u8 = base[f*.offset..f*.offset + f*.size];
kayan_indent(depth);
printf("%s:", f*.name);
info: *const reflect = f*.info;
if (info != null && info*.kind == reflect_kind_struct) {
printf("\n");
kayan_print_fields(info, at, depth + 1);
} else {
printf(" ");
kayan_print_builtin(f*.type, at);
printf("\n");
}
}
}
For each field the record says, cut its bytes out of the component by offset and size. A struct field (at: vec2 inside position) has a record of its own, so the printer recurses into it. A builtin field (f32, u32) is printed by its type's name.
Because each field's bytes are cut with a bounds-checked slice, a record that disagreed with the data would stop a Debug build instead of printing garbage.
Step 3: hiding bookkeeping
Fields can carry tags with @:
turret: struct = { cooldown: f32; timer: f32 @ "hidden"; }
The inspector skips anything tagged "hidden". The component decides what a tool shows; the tool doesn't need to know the component.
Step 4: an entity, and the layout
kayan_inspect walks the entity's signature and prints each component with kayan_print_fields. kayan_print_layout walks every table and prints each column's row size and bytes used out of bytes allocated, which shows where memory goes.
Run it
0:1 {position turret}
position
at:
x: 0.00
y: 0.00
turret
cooldown: 0.50
...
2:1 {position velocity health burning}
...
health
points: 60.00
max: 100.00
burning
per_second: 10.00
...
1:1 (dead)
table 1 {position velocity health}: 0 rows, room for 4
position 8 bytes a row, 0 of 32 bytes in use
velocity 8 bytes a row, 0 of 32 bytes in use
health 8 bytes a row, 0 of 32 bytes in use
The turret's timer isn't printed. Table 1 is empty but keeps its capacity: drone 1:1 was its only remaining row after 2:1 caught fire and moved out, and then 1:1 was despawned. Tables never shrink in this course. A real engine might trim empty ones between levels.
Part 2 in one picture
handle 2:1 ──> slots[2] { generation 1, table 2, row 0 }
│
table 2 {position velocity health burning}
entities [ 2:1 ]
position [ (30, -5) ] velocity [ (-1, 0) ] health [ 60/100 ] burning [ 10 ]
add_edges / remove_edges ──> neighbouring tables in the archetype graph
The storage half of the ECS is done: identity (Part 1), and any component, in any combination, changing at run time (Part 2). Part 3 is the behaviour: queries to find tables, and systems to run over them.
Try
- Add a field to
healthand run it. Nothing in the inspector changes. - Tag
health.maxas hidden. - Write
kayan_set_f32_by_name(w, e, "health", "points", 10.0)with the reflect records: find the component by name, the field by name, check it's anf32, write it. That's the core of an editor or a console.
1// 2.7 Inspecting the world2//3// The world can store, copy and move any component, but it cannot show you4// one. When something goes wrong at frame 3,000 you want to see an entity:5// every component it has, every field, with names. In C that means writing a6// printer per component and keeping it in step with the struct. In rin the7// compiler already wrote down everything a printer needs: each component's8// reflect record lists its fields with their names, types and offsets. The9// world registered those records in 2.1 to know component sizes; here they10// pay off a second time.11//12// kayan_inspect(w, e) every component of e, field by field13// kayan_print_layout(w) every table, its columns, and their memory14//15// Neither names a single component type. Add a component tomorrow and both16// print it.17 18import "std/print.rin"19import "std/memops.rin"20import "std/vec.rin"21import "std/hashmap.rin"22 23vec2: struct = {24 x: f32;25 y: f32;26}27 28position: struct = { at: vec2; }29velocity: struct = { per_second: vec2; }30health: struct = { points: f32; max: f32; }31damage: struct = { amount: f32; }32lifetime: struct = { seconds: f32; }33// A field tagged "hidden" is bookkeeping the inspector does not show.34turret: struct = { cooldown: f32; timer: f32 @ "hidden"; }35burning: struct = { per_second: f32; }36 37component: enum = {38 position,39 velocity,40 health,41 damage,42 lifetime,43 turret,44 burning,45}46 47component_of: proc<position>() -> component = { return component.position; }48component_of: proc<velocity>() -> component = { return component.velocity; }49component_of: proc<health>() -> component = { return component.health; }50component_of: proc<damage>() -> component = { return component.damage; }51component_of: proc<lifetime>() -> component = { return component.lifetime; }52component_of: proc<turret>() -> component = { return component.turret; }53component_of: proc<burning>() -> component = { return component.burning; }54 55// ---------------------------------------------------------------------------56// Entities (Part 1)57// ---------------------------------------------------------------------------58 59kayan_entity: struct = {60 index: u32;61 generation: u32;62}63 64// `missing`: the entity is alive but has no such component.65kayan_status: enum[must_use] = {66 ok,67 dead,68 missing,69}70 71// Where an entity lives: which table, and which row of it.72kayan_slot: struct = {73 generation: u32;74 table: u32;75 row: u32;76}77 78// ---------------------------------------------------------------------------79// Columns (2.2)80// ---------------------------------------------------------------------------81 82kayan_column: struct = {83 data: ^u8;84 size: u64;85 align: u64;86}87 88kayan_column_init: proc(info: *const reflect) -> kayan_column = {89 c: kayan_column = {};90 c.size = info*.size;91 c.align = info*.align;92 return c;93}94 95kayan_column_row: proc(c: *kayan_column, row: u64) -> *u8 = {96 return c*.data[row * c*.size]&;97}98 99kayan_column_grow: proc(arena: *memops_arena, c: *kayan_column, count: u64, capacity: u64) -> void = {100 data: ^u8 = cast(memops_arena_push_zero(arena, c*.size * capacity, c*.align), ^u8);101 if (count > 0) {102 memcpy(data, c*.data, c*.size * count);103 }104 c*.data = data;105}106 107kayan_column_copy_row: proc(c: *kayan_column, to: u64, from: u64) -> void = {108 memcpy(kayan_column_row(c, to), kayan_column_row(c, from), c*.size);109}110 111kayan_column_zero_row: proc(c: *kayan_column, row: u64) -> void = {112 memset(kayan_column_row(c, row), 0, c*.size);113}114 115kayan_column_slice: proc<T>(c: *kayan_column, count: u64) -> []T = {116 if (c*.size != sizeof(T)) {117 printfmt("kayan: a column of {}-byte rows read as a {}-byte type\n", c*.size, sizeof(T));118 exit(1);119 }120 raw: ^T = cast(c*.data, ^T);121 return raw[0..count];122}123 124// ---------------------------------------------------------------------------125// The table126// ---------------------------------------------------------------------------127 128// `columns[id]` is in use when the signature has bit `id`; the others stay129// zeroed. Count and capacity are the table's, shared by its columns and by130// `entities`, so they can never disagree.131//132// The edges: `add_edges[id]` is the table this one's entities move to when133// they gain component `id`, and `remove_edges[id]` when they lose it. Each is134// the table's index plus one, so 0, which a new table starts with, means "not135// known yet".136kayan_table: struct = {137 signature: u64;138 count: u64;139 capacity: u64;140 entities: ^kayan_entity;141 columns: [64]kayan_column;142 add_edges: [64]u32;143 remove_edges: [64]u32;144}145 146// The world: the registry, the slots and free list, and every table, with a147// map from signature to table index so finding one is a lookup, not a search.148// Tables are referred to by index, never by pointer: making a table can grow149// `tables` and move all of them, and an index survives that.150kayan_world: struct = {151 arena: *memops_arena;152 infos: [64]*const reflect;153 component_count: u32;154 tables: vec<kayan_table>;155 by_signature: hashmap<u64, u32>;156 slots: vec<kayan_slot>;157 free: vec<u32>;158 live: u64;159 // How table moves found their destination, for this chapter's output.160 lookups: u64;161 edge_hits: u64;162}163 164kayan_bit: proc(id: u32) -> u64 = {165 return 1'u64 shl cast(id, u64);166}167 168kayan_id_of: proc<T>() -> u32 = {169 return cast(component_of<T>(), u32);170}171 172kayan_mask_of: proc<T>() -> u64 = {173 return kayan_bit(kayan_id_of<T>());174}175 176kayan_register: proc(w: *kayan_world, id: u32, info: *const reflect) -> void = {177 if (id >= 64) {178 printfmt("kayan: component id {} is past the 64 a signature can hold\n", id);179 exit(1);180 }181 w*.infos[id] = info;182 if (id + 1 > w*.component_count) {183 w*.component_count = id + 1;184 }185}186 187kayan_table_has: proc(t: *kayan_table, id: u32) -> b32 = {188 return (t*.signature & kayan_bit(id)) != 0;189}190 191// A table for a signature: one empty column per component in it, sized from192// the registry. Every component in the signature must be registered.193kayan_table_init: proc(w: *kayan_world, signature: u64) -> kayan_table = {194 t: kayan_table = {};195 t.signature = signature;196 for (id: u32 = 0; id < 64; id += 1) {197 if ((signature & kayan_bit(id)) != 0) {198 if (id >= w*.component_count || w*.infos[id] == null) {199 printfmt("kayan: signature {} uses component {}, which was never registered\n", signature, id);200 exit(1);201 }202 t.columns[id] = kayan_column_init(w*.infos[id]);203 }204 }205 return t;206}207 208// Double the room in `entities` and in every column at once.209kayan_table_grow: proc(w: *kayan_world, t: *kayan_table) -> void = {210 capacity: u64 = t*.capacity == 0 ? 4 : t*.capacity * 2;211 entities: ^kayan_entity = cast(memops_arena_push_zero(w*.arena,212 sizeof(kayan_entity) * capacity, alignof(kayan_entity)), ^kayan_entity);213 if (t*.count > 0) {214 memcpy(entities, t*.entities, sizeof(kayan_entity) * t*.count);215 }216 t*.entities = entities;217 for (id: u32 = 0; id < w*.component_count; id += 1) {218 if (kayan_table_has(t, id)) {219 kayan_column_grow(w*.arena, t*.columns[id]&, t*.count, capacity);220 }221 }222 t*.capacity = capacity;223}224 225// A new row at the end, owned by `e`, with every component zeroed.226kayan_table_add_row: proc(w: *kayan_world, t: *kayan_table, e: kayan_entity) -> u64 = {227 if (t*.count == t*.capacity) {228 kayan_table_grow(w, t);229 }230 row: u64 = t*.count;231 t*.count += 1;232 t*.entities[row] = e;233 for (id: u32 = 0; id < w*.component_count; id += 1) {234 if (kayan_table_has(t, id)) {235 kayan_column_zero_row(t*.columns[id]&, row);236 }237 }238 return row;239}240 241// 1.4's swap-remove, for every column: the last row is copied over `row`, and242// the slot of the entity that owned the last row is told where it went.243kayan_table_remove_row: proc(w: *kayan_world, t: *kayan_table, row: u64) -> void = {244 last: u64 = t*.count - 1;245 if (row != last) {246 for (id: u32 = 0; id < w*.component_count; id += 1) {247 if (kayan_table_has(t, id)) {248 kayan_column_copy_row(t*.columns[id]&, row, last);249 }250 }251 moved: kayan_entity = t*.entities[last];252 t*.entities[row] = moved;253 vec<kayan_slot>at(w*.slots&, cast(moved.index, u64))*.row = cast(row, u32);254 }255 t*.count = last;256}257 258// The typed view of one of the table's columns: T's rows, all of them in use.259// Asking for a component the table does not have is a bug, so it stops.260kayan_slice: proc<T>(t: *kayan_table) -> []T = {261 id: u32 = kayan_id_of<T>();262 if (!kayan_table_has(t, id)) {263 printfmt("kayan: the table has no column for component {}\n", id);264 exit(1);265 }266 return kayan_column_slice<T>(t*.columns[id]&, t*.count);267}268 269// A pointer to a table is good until the next table is made.270kayan_table_at: proc(w: *kayan_world, index: u32) -> *kayan_table = {271 return vec<kayan_table>at(w*.tables&, cast(index, u64));272}273 274// The table for a signature: found if it exists, made if it does not. This is275// where archetypes come from.276kayan_table_for: proc(w: *kayan_world, signature: u64) -> u32 = {277 found: *u32 = hashmap<u64, u32>get(w*.by_signature&, signature);278 if (found != null) {279 return found*;280 }281 index: u32 = cast(w*.tables.length, u32);282 _ = vec<kayan_table>append(w*.arena, w*.tables&, kayan_table_init(w, signature));283 _ = hashmap<u64, u32>put(w*.arena, w*.by_signature&, signature, index);284 return index;285}286 287// ---------------------------------------------------------------------------288// The world289// ---------------------------------------------------------------------------290 291kayan_world_init: proc(w: *kayan_world, arena: *memops_arena) -> void = {292 w* = {};293 w*.arena = arena;294}295 296kayan_alive: proc(w: *kayan_world, e: kayan_entity) -> b32 = {297 if (cast(e.index, u64) >= w*.slots.length) {298 return false;299 }300 return vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.generation == e.generation;301}302 303// A new entity with the components in `signature`, all zeroed, in the table304// for that signature.305kayan_spawn: proc(w: *kayan_world, signature: u64) -> kayan_entity = {306 e: kayan_entity = {};307 if (!vec<u32>is_empty(w*.free&)) {308 e.index = vec<u32>pop(w*.free&);309 } else {310 e.index = cast(w*.slots.length, u32);311 _ = vec<kayan_slot>append(w*.arena, w*.slots&, { .generation = 1 });312 }313 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));314 e.generation = slot*.generation;315 // kayan_table_for can make a table and move the others, so the table's316 // pointer is taken after it.317 table_index: u32 = kayan_table_for(w, signature);318 t: *kayan_table = kayan_table_at(w, table_index);319 slot*.table = table_index;320 slot*.row = cast(kayan_table_add_row(w, t, e), u32);321 w*.live += 1;322 return e;323}324 325kayan_despawn: proc(w: *kayan_world, e: kayan_entity) -> kayan_status = {326 if (!kayan_alive(w, e)) {327 return kayan_status.dead;328 }329 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));330 kayan_table_remove_row(w, kayan_table_at(w, slot*.table), cast(slot*.row, u64));331 slot*.generation += 1;332 _ = vec<u32>append(w*.arena, w*.free&, e.index);333 w*.live -= 1;334 return kayan_status.ok;335}336 337// The components a live entity has: its table's signature. 0 for a dead one.338kayan_signature: proc(w: *kayan_world, e: kayan_entity) -> u64 = {339 if (!kayan_alive(w, e)) {340 return 0;341 }342 return kayan_table_at(w, vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.table)*.signature;343}344 345// ---------------------------------------------------------------------------346// Moving between tables347// ---------------------------------------------------------------------------348 349// The table an entity of table `from` moves to when it gains component `id`.350// The first time, it is found by signature, which can make the table; the351// answer is written on both tables, as from's add edge and the destination's352// remove edge, so neither direction is looked up again.353kayan_table_with: proc(w: *kayan_world, from: u32, id: u32) -> u32 = {354 edge: u32 = kayan_table_at(w, from)*.add_edges[id];355 if (edge != 0) {356 w*.edge_hits += 1;357 return edge - 1;358 }359 w*.lookups += 1;360 to: u32 = kayan_table_for(w, kayan_table_at(w, from)*.signature | kayan_bit(id));361 // kayan_table_for can move the tables, so both pointers are taken after it.362 kayan_table_at(w, from)*.add_edges[id] = to + 1;363 kayan_table_at(w, to)*.remove_edges[id] = from + 1;364 return to;365}366 367// The same for losing component `id`.368kayan_table_without: proc(w: *kayan_world, from: u32, id: u32) -> u32 = {369 edge: u32 = kayan_table_at(w, from)*.remove_edges[id];370 if (edge != 0) {371 w*.edge_hits += 1;372 return edge - 1;373 }374 w*.lookups += 1;375 to: u32 = kayan_table_for(w, kayan_table_at(w, from)*.signature & ~kayan_bit(id));376 kayan_table_at(w, from)*.remove_edges[id] = to + 1;377 kayan_table_at(w, to)*.add_edges[id] = from + 1;378 return to;379}380 381// Move a live entity to table `to_index`. Components both tables have are382// copied, byte for byte; components only the new table has start zeroed;383// components only the old table has are dropped with its old row.384kayan_move: proc(w: *kayan_world, e: kayan_entity, to_index: u32) -> void = {385 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));386 from_index: u32 = slot*.table;387 if (from_index == to_index) {388 return;389 }390 from: *kayan_table = kayan_table_at(w, from_index);391 to: *kayan_table = kayan_table_at(w, to_index);392 from_row: u64 = cast(slot*.row, u64);393 to_row: u64 = kayan_table_add_row(w, to, e);394 for (id: u32 = 0; id < w*.component_count; id += 1) {395 if (kayan_table_has(from, id) && kayan_table_has(to, id)) {396 memcpy(kayan_column_row(to*.columns[id]&, to_row),397 kayan_column_row(from*.columns[id]&, from_row), from*.columns[id].size);398 }399 }400 kayan_table_remove_row(w, from, from_row);401 slot*.table = to_index;402 slot*.row = cast(to_row, u32);403}404 405// ---------------------------------------------------------------------------406// The byte layer, for tools407// ---------------------------------------------------------------------------408 409// e's component `id` as its bytes: an empty slice if e is dead or has no such410// component. The typed layer is for game code, which knows its types; this is411// for tools that work from reflect records instead.412kayan_get_bytes: proc(w: *kayan_world, e: kayan_entity, id: u32) -> []u8 = {413 none: []u8 = {};414 if (!kayan_alive(w, e) || id >= w*.component_count) {415 return none;416 }417 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));418 t: *kayan_table = kayan_table_at(w, slot*.table);419 if (!kayan_table_has(t, id)) {420 return none;421 }422 c: *kayan_column = t*.columns[id]&;423 row: u64 = cast(slot*.row, u64);424 all: []u8 = c*.data[0..t*.count * c*.size];425 return all[row * c*.size..(row + 1) * c*.size];426}427 428// ---------------------------------------------------------------------------429// The typed layer430// ---------------------------------------------------------------------------431 432// e's T, or null if e is dead or has no T. Handle, slot, table, row, typed433// view: the same path as 1.4's get_position, for any component. The pointer is434// good until the next structural change, so use it now and keep the handle.435kayan_get: proc<T>(w: *kayan_world, e: kayan_entity) -> *T = {436 if (!kayan_alive(w, e)) {437 return null;438 }439 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));440 t: *kayan_table = kayan_table_at(w, slot*.table);441 if (!kayan_table_has(t, kayan_id_of<T>())) {442 return null;443 }444 rows: []T = kayan_slice<T>(t);445 return rows[cast(slot*.row, u64)]&;446}447 448kayan_has: proc<T>(w: *kayan_world, e: kayan_entity) -> b32 = {449 return (kayan_signature(w, e) & kayan_mask_of<T>()) != 0;450}451 452// Write e's T. It must already have one: giving an entity a component it does453// not have changes its table, which is 2.6.454kayan_set: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> kayan_status = {455 if (!kayan_alive(w, e)) {456 return kayan_status.dead;457 }458 p: *T = kayan_get<T>(w, e);459 if (p == null) {460 return kayan_status.missing;461 }462 p* = value;463 return kayan_status.ok;464}465 466// Give e a T. If it already has one, this is kayan_set; if not, it moves to467// the table with T and then sets it.468kayan_add: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> kayan_status = {469 if (!kayan_alive(w, e)) {470 return kayan_status.dead;471 }472 if (!kayan_has<T>(w, e)) {473 from: u32 = vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.table;474 kayan_move(w, e, kayan_table_with(w, from, kayan_id_of<T>()));475 }476 return kayan_set<T>(w, e, value);477}478 479// Take e's T away, moving it to the table without T.480kayan_remove: proc<T>(w: *kayan_world, e: kayan_entity) -> kayan_status = {481 if (!kayan_alive(w, e)) {482 return kayan_status.dead;483 }484 if (!kayan_has<T>(w, e)) {485 return kayan_status.missing;486 }487 from: u32 = vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.table;488 kayan_move(w, e, kayan_table_without(w, from, kayan_id_of<T>()));489 return kayan_status.ok;490}491 492// The invariant, over every table: each row belongs to a live entity whose493// slot points back at that table and row, and the rows of all tables add up to494// the live count.495kayan_verify: proc(w: *kayan_world) -> u64 = {496 problems: u64 = 0;497 rows: u64 = 0;498 for (ti: u64 = 0; ti < w*.tables.length; ti += 1) {499 t: *kayan_table = kayan_table_at(w, cast(ti, u32));500 for (row: u64 = 0; row < t*.count; row += 1) {501 e: kayan_entity = t*.entities[row];502 if (!kayan_alive(w, e)) {503 printfmt("kayan_verify: table {} row {} holds dead entity {}:{}\n", ti, row, e.index, e.generation);504 problems += 1;505 continue;506 }507 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));508 if (cast(slot*.table, u64) != ti || cast(slot*.row, u64) != row) {509 printfmt("kayan_verify: {}:{} is in table {} row {}, its slot says table {} row {}\n",510 e.index, e.generation, ti, row, slot*.table, slot*.row);511 problems += 1;512 }513 }514 rows += t*.count;515 }516 if (rows != w*.live) {517 printfmt("kayan_verify: {} rows in tables, {} live entities\n", rows, w*.live);518 problems += 1;519 }520 return problems;521}522 523kayan_print_signature: proc(w: *kayan_world, signature: u64) -> void = {524 printf("{");525 first: b32 = true;526 for (id: u32 = 0; id < w*.component_count; id += 1) {527 if ((signature & kayan_bit(id)) != 0) {528 printf(first ? "%s" : " %s", w*.infos[id]*.name);529 first = false;530 }531 }532 printf("}");533}534 535kayan_print_tables: proc(w: *kayan_world) -> void = {536 for (ti: u64 = 0; ti < w*.tables.length; ti += 1) {537 t: *kayan_table = kayan_table_at(w, cast(ti, u32));538 printf(" table %u ", cast(ti, u32));539 kayan_print_signature(w, t*.signature);540 printf(":");541 for (row: u64 = 0; row < t*.count; row += 1) {542 printf(" %u:%u", t*.entities[row].index, t*.entities[row].generation);543 }544 printf("\n");545 }546}547 548// ---------------------------------------------------------------------------549// The inspector: reflect records, no component types550// ---------------------------------------------------------------------------551 552kayan_indent: proc(depth: u32) -> void = {553 for (i: u32 = 0; i < depth; i += 1) {554 printf(" ");555 }556}557 558// One value of a builtin type, read from its bytes by its type's name.559kayan_print_builtin: proc(type_name: ^const char, at: []const u8) -> void = {560 if (strcmp(type_name, "f32") == 0) {561 printf("%.2f", cast(cast(at[0]&, *const f32)*, f64));562 } else if (strcmp(type_name, "u32") == 0) {563 printf("%u", cast(at[0]&, *const u32)*);564 } else if (strcmp(type_name, "i32") == 0) {565 printf("%d", cast(at[0]&, *const i32)*);566 } else {567 printf("<%s>", type_name);568 }569}570 571// Every field of a struct whose bytes are `base`, nested structs indented572// under their field. A field's bytes are cut from `base` by its offset and573// size, both from the reflect record.574kayan_print_fields: proc(type: *const reflect, base: []const u8, depth: u32) -> void = {575 fields: ^const reflect_field = reflect_fields(type);576 for (i: u64 = 0; i < type*.count; i += 1) {577 f: *const reflect_field = fields[i]&;578 if (reflect_field_has_attr(f, "hidden")) {579 continue;580 }581 at: []const u8 = base[f*.offset..f*.offset + f*.size];582 kayan_indent(depth);583 printf("%s:", f*.name);584 info: *const reflect = f*.info;585 if (info != null && info*.kind == reflect_kind_struct) {586 printf("\n");587 kayan_print_fields(info, at, depth + 1);588 } else {589 printf(" ");590 kayan_print_builtin(f*.type, at);591 printf("\n");592 }593 }594}595 596kayan_inspect: proc(w: *kayan_world, e: kayan_entity) -> void = {597 if (!kayan_alive(w, e)) {598 printfmt("{}:{} (dead)\n", e.index, e.generation);599 return;600 }601 printf("%u:%u ", e.index, e.generation);602 signature: u64 = kayan_signature(w, e);603 kayan_print_signature(w, signature);604 printf("\n");605 for (id: u32 = 0; id < w*.component_count; id += 1) {606 if ((signature & kayan_bit(id)) != 0) {607 info: *const reflect = w*.infos[id];608 printf(" %s\n", info*.name);609 bytes: []u8 = kayan_get_bytes(w, e, id);610 kayan_print_fields(info, bytes, 2);611 }612 }613}614 615// Every table: rows in use out of room, and per column its row size and its616// bytes in use out of bytes allocated. The memory a table holds and does not617// use is the cost of doubling on growth.618kayan_print_layout: proc(w: *kayan_world) -> void = {619 for (ti: u32 = 0; ti < cast(w*.tables.length, u32); ti += 1) {620 t: *kayan_table = kayan_table_at(w, ti);621 printf("table %u ", ti);622 kayan_print_signature(w, t*.signature);623 printf(": %u rows, room for %u\n", cast(t*.count, u32), cast(t*.capacity, u32));624 for (id: u32 = 0; id < w*.component_count; id += 1) {625 if (kayan_table_has(t, id)) {626 size: u64 = t*.columns[id].size;627 printf(" %-9s %2u bytes a row, %4u of %4u bytes in use\n", w*.infos[id]*.name, cast(size, u32),628 cast(size * t*.count, u32), cast(size * t*.capacity, u32));629 }630 }631 }632}633 634// ---------------------------------------------------------------------------635// The program636// ---------------------------------------------------------------------------637 638spawn_drone: proc(w: *kayan_world, x: f32, y: f32) -> kayan_entity = {639 e: kayan_entity = kayan_spawn(w, kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<health>());640 _ = kayan_set<position>(w, e, { .at = { .x = x, .y = y } });641 _ = kayan_set<velocity>(w, e, { .per_second = { .x = -1.0, .y = 0.0 } });642 _ = kayan_set<health>(w, e, { .points = 100.0, .max = 100.0 });643 return e;644}645 646main: proc() -> i32 = {647 arena: memops_arena = {};648 memops_arena_initialize(arena&);649 w: kayan_world = {};650 kayan_world_init(w&, arena&);651 kayan_register(w&, kayan_id_of<position>(), position<>&);652 kayan_register(w&, kayan_id_of<velocity>(), velocity<>&);653 kayan_register(w&, kayan_id_of<health>(), health<>&);654 kayan_register(w&, kayan_id_of<damage>(), damage<>&);655 kayan_register(w&, kayan_id_of<lifetime>(), lifetime<>&);656 kayan_register(w&, kayan_id_of<turret>(), turret<>&);657 kayan_register(w&, kayan_id_of<burning>(), burning<>&);658 659 gun: kayan_entity = kayan_spawn(w&, kayan_mask_of<position>() | kayan_mask_of<turret>());660 _ = kayan_set<turret>(w&, gun, { .cooldown = 0.5, .timer = 0.25 });661 a: kayan_entity = spawn_drone(w&, 20.0, 5.0);662 b: kayan_entity = spawn_drone(w&, 30.0, -5.0);663 _ = kayan_add<burning>(w&, b, { .per_second = 10.0 });664 h: *health = kayan_get<health>(w&, b);665 h*.points = 60.0;666 shot: kayan_entity = kayan_spawn(w&, kayan_mask_of<position>() | kayan_mask_of<velocity>()667 | kayan_mask_of<damage>() | kayan_mask_of<lifetime>());668 _ = kayan_set<velocity>(w&, shot, { .per_second = { .x = 8.0, .y = 0.0 } });669 _ = kayan_set<damage>(w&, shot, { .amount = 25.0 });670 _ = kayan_set<lifetime>(w&, shot, { .seconds = 2.0 });671 672 kayan_inspect(w&, gun);673 kayan_inspect(w&, a);674 kayan_inspect(w&, b);675 kayan_inspect(w&, shot);676 _ = kayan_despawn(w&, a);677 kayan_inspect(w&, a);678 printfmt("\n");679 kayan_print_layout(w&);680 printfmt("kayan_verify: {} problems\n", kayan_verify(w&));681 return 0;682}What inspecting prints. tools/check.py fails the chapter if a single byte differs.
0:1 {position turret}
position
at:
x: 0.00
y: 0.00
turret
cooldown: 0.50
1:1 {position velocity health}
position
at:
x: 20.00
y: 5.00
velocity
per_second:
x: -1.00
y: 0.00
health
points: 100.00
max: 100.00
2:1 {position velocity health burning}
position
at:
x: 30.00
y: -5.00
velocity
per_second:
x: -1.00
y: 0.00
health
points: 60.00
max: 100.00
burning
per_second: 10.00
3:1 {position velocity damage lifetime}
position
at:
x: 0.00
y: 0.00
velocity
per_second:
x: 8.00
y: 0.00
damage
amount: 25.00
lifetime
seconds: 2.00
1:1 (dead)
table 0 {position turret}: 1 rows, room for 4
position 8 bytes a row, 8 of 32 bytes in use
turret 8 bytes a row, 8 of 32 bytes in use
table 1 {position velocity health}: 0 rows, room for 4
position 8 bytes a row, 0 of 32 bytes in use
velocity 8 bytes a row, 0 of 32 bytes in use
health 8 bytes a row, 0 of 32 bytes in use
table 2 {position velocity health burning}: 1 rows, room for 4
position 8 bytes a row, 8 of 32 bytes in use
velocity 8 bytes a row, 8 of 32 bytes in use
health 8 bytes a row, 8 of 32 bytes in use
burning 4 bytes a row, 4 of 16 bytes in use
table 3 {position velocity damage lifetime}: 1 rows, room for 4
position 8 bytes a row, 8 of 32 bytes in use
velocity 8 bytes a row, 8 of 32 bytes in use
damage 4 bytes a row, 4 of 16 bytes in use
lifetime 4 bytes a row, 4 of 16 bytes in use
kayan_verify: 0 problems
Build an ECS · part 3, systems · 3.1
Queries
ecs/src/3.systems/3.1.queries/queries.rin
In the dojo: 6. bitsets, 12. a watermark, each on its own, to watch and to practise.
Part 2 finished the storage. Everything from here on is behaviour, and in an ECS behaviour lives in systems: code that runs over every entity with some set of components. "Every entity with a position and a velocity" isn't in one place. It's spread over every table whose signature has both bits. So a system's first job is finding those tables, and the thing that finds them is a query:
all {position velocity}, none {} -> tables 1, 2, 3
A query matches tables, not entities. A game has a handful of tables and rarely makes new ones, so the matching is done once and kept. Each frame, a system walks the kept list, and inside each table streams the columns.
Step 1: the query
kayan_query: struct = {
all: u64;
none: u64;
tables: vec<u32>;
checked: u64;
}
all: a table must have every one of these components.none: and none of these. "Drones that aren't burning" isall {health}, none {burning}.tables: the matching tables, by index. Indices, as always, because pointers to tables go stale.checked: how many of the world's tables the query has looked at.
Matching is two ANDs on the signature:
kayan_matches: proc(signature: u64, all: u64, none: u64) -> b32 = {
return (signature & all) == all && (signature & none) == 0;
}
Step 2: keeping it up to date
Tables are only ever added, always at the end of w.tables, and never removed. So a query that has checked the first checked tables only needs to look at the ones after that:
kayan_query_update: proc(w: *kayan_world, q: *kayan_query) -> void = {
for (ti: u64 = q*.checked; ti < w*.tables.length; ti += 1) {
if (kayan_matches(kayan_table_at(w, cast(ti, u32))*.signature, q*.all, q*.none)) {
_ = vec<u32>append(w*.arena, q*.tables&, cast(ti, u32));
}
}
q*.checked = w*.tables.length;
}
The first update checks every table. After that, a frame with no new tables costs one comparison. A new table is checked once, by each query, the next time each is updated. flecs does the same thing in reverse: when a table is made, it tells the queries.
Step 3: walking a query
This is the shape of every system:
kayan_query_update(w&, movers&);
for (i: u64 = 0; i < movers.tables.length; i += 1) {
t: *kayan_table = kayan_query_table(w&, movers&, i);
ps: []position = kayan_slice<position>(t);
vs: []velocity = kayan_slice<velocity>(t);
for (row: u64 = 0; row < t*.count; row += 1) {
ps[row].at.x += vs[row].per_second.x * 0.5;
ps[row].at.y += vs[row].per_second.y * 0.5;
}
}
Two loops. The outer one is over tables, a few iterations. The inner one is over rows: two dense arrays, read and written in order. That's lesson 1's move_soa. The query's job is to hand you the arrays; what you do with them is the same data-oriented loop as always.
The drones and the bullet are in different tables, and the loop doesn't care. The turret isn't matched (no velocity), and the loop never sees it, not even to skip it.
Run it
movers: all {position velocity}, none {}; tables: 2, entities: 4
table 1 {position velocity health}: 1:1 2:1 3:1
table 2 {position velocity damage lifetime}: 4:1
targets: all {position health}, none {burning}; tables: 1, entities: 3
table 1 {position velocity health}: 1:1 2:1 3:1
on_fire: all {health burning}, none {}; tables: 0, entities: 0
Then drone 2:1 catches fire, which makes table 3, after the queries were made:
2:1 catches fire, and the world makes table 3 for it
the queries have checked 3 tables; updating them checks the new one only
movers: all {position velocity}, none {}; tables: 3, entities: 4
table 1 {position velocity health}: 1:1 3:1
table 2 {position velocity damage lifetime}: 4:1
table 3 {position velocity health burning}: 2:1
targets: all {position health}, none {burning}; tables: 1, entities: 2
table 1 {position velocity health}: 1:1 3:1
on_fire: all {health burning}, none {}; tables: 1, entities: 1
table 3 {position velocity health burning}: 2:1
movers picked up the new table, targets refused it (none {burning}), and on_fire got its first. Nobody re-scanned the old tables.
Last, the bullet is despawned. Its table is now empty, and stays in movers. Walking an empty table costs one look at count.
Why this is fast
A query's cost doesn't depend on how many entities there are. It depends on how many tables match, and that number is small and stable. 10,000 drones in one table is one table to visit. That's the payoff of grouping entities by signature back in 2.4: the "which entities have X and Y?" question gets answered per table, once, instead of per entity, every frame.
The flip side is archetype fragmentation (2.4's Try): if every entity had a unique signature, every query would match thousands of one-row tables.
Try
- Write a query for "everything with a position that isn't a turret" and print it.
- Make 100 drones and walk
movers. How many times does the outer loop run? - Add
any: u64to the query: at least one of these components. Where inkayan_matchesdoes it go?
1// 3.1 Queries2//3// Part 2 finished the storage. Everything from here is behaviour, and in an4// ECS behaviour is a system: code that runs over every entity with some set of5// components. "Every entity with a position and a velocity" is spread over6// every table whose signature has both bits, so a system's first job is7// finding those tables. That is a query:8//9// all {position velocity}, none {} -> tables 1, 2, 310//11// A query matches tables, not entities. A game has a handful of tables and12// makes new ones rarely, so the matching is done once and kept; each frame a13// system walks the kept list, and inside each table streams the columns.14 15import "std/print.rin"16import "std/memops.rin"17import "std/vec.rin"18import "std/hashmap.rin"19 20vec2: struct = {21 x: f32;22 y: f32;23}24 25position: struct = { at: vec2; }26velocity: struct = { per_second: vec2; }27health: struct = { points: f32; max: f32; }28damage: struct = { amount: f32; }29lifetime: struct = { seconds: f32; }30// A field tagged "hidden" is bookkeeping the inspector does not show.31turret: struct = { cooldown: f32; timer: f32 @ "hidden"; }32burning: struct = { per_second: f32; }33 34component: enum = {35 position,36 velocity,37 health,38 damage,39 lifetime,40 turret,41 burning,42}43 44component_of: proc<position>() -> component = { return component.position; }45component_of: proc<velocity>() -> component = { return component.velocity; }46component_of: proc<health>() -> component = { return component.health; }47component_of: proc<damage>() -> component = { return component.damage; }48component_of: proc<lifetime>() -> component = { return component.lifetime; }49component_of: proc<turret>() -> component = { return component.turret; }50component_of: proc<burning>() -> component = { return component.burning; }51 52// ---------------------------------------------------------------------------53// Entities (Part 1)54// ---------------------------------------------------------------------------55 56kayan_entity: struct = {57 index: u32;58 generation: u32;59}60 61// `missing`: the entity is alive but has no such component.62kayan_status: enum[must_use] = {63 ok,64 dead,65 missing,66}67 68// Where an entity lives: which table, and which row of it.69kayan_slot: struct = {70 generation: u32;71 table: u32;72 row: u32;73}74 75// ---------------------------------------------------------------------------76// Columns (2.2)77// ---------------------------------------------------------------------------78 79kayan_column: struct = {80 data: ^u8;81 size: u64;82 align: u64;83}84 85kayan_column_init: proc(info: *const reflect) -> kayan_column = {86 c: kayan_column = {};87 c.size = info*.size;88 c.align = info*.align;89 return c;90}91 92kayan_column_row: proc(c: *kayan_column, row: u64) -> *u8 = {93 return c*.data[row * c*.size]&;94}95 96kayan_column_grow: proc(arena: *memops_arena, c: *kayan_column, count: u64, capacity: u64) -> void = {97 data: ^u8 = cast(memops_arena_push_zero(arena, c*.size * capacity, c*.align), ^u8);98 if (count > 0) {99 memcpy(data, c*.data, c*.size * count);100 }101 c*.data = data;102}103 104kayan_column_copy_row: proc(c: *kayan_column, to: u64, from: u64) -> void = {105 memcpy(kayan_column_row(c, to), kayan_column_row(c, from), c*.size);106}107 108kayan_column_zero_row: proc(c: *kayan_column, row: u64) -> void = {109 memset(kayan_column_row(c, row), 0, c*.size);110}111 112kayan_column_slice: proc<T>(c: *kayan_column, count: u64) -> []T = {113 if (c*.size != sizeof(T)) {114 printfmt("kayan: a column of {}-byte rows read as a {}-byte type\n", c*.size, sizeof(T));115 exit(1);116 }117 raw: ^T = cast(c*.data, ^T);118 return raw[0..count];119}120 121// ---------------------------------------------------------------------------122// The table123// ---------------------------------------------------------------------------124 125// `columns[id]` is in use when the signature has bit `id`; the others stay126// zeroed. Count and capacity are the table's, shared by its columns and by127// `entities`, so they can never disagree.128//129// The edges: `add_edges[id]` is the table this one's entities move to when130// they gain component `id`, and `remove_edges[id]` when they lose it. Each is131// the table's index plus one, so 0, which a new table starts with, means "not132// known yet".133kayan_table: struct = {134 signature: u64;135 count: u64;136 capacity: u64;137 entities: ^kayan_entity;138 columns: [64]kayan_column;139 add_edges: [64]u32;140 remove_edges: [64]u32;141}142 143// The world: the registry, the slots and free list, and every table, with a144// map from signature to table index so finding one is a lookup, not a search.145// Tables are referred to by index, never by pointer: making a table can grow146// `tables` and move all of them, and an index survives that.147kayan_world: struct = {148 arena: *memops_arena;149 infos: [64]*const reflect;150 component_count: u32;151 tables: vec<kayan_table>;152 by_signature: hashmap<u64, u32>;153 slots: vec<kayan_slot>;154 free: vec<u32>;155 live: u64;156}157 158kayan_bit: proc(id: u32) -> u64 = {159 return 1'u64 shl cast(id, u64);160}161 162kayan_id_of: proc<T>() -> u32 = {163 return cast(component_of<T>(), u32);164}165 166kayan_mask_of: proc<T>() -> u64 = {167 return kayan_bit(kayan_id_of<T>());168}169 170kayan_register: proc(w: *kayan_world, id: u32, info: *const reflect) -> void = {171 if (id >= 64) {172 printfmt("kayan: component id {} is past the 64 a signature can hold\n", id);173 exit(1);174 }175 w*.infos[id] = info;176 if (id + 1 > w*.component_count) {177 w*.component_count = id + 1;178 }179}180 181kayan_table_has: proc(t: *kayan_table, id: u32) -> b32 = {182 return (t*.signature & kayan_bit(id)) != 0;183}184 185// A table for a signature: one empty column per component in it, sized from186// the registry. Every component in the signature must be registered.187kayan_table_init: proc(w: *kayan_world, signature: u64) -> kayan_table = {188 t: kayan_table = {};189 t.signature = signature;190 for (id: u32 = 0; id < 64; id += 1) {191 if ((signature & kayan_bit(id)) != 0) {192 if (id >= w*.component_count || w*.infos[id] == null) {193 printfmt("kayan: signature {} uses component {}, which was never registered\n", signature, id);194 exit(1);195 }196 t.columns[id] = kayan_column_init(w*.infos[id]);197 }198 }199 return t;200}201 202// Double the room in `entities` and in every column at once.203kayan_table_grow: proc(w: *kayan_world, t: *kayan_table) -> void = {204 capacity: u64 = t*.capacity == 0 ? 4 : t*.capacity * 2;205 entities: ^kayan_entity = cast(memops_arena_push_zero(w*.arena,206 sizeof(kayan_entity) * capacity, alignof(kayan_entity)), ^kayan_entity);207 if (t*.count > 0) {208 memcpy(entities, t*.entities, sizeof(kayan_entity) * t*.count);209 }210 t*.entities = entities;211 for (id: u32 = 0; id < w*.component_count; id += 1) {212 if (kayan_table_has(t, id)) {213 kayan_column_grow(w*.arena, t*.columns[id]&, t*.count, capacity);214 }215 }216 t*.capacity = capacity;217}218 219// A new row at the end, owned by `e`, with every component zeroed.220kayan_table_add_row: proc(w: *kayan_world, t: *kayan_table, e: kayan_entity) -> u64 = {221 if (t*.count == t*.capacity) {222 kayan_table_grow(w, t);223 }224 row: u64 = t*.count;225 t*.count += 1;226 t*.entities[row] = e;227 for (id: u32 = 0; id < w*.component_count; id += 1) {228 if (kayan_table_has(t, id)) {229 kayan_column_zero_row(t*.columns[id]&, row);230 }231 }232 return row;233}234 235// 1.4's swap-remove, for every column: the last row is copied over `row`, and236// the slot of the entity that owned the last row is told where it went.237kayan_table_remove_row: proc(w: *kayan_world, t: *kayan_table, row: u64) -> void = {238 last: u64 = t*.count - 1;239 if (row != last) {240 for (id: u32 = 0; id < w*.component_count; id += 1) {241 if (kayan_table_has(t, id)) {242 kayan_column_copy_row(t*.columns[id]&, row, last);243 }244 }245 moved: kayan_entity = t*.entities[last];246 t*.entities[row] = moved;247 vec<kayan_slot>at(w*.slots&, cast(moved.index, u64))*.row = cast(row, u32);248 }249 t*.count = last;250}251 252// The typed view of one of the table's columns: T's rows, all of them in use.253// Asking for a component the table does not have is a bug, so it stops.254kayan_slice: proc<T>(t: *kayan_table) -> []T = {255 id: u32 = kayan_id_of<T>();256 if (!kayan_table_has(t, id)) {257 printfmt("kayan: the table has no column for component {}\n", id);258 exit(1);259 }260 return kayan_column_slice<T>(t*.columns[id]&, t*.count);261}262 263// A pointer to a table is good until the next table is made.264kayan_table_at: proc(w: *kayan_world, index: u32) -> *kayan_table = {265 return vec<kayan_table>at(w*.tables&, cast(index, u64));266}267 268// The table for a signature: found if it exists, made if it does not. This is269// where archetypes come from.270kayan_table_for: proc(w: *kayan_world, signature: u64) -> u32 = {271 found: *u32 = hashmap<u64, u32>get(w*.by_signature&, signature);272 if (found != null) {273 return found*;274 }275 index: u32 = cast(w*.tables.length, u32);276 _ = vec<kayan_table>append(w*.arena, w*.tables&, kayan_table_init(w, signature));277 _ = hashmap<u64, u32>put(w*.arena, w*.by_signature&, signature, index);278 return index;279}280 281// ---------------------------------------------------------------------------282// The world283// ---------------------------------------------------------------------------284 285kayan_world_init: proc(w: *kayan_world, arena: *memops_arena) -> void = {286 w* = {};287 w*.arena = arena;288}289 290kayan_alive: proc(w: *kayan_world, e: kayan_entity) -> b32 = {291 if (cast(e.index, u64) >= w*.slots.length) {292 return false;293 }294 return vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.generation == e.generation;295}296 297// A new entity with the components in `signature`, all zeroed, in the table298// for that signature.299kayan_spawn: proc(w: *kayan_world, signature: u64) -> kayan_entity = {300 e: kayan_entity = {};301 if (!vec<u32>is_empty(w*.free&)) {302 e.index = vec<u32>pop(w*.free&);303 } else {304 e.index = cast(w*.slots.length, u32);305 _ = vec<kayan_slot>append(w*.arena, w*.slots&, { .generation = 1 });306 }307 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));308 e.generation = slot*.generation;309 // kayan_table_for can make a table and move the others, so the table's310 // pointer is taken after it.311 table_index: u32 = kayan_table_for(w, signature);312 t: *kayan_table = kayan_table_at(w, table_index);313 slot*.table = table_index;314 slot*.row = cast(kayan_table_add_row(w, t, e), u32);315 w*.live += 1;316 return e;317}318 319kayan_despawn: proc(w: *kayan_world, e: kayan_entity) -> kayan_status = {320 if (!kayan_alive(w, e)) {321 return kayan_status.dead;322 }323 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));324 kayan_table_remove_row(w, kayan_table_at(w, slot*.table), cast(slot*.row, u64));325 slot*.generation += 1;326 _ = vec<u32>append(w*.arena, w*.free&, e.index);327 w*.live -= 1;328 return kayan_status.ok;329}330 331// The components a live entity has: its table's signature. 0 for a dead one.332kayan_signature: proc(w: *kayan_world, e: kayan_entity) -> u64 = {333 if (!kayan_alive(w, e)) {334 return 0;335 }336 return kayan_table_at(w, vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.table)*.signature;337}338 339// ---------------------------------------------------------------------------340// Moving between tables341// ---------------------------------------------------------------------------342 343// The table an entity of table `from` moves to when it gains component `id`.344// The first time, it is found by signature, which can make the table; the345// answer is written on both tables, as from's add edge and the destination's346// remove edge, so neither direction is looked up again.347kayan_table_with: proc(w: *kayan_world, from: u32, id: u32) -> u32 = {348 edge: u32 = kayan_table_at(w, from)*.add_edges[id];349 if (edge != 0) {350 return edge - 1;351 }352 to: u32 = kayan_table_for(w, kayan_table_at(w, from)*.signature | kayan_bit(id));353 // kayan_table_for can move the tables, so both pointers are taken after it.354 kayan_table_at(w, from)*.add_edges[id] = to + 1;355 kayan_table_at(w, to)*.remove_edges[id] = from + 1;356 return to;357}358 359// The same for losing component `id`.360kayan_table_without: proc(w: *kayan_world, from: u32, id: u32) -> u32 = {361 edge: u32 = kayan_table_at(w, from)*.remove_edges[id];362 if (edge != 0) {363 return edge - 1;364 }365 to: u32 = kayan_table_for(w, kayan_table_at(w, from)*.signature & ~kayan_bit(id));366 kayan_table_at(w, from)*.remove_edges[id] = to + 1;367 kayan_table_at(w, to)*.add_edges[id] = from + 1;368 return to;369}370 371// Move a live entity to table `to_index`. Components both tables have are372// copied, byte for byte; components only the new table has start zeroed;373// components only the old table has are dropped with its old row.374kayan_move: proc(w: *kayan_world, e: kayan_entity, to_index: u32) -> void = {375 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));376 from_index: u32 = slot*.table;377 if (from_index == to_index) {378 return;379 }380 from: *kayan_table = kayan_table_at(w, from_index);381 to: *kayan_table = kayan_table_at(w, to_index);382 from_row: u64 = cast(slot*.row, u64);383 to_row: u64 = kayan_table_add_row(w, to, e);384 for (id: u32 = 0; id < w*.component_count; id += 1) {385 if (kayan_table_has(from, id) && kayan_table_has(to, id)) {386 memcpy(kayan_column_row(to*.columns[id]&, to_row),387 kayan_column_row(from*.columns[id]&, from_row), from*.columns[id].size);388 }389 }390 kayan_table_remove_row(w, from, from_row);391 slot*.table = to_index;392 slot*.row = cast(to_row, u32);393}394 395// ---------------------------------------------------------------------------396// The byte layer, for tools397// ---------------------------------------------------------------------------398 399// e's component `id` as its bytes: an empty slice if e is dead or has no such400// component. The typed layer is for game code, which knows its types; this is401// for tools that work from reflect records instead.402kayan_get_bytes: proc(w: *kayan_world, e: kayan_entity, id: u32) -> []u8 = {403 none: []u8 = {};404 if (!kayan_alive(w, e) || id >= w*.component_count) {405 return none;406 }407 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));408 t: *kayan_table = kayan_table_at(w, slot*.table);409 if (!kayan_table_has(t, id)) {410 return none;411 }412 c: *kayan_column = t*.columns[id]&;413 row: u64 = cast(slot*.row, u64);414 all: []u8 = c*.data[0..t*.count * c*.size];415 return all[row * c*.size..(row + 1) * c*.size];416}417 418// ---------------------------------------------------------------------------419// The typed layer420// ---------------------------------------------------------------------------421 422// e's T, or null if e is dead or has no T. Handle, slot, table, row, typed423// view: the same path as 1.4's get_position, for any component. The pointer is424// good until the next structural change, so use it now and keep the handle.425kayan_get: proc<T>(w: *kayan_world, e: kayan_entity) -> *T = {426 if (!kayan_alive(w, e)) {427 return null;428 }429 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));430 t: *kayan_table = kayan_table_at(w, slot*.table);431 if (!kayan_table_has(t, kayan_id_of<T>())) {432 return null;433 }434 rows: []T = kayan_slice<T>(t);435 return rows[cast(slot*.row, u64)]&;436}437 438kayan_has: proc<T>(w: *kayan_world, e: kayan_entity) -> b32 = {439 return (kayan_signature(w, e) & kayan_mask_of<T>()) != 0;440}441 442// Write e's T. It must already have one: giving an entity a component it does443// not have changes its table, which is 2.6.444kayan_set: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> kayan_status = {445 if (!kayan_alive(w, e)) {446 return kayan_status.dead;447 }448 p: *T = kayan_get<T>(w, e);449 if (p == null) {450 return kayan_status.missing;451 }452 p* = value;453 return kayan_status.ok;454}455 456// Give e a T. If it already has one, this is kayan_set; if not, it moves to457// the table with T and then sets it.458kayan_add: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> kayan_status = {459 if (!kayan_alive(w, e)) {460 return kayan_status.dead;461 }462 if (!kayan_has<T>(w, e)) {463 from: u32 = vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.table;464 kayan_move(w, e, kayan_table_with(w, from, kayan_id_of<T>()));465 }466 return kayan_set<T>(w, e, value);467}468 469// Take e's T away, moving it to the table without T.470kayan_remove: proc<T>(w: *kayan_world, e: kayan_entity) -> kayan_status = {471 if (!kayan_alive(w, e)) {472 return kayan_status.dead;473 }474 if (!kayan_has<T>(w, e)) {475 return kayan_status.missing;476 }477 from: u32 = vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.table;478 kayan_move(w, e, kayan_table_without(w, from, kayan_id_of<T>()));479 return kayan_status.ok;480}481 482// The invariant, over every table: each row belongs to a live entity whose483// slot points back at that table and row, and the rows of all tables add up to484// the live count.485kayan_verify: proc(w: *kayan_world) -> u64 = {486 problems: u64 = 0;487 rows: u64 = 0;488 for (ti: u64 = 0; ti < w*.tables.length; ti += 1) {489 t: *kayan_table = kayan_table_at(w, cast(ti, u32));490 for (row: u64 = 0; row < t*.count; row += 1) {491 e: kayan_entity = t*.entities[row];492 if (!kayan_alive(w, e)) {493 printfmt("kayan_verify: table {} row {} holds dead entity {}:{}\n", ti, row, e.index, e.generation);494 problems += 1;495 continue;496 }497 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));498 if (cast(slot*.table, u64) != ti || cast(slot*.row, u64) != row) {499 printfmt("kayan_verify: {}:{} is in table {} row {}, its slot says table {} row {}\n",500 e.index, e.generation, ti, row, slot*.table, slot*.row);501 problems += 1;502 }503 }504 rows += t*.count;505 }506 if (rows != w*.live) {507 printfmt("kayan_verify: {} rows in tables, {} live entities\n", rows, w*.live);508 problems += 1;509 }510 return problems;511}512 513kayan_print_signature: proc(w: *kayan_world, signature: u64) -> void = {514 printf("{");515 first: b32 = true;516 for (id: u32 = 0; id < w*.component_count; id += 1) {517 if ((signature & kayan_bit(id)) != 0) {518 printf(first ? "%s" : " %s", w*.infos[id]*.name);519 first = false;520 }521 }522 printf("}");523}524 525kayan_print_table: proc(w: *kayan_world, ti: u32) -> void = {526 t: *kayan_table = kayan_table_at(w, ti);527 printf(" table %u ", ti);528 kayan_print_signature(w, t*.signature);529 printf(":");530 for (row: u64 = 0; row < t*.count; row += 1) {531 printf(" %u:%u", t*.entities[row].index, t*.entities[row].generation);532 }533 printf("\n");534}535 536kayan_print_tables: proc(w: *kayan_world) -> void = {537 for (ti: u32 = 0; ti < cast(w*.tables.length, u32); ti += 1) {538 kayan_print_table(w, ti);539 }540}541 542// ---------------------------------------------------------------------------543// The inspector: reflect records, no component types544// ---------------------------------------------------------------------------545 546kayan_indent: proc(depth: u32) -> void = {547 for (i: u32 = 0; i < depth; i += 1) {548 printf(" ");549 }550}551 552// One value of a builtin type, read from its bytes by its type's name.553kayan_print_builtin: proc(type_name: ^const char, at: []const u8) -> void = {554 if (strcmp(type_name, "f32") == 0) {555 printf("%.2f", cast(cast(at[0]&, *const f32)*, f64));556 } else if (strcmp(type_name, "u32") == 0) {557 printf("%u", cast(at[0]&, *const u32)*);558 } else if (strcmp(type_name, "i32") == 0) {559 printf("%d", cast(at[0]&, *const i32)*);560 } else {561 printf("<%s>", type_name);562 }563}564 565// Every field of a struct whose bytes are `base`, nested structs indented566// under their field. A field's bytes are cut from `base` by its offset and567// size, both from the reflect record.568kayan_print_fields: proc(type: *const reflect, base: []const u8, depth: u32) -> void = {569 fields: ^const reflect_field = reflect_fields(type);570 for (i: u64 = 0; i < type*.count; i += 1) {571 f: *const reflect_field = fields[i]&;572 if (reflect_field_has_attr(f, "hidden")) {573 continue;574 }575 at: []const u8 = base[f*.offset..f*.offset + f*.size];576 kayan_indent(depth);577 printf("%s:", f*.name);578 info: *const reflect = f*.info;579 if (info != null && info*.kind == reflect_kind_struct) {580 printf("\n");581 kayan_print_fields(info, at, depth + 1);582 } else {583 printf(" ");584 kayan_print_builtin(f*.type, at);585 printf("\n");586 }587 }588}589 590kayan_inspect: proc(w: *kayan_world, e: kayan_entity) -> void = {591 if (!kayan_alive(w, e)) {592 printfmt("{}:{} (dead)\n", e.index, e.generation);593 return;594 }595 printf("%u:%u ", e.index, e.generation);596 signature: u64 = kayan_signature(w, e);597 kayan_print_signature(w, signature);598 printf("\n");599 for (id: u32 = 0; id < w*.component_count; id += 1) {600 if ((signature & kayan_bit(id)) != 0) {601 info: *const reflect = w*.infos[id];602 printf(" %s\n", info*.name);603 bytes: []u8 = kayan_get_bytes(w, e, id);604 kayan_print_fields(info, bytes, 2);605 }606 }607}608 609// Every table: rows in use out of room, and per column its row size and its610// bytes in use out of bytes allocated. The memory a table holds and does not611// use is the cost of doubling on growth.612kayan_print_layout: proc(w: *kayan_world) -> void = {613 for (ti: u32 = 0; ti < cast(w*.tables.length, u32); ti += 1) {614 t: *kayan_table = kayan_table_at(w, ti);615 printf("table %u ", ti);616 kayan_print_signature(w, t*.signature);617 printf(": %u rows, room for %u\n", cast(t*.count, u32), cast(t*.capacity, u32));618 for (id: u32 = 0; id < w*.component_count; id += 1) {619 if (kayan_table_has(t, id)) {620 size: u64 = t*.columns[id].size;621 printf(" %-9s %2u bytes a row, %4u of %4u bytes in use\n", w*.infos[id]*.name, cast(size, u32),622 cast(size * t*.count, u32), cast(size * t*.capacity, u32));623 }624 }625 }626}627 628// ---------------------------------------------------------------------------629// Queries630// ---------------------------------------------------------------------------631 632// The tables a system runs over: every table whose signature has all of `all`633// and none of `none`, kept by index. Tables are only ever added, at the end of634// w.tables, so the query remembers how many it has checked and, brought up to635// date, looks at the new ones only.636kayan_query: struct = {637 all: u64;638 none: u64;639 tables: vec<u32>;640 checked: u64;641}642 643kayan_query_init: proc(all: u64, none: u64) -> kayan_query = {644 q: kayan_query = {};645 q.all = all;646 q.none = none;647 return q;648}649 650kayan_matches: proc(signature: u64, all: u64, none: u64) -> b32 = {651 return (signature & all) == all && (signature & none) == 0;652}653 654// Check the tables made since the query last looked. With none new, this is655// one comparison, so call it before every walk.656kayan_query_update: proc(w: *kayan_world, q: *kayan_query) -> void = {657 for (ti: u64 = q*.checked; ti < w*.tables.length; ti += 1) {658 if (kayan_matches(kayan_table_at(w, cast(ti, u32))*.signature, q*.all, q*.none)) {659 _ = vec<u32>append(w*.arena, q*.tables&, cast(ti, u32));660 }661 }662 q*.checked = w*.tables.length;663}664 665// The query's i-th table. Like any table pointer, good until a table is made.666kayan_query_table: proc(w: *kayan_world, q: *kayan_query, i: u64) -> *kayan_table = {667 return kayan_table_at(w, vec<u32>at(q*.tables&, i)*);668}669 670// How many entities the query matches: the rows of its tables.671kayan_query_count: proc(w: *kayan_world, q: *kayan_query) -> u64 = {672 kayan_query_update(w, q);673 n: u64 = 0;674 for (i: u64 = 0; i < q*.tables.length; i += 1) {675 n += kayan_query_table(w, q, i)*.count;676 }677 return n;678}679 680kayan_print_query: proc(w: *kayan_world, name: ^const char, q: *kayan_query) -> void = {681 kayan_query_update(w, q);682 printf("%s: all ", name);683 kayan_print_signature(w, q*.all);684 printf(", none ");685 kayan_print_signature(w, q*.none);686 printfmt("; tables: {}, entities: {}\n", q*.tables.length, kayan_query_count(w, q));687 for (i: u64 = 0; i < q*.tables.length; i += 1) {688 kayan_print_table(w, vec<u32>at(q*.tables&, i)*);689 }690}691 692// ---------------------------------------------------------------------------693// The program694// ---------------------------------------------------------------------------695 696spawn_turret: proc(w: *kayan_world, x: f32, y: f32) -> kayan_entity = {697 e: kayan_entity = kayan_spawn(w, kayan_mask_of<position>() | kayan_mask_of<turret>());698 _ = kayan_set<position>(w, e, { .at = { .x = x, .y = y } });699 _ = kayan_set<turret>(w, e, { .cooldown = 0.5, .timer = 0.0 });700 return e;701}702 703spawn_drone: proc(w: *kayan_world, x: f32, y: f32) -> kayan_entity = {704 e: kayan_entity = kayan_spawn(w, kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<health>());705 _ = kayan_set<position>(w, e, { .at = { .x = x, .y = y } });706 _ = kayan_set<velocity>(w, e, { .per_second = { .x = -1.0, .y = 0.0 } });707 _ = kayan_set<health>(w, e, { .points = 100.0, .max = 100.0 });708 return e;709}710 711spawn_bullet: proc(w: *kayan_world, x: f32, y: f32) -> kayan_entity = {712 e: kayan_entity = kayan_spawn(w, kayan_mask_of<position>() | kayan_mask_of<velocity>()713 | kayan_mask_of<damage>() | kayan_mask_of<lifetime>());714 _ = kayan_set<position>(w, e, { .at = { .x = x, .y = y } });715 _ = kayan_set<velocity>(w, e, { .per_second = { .x = 8.0, .y = 0.0 } });716 _ = kayan_set<damage>(w, e, { .amount = 25.0 });717 _ = kayan_set<lifetime>(w, e, { .seconds = 2.0 });718 return e;719}720 721// Every entity a query matches, table by table, with its position.722print_positions: proc(w: *kayan_world, q: *kayan_query) -> void = {723 kayan_query_update(w, q);724 for (i: u64 = 0; i < q*.tables.length; i += 1) {725 t: *kayan_table = kayan_query_table(w, q, i);726 ps: []position = kayan_slice<position>(t);727 for (row: u64 = 0; row < t*.count; row += 1) {728 e: kayan_entity = t*.entities[row];729 printf(" %u:%u at (%5.2f, %5.2f)\n", e.index, e.generation,730 cast(ps[row].at.x, f64), cast(ps[row].at.y, f64));731 }732 }733}734 735main: proc() -> i32 = {736 arena: memops_arena = {};737 memops_arena_initialize(arena&);738 w: kayan_world = {};739 kayan_world_init(w&, arena&);740 kayan_register(w&, kayan_id_of<position>(), position<>&);741 kayan_register(w&, kayan_id_of<velocity>(), velocity<>&);742 kayan_register(w&, kayan_id_of<health>(), health<>&);743 kayan_register(w&, kayan_id_of<damage>(), damage<>&);744 kayan_register(w&, kayan_id_of<lifetime>(), lifetime<>&);745 kayan_register(w&, kayan_id_of<turret>(), turret<>&);746 kayan_register(w&, kayan_id_of<burning>(), burning<>&);747 748 _ = spawn_turret(w&, 0.0, 0.0);749 _ = spawn_drone(w&, 20.0, 5.0);750 b: kayan_entity = spawn_drone(w&, 30.0, -5.0);751 _ = spawn_drone(w&, 40.0, 0.0);752 shot: kayan_entity = spawn_bullet(w&, 1.0, 0.0);753 printfmt("{} entities in {} tables:\n", w.live, w.tables.length);754 kayan_print_tables(w&);755 756 // Three queries. Making one costs nothing: it matches when first updated.757 movers: kayan_query = kayan_query_init(kayan_mask_of<position>() | kayan_mask_of<velocity>(), 0);758 targets: kayan_query = kayan_query_init(kayan_mask_of<position>() | kayan_mask_of<health>(), kayan_mask_of<burning>());759 on_fire: kayan_query = kayan_query_init(kayan_mask_of<health>() | kayan_mask_of<burning>(), 0);760 printfmt("\n");761 kayan_print_query(w&, "movers", movers&);762 kayan_print_query(w&, "targets", targets&);763 kayan_print_query(w&, "on_fire", on_fire&);764 765 // A system's shape: bring the query up to date, then for each of its766 // tables take the columns as slices and loop over the rows.767 kayan_query_update(w&, movers&);768 for (i: u64 = 0; i < movers.tables.length; i += 1) {769 t: *kayan_table = kayan_query_table(w&, movers&, i);770 ps: []position = kayan_slice<position>(t);771 vs: []velocity = kayan_slice<velocity>(t);772 for (row: u64 = 0; row < t*.count; row += 1) {773 ps[row].at.x += vs[row].per_second.x * 0.5;774 ps[row].at.y += vs[row].per_second.y * 0.5;775 }776 }777 printfmt("\nhalf a second of movement, over the movers' tables:\n");778 print_positions(w&, movers&);779 780 // A table made after the queries were.781 _ = kayan_add<burning>(w&, b, { .per_second = 10.0 });782 printfmt("\n{}:{} catches fire, and the world makes table {} for it\n", b.index, b.generation, w.tables.length - 1);783 printfmt("the queries have checked {} tables; updating them checks the new one only\n", movers.checked);784 kayan_print_query(w&, "movers", movers&);785 kayan_print_query(w&, "targets", targets&);786 kayan_print_query(w&, "on_fire", on_fire&);787 788 // A table emptied after the queries matched it.789 _ = kayan_despawn(w&, shot);790 printfmt("\nthe bullet is gone; its table is empty and stays matched:\n");791 kayan_print_query(w&, "movers", movers&);792 printfmt("kayan_verify: {} problems\n", kayan_verify(w&));793 return 0;794}What queries prints. tools/check.py fails the chapter if a single byte differs.
5 entities in 3 tables:
table 0 {position turret}: 0:1
table 1 {position velocity health}: 1:1 2:1 3:1
table 2 {position velocity damage lifetime}: 4:1
movers: all {position velocity}, none {}; tables: 2, entities: 4
table 1 {position velocity health}: 1:1 2:1 3:1
table 2 {position velocity damage lifetime}: 4:1
targets: all {position health}, none {burning}; tables: 1, entities: 3
table 1 {position velocity health}: 1:1 2:1 3:1
on_fire: all {health burning}, none {}; tables: 0, entities: 0
half a second of movement, over the movers' tables:
1:1 at (19.50, 5.00)
2:1 at (29.50, -5.00)
3:1 at (39.50, 0.00)
4:1 at ( 5.00, 0.00)
2:1 catches fire, and the world makes table 3 for it
the queries have checked 3 tables; updating them checks the new one only
movers: all {position velocity}, none {}; tables: 3, entities: 4
table 1 {position velocity health}: 1:1 3:1
table 2 {position velocity damage lifetime}: 4:1
table 3 {position velocity health burning}: 2:1
targets: all {position health}, none {burning}; tables: 1, entities: 2
table 1 {position velocity health}: 1:1 3:1
on_fire: all {health burning}, none {}; tables: 1, entities: 1
table 3 {position velocity health burning}: 2:1
the bullet is gone; its table is empty and stays matched:
movers: all {position velocity}, none {}; tables: 3, entities: 3
table 1 {position velocity health}: 1:1 3:1
table 2 {position velocity damage lifetime}:
table 3 {position velocity health burning}: 2:1
kayan_verify: 0 problems
Build an ECS · part 3, systems · 3.2
Systems
ecs/src/3.systems/3.2.systems/systems.rin
In the dojo: 13. a dispatch table, on its own, to watch and to practise.
3.1's movement loop had the shape of every system: bring a query up to date, then for each of its tables take the columns as slices and loop over the rows. This chapter names that shape and splits it in two:
- the kernel: a plain loop over plain slices. It knows nothing about tables, entities or worlds. This is lessons 1 to 7:
noalias, streaming, vectorization. - the system: a query, plus a small proc kayan calls on each table the query matches. It takes the table's columns and calls the kernel.
A frame runs the systems in order, each over all of its tables. The order is the schedule, and it's written down once, as data.
Step 1: the kernels
integrate: proc(ps: noalias []position, vs: []const velocity, dt: f32) -> void = {
for (i: u64 = 0; i < ps.length; i += 1) {
ps[i].at.x += vs[i].per_second.x * dt;
ps[i].at.y += vs[i].per_second.y * dt;
}
}
That's the whole thing, and it's the code you'd want to write if there were no ECS at all. Kernels are where you profile, read assembly and reach for SIMD (the lessons). Everything else in this course exists to hand them the right slices. burn and age look the same.
Step 2: what a system gets
kayan_iter: struct = {
world: *kayan_world;
table: *kayan_table;
dt: f32;
}
kayan_system_proc: alias = *proc(it: *kayan_iter) -> void;
One struct instead of three parameters. It's the same idea as flecs' ecs_iter_t, and it has two practical reasons. Adding to it later (3.3 adds a field) changes no system's signature. And a system that uses only the table still uses its one parameter. rin compiles to C with -Wextra -Werror, and an unused parameter is an error there.
The system itself is the adapter:
move_system: proc(it: *kayan_iter) -> void = {
integrate(kayan_slice<position>(it*.table), kayan_slice<velocity>(it*.table), it*.dt);
}
Columns out, kernel in. No loop here: the kernel has it.
Step 3: a system is data
kayan_system: struct = {
name: ^const char;
query: kayan_query;
run: kayan_system_proc;
tables: u64;
rows: u64;
}
A name, a query, a proc pointer. tables and rows count the work, for the output. The schedule is an array of these:
systems: [3]kayan_system = {};
systems[0] = kayan_system_init("burn", kayan_mask_of<health>() | kayan_mask_of<burning>(), 0, burn_system);
systems[1] = kayan_system_init("age", kayan_mask_of<lifetime>(), 0, age_system);
systems[2] = kayan_system_init("move", kayan_mask_of<position>() | kayan_mask_of<velocity>(), 0, move_system);
Step 4: a frame
kayan_progress: proc(w: *kayan_world, systems: []kayan_system, dt: f32) -> void = {
for (s: u64 = 0; s < systems.length; s += 1) {
sys: *kayan_system = systems[s]&;
kayan_query_update(w, sys*.query&);
for (i: u64 = 0; i < sys*.query.tables.length; i += 1) {
it: kayan_iter = {};
it.world = w;
it.table = kayan_query_table(w, sys*.query&, i);
it.dt = dt;
if (it.table*.count > 0) {
sys*.tables += 1;
sys*.rows += it.table*.count;
sys*.run(it&);
}
}
}
}
kayan_progress is flecs' name for the same thing. For each system in order, for each of its tables, one call. The game loop becomes:
for (frame: u32 = 1; frame <= 8; frame += 1) {
kayan_progress(w&, systems, dt);
Order is data flow
Systems run one after another, never interleaved, and each sees everything the earlier ones wrote this frame. So the order is a statement about data: a system that reads what another writes runs after it. Move before collide means bullets hit where they are now, not where they were. Damage before death means a drone dies in the frame its health hits zero, not the next. 3.4's schedule has nine systems, and each position in it is one of these decisions.
Bigger engines go further. Each system declares what it reads and writes, and the scheduler uses that to run systems that don't conflict at the same time, on other threads. Underneath, the order is still a list like this one.
Run it
the schedule, run in this order every frame:
burn: all {health burning}, none {}; tables: 1, entities: 1
table 2 {position velocity health burning}: 2:1
age: all {lifetime}, none {}; tables: 1, entities: 3
table 3 {position velocity damage lifetime}: 4:1 5:1 6:1
move: all {position velocity}, none {}; tables: 3, entities: 6
table 1 {position velocity health}: 1:1 3:1
table 2 {position velocity health burning}: 2:1
table 3 {position velocity damage lifetime}: 4:1 5:1 6:1
...
after frame 8, 2.00 seconds:
0:1 at ( 0.00, 0.00)
1:1 at (18.00, 4.00) health 100
3:1 at (26.00, -4.00) health 100
2:1 at (22.00, 0.00) health 84, burning
4:1 at (17.00, 0.00) lifetime -1.50
5:1 at (17.00, 4.00) lifetime -1.00
6:1 at (17.00, -4.00) lifetime -0.50
8 frames, one call per table per frame:
burn called 8 times, over 8 rows
age called 8 times, over 24 rows
move called 24 times, over 48 rows
3 bullets are out of time and still in the world
move made 24 calls in 8 frames, three tables a frame, and each call is one tight loop. Call overhead is per table, never per entity.
And the last line is the cliffhanger. Each bullet's lifetime ran out, and nothing removed it, because a system in this chapter changes values, never structure. Despawning from inside a system looks like a one-line fix. It isn't, and 3.3 shows why.
Try
- Swap
burnandmovein the schedule. Does anything change? Now think of a pair where it would. - Add a
healsystem for drones that aren't burning (all {health}, none {burning}). - Build in Release and find
integratein the assembly. Is it vectorized? (Lesson 1 shows how to look.)
1// 3.2 Systems2//3// 3.1's movement loop had the shape of every system: bring a query up to4// date, then for each of its tables take the columns as slices and loop over5// the rows. This chapter names that shape and splits it in two:6//7// the kernel a plain loop over plain slices, like lesson 1's move_soa. It8// knows nothing about tables, entities or the world.9// the system a query, and a small proc kayan calls on each table the10// query matches. It takes the table's columns and calls the11// kernel.12//13// A frame runs the systems in order, each over all of its tables. The order is14// the schedule, and it is written down once, as data.15 16import "std/print.rin"17import "std/memops.rin"18import "std/vec.rin"19import "std/hashmap.rin"20 21vec2: struct = {22 x: f32;23 y: f32;24}25 26position: struct = { at: vec2; }27velocity: struct = { per_second: vec2; }28health: struct = { points: f32; max: f32; }29damage: struct = { amount: f32; }30lifetime: struct = { seconds: f32; }31// A field tagged "hidden" is bookkeeping the inspector does not show.32turret: struct = { cooldown: f32; timer: f32 @ "hidden"; }33burning: struct = { per_second: f32; }34 35component: enum = {36 position,37 velocity,38 health,39 damage,40 lifetime,41 turret,42 burning,43}44 45component_of: proc<position>() -> component = { return component.position; }46component_of: proc<velocity>() -> component = { return component.velocity; }47component_of: proc<health>() -> component = { return component.health; }48component_of: proc<damage>() -> component = { return component.damage; }49component_of: proc<lifetime>() -> component = { return component.lifetime; }50component_of: proc<turret>() -> component = { return component.turret; }51component_of: proc<burning>() -> component = { return component.burning; }52 53// ---------------------------------------------------------------------------54// Entities (Part 1)55// ---------------------------------------------------------------------------56 57kayan_entity: struct = {58 index: u32;59 generation: u32;60}61 62// `missing`: the entity is alive but has no such component.63kayan_status: enum[must_use] = {64 ok,65 dead,66 missing,67}68 69// Where an entity lives: which table, and which row of it.70kayan_slot: struct = {71 generation: u32;72 table: u32;73 row: u32;74}75 76// ---------------------------------------------------------------------------77// Columns (2.2)78// ---------------------------------------------------------------------------79 80kayan_column: struct = {81 data: ^u8;82 size: u64;83 align: u64;84}85 86kayan_column_init: proc(info: *const reflect) -> kayan_column = {87 c: kayan_column = {};88 c.size = info*.size;89 c.align = info*.align;90 return c;91}92 93kayan_column_row: proc(c: *kayan_column, row: u64) -> *u8 = {94 return c*.data[row * c*.size]&;95}96 97kayan_column_grow: proc(arena: *memops_arena, c: *kayan_column, count: u64, capacity: u64) -> void = {98 data: ^u8 = cast(memops_arena_push_zero(arena, c*.size * capacity, c*.align), ^u8);99 if (count > 0) {100 memcpy(data, c*.data, c*.size * count);101 }102 c*.data = data;103}104 105kayan_column_copy_row: proc(c: *kayan_column, to: u64, from: u64) -> void = {106 memcpy(kayan_column_row(c, to), kayan_column_row(c, from), c*.size);107}108 109kayan_column_zero_row: proc(c: *kayan_column, row: u64) -> void = {110 memset(kayan_column_row(c, row), 0, c*.size);111}112 113kayan_column_slice: proc<T>(c: *kayan_column, count: u64) -> []T = {114 if (c*.size != sizeof(T)) {115 printfmt("kayan: a column of {}-byte rows read as a {}-byte type\n", c*.size, sizeof(T));116 exit(1);117 }118 raw: ^T = cast(c*.data, ^T);119 return raw[0..count];120}121 122// ---------------------------------------------------------------------------123// The table124// ---------------------------------------------------------------------------125 126// `columns[id]` is in use when the signature has bit `id`; the others stay127// zeroed. Count and capacity are the table's, shared by its columns and by128// `entities`, so they can never disagree.129//130// The edges: `add_edges[id]` is the table this one's entities move to when131// they gain component `id`, and `remove_edges[id]` when they lose it. Each is132// the table's index plus one, so 0, which a new table starts with, means "not133// known yet".134kayan_table: struct = {135 signature: u64;136 count: u64;137 capacity: u64;138 entities: ^kayan_entity;139 columns: [64]kayan_column;140 add_edges: [64]u32;141 remove_edges: [64]u32;142}143 144// The world: the registry, the slots and free list, and every table, with a145// map from signature to table index so finding one is a lookup, not a search.146// Tables are referred to by index, never by pointer: making a table can grow147// `tables` and move all of them, and an index survives that.148kayan_world: struct = {149 arena: *memops_arena;150 infos: [64]*const reflect;151 component_count: u32;152 tables: vec<kayan_table>;153 by_signature: hashmap<u64, u32>;154 slots: vec<kayan_slot>;155 free: vec<u32>;156 live: u64;157}158 159kayan_bit: proc(id: u32) -> u64 = {160 return 1'u64 shl cast(id, u64);161}162 163kayan_id_of: proc<T>() -> u32 = {164 return cast(component_of<T>(), u32);165}166 167kayan_mask_of: proc<T>() -> u64 = {168 return kayan_bit(kayan_id_of<T>());169}170 171kayan_register: proc(w: *kayan_world, id: u32, info: *const reflect) -> void = {172 if (id >= 64) {173 printfmt("kayan: component id {} is past the 64 a signature can hold\n", id);174 exit(1);175 }176 w*.infos[id] = info;177 if (id + 1 > w*.component_count) {178 w*.component_count = id + 1;179 }180}181 182kayan_table_has: proc(t: *kayan_table, id: u32) -> b32 = {183 return (t*.signature & kayan_bit(id)) != 0;184}185 186// A table for a signature: one empty column per component in it, sized from187// the registry. Every component in the signature must be registered.188kayan_table_init: proc(w: *kayan_world, signature: u64) -> kayan_table = {189 t: kayan_table = {};190 t.signature = signature;191 for (id: u32 = 0; id < 64; id += 1) {192 if ((signature & kayan_bit(id)) != 0) {193 if (id >= w*.component_count || w*.infos[id] == null) {194 printfmt("kayan: signature {} uses component {}, which was never registered\n", signature, id);195 exit(1);196 }197 t.columns[id] = kayan_column_init(w*.infos[id]);198 }199 }200 return t;201}202 203// Double the room in `entities` and in every column at once.204kayan_table_grow: proc(w: *kayan_world, t: *kayan_table) -> void = {205 capacity: u64 = t*.capacity == 0 ? 4 : t*.capacity * 2;206 entities: ^kayan_entity = cast(memops_arena_push_zero(w*.arena,207 sizeof(kayan_entity) * capacity, alignof(kayan_entity)), ^kayan_entity);208 if (t*.count > 0) {209 memcpy(entities, t*.entities, sizeof(kayan_entity) * t*.count);210 }211 t*.entities = entities;212 for (id: u32 = 0; id < w*.component_count; id += 1) {213 if (kayan_table_has(t, id)) {214 kayan_column_grow(w*.arena, t*.columns[id]&, t*.count, capacity);215 }216 }217 t*.capacity = capacity;218}219 220// A new row at the end, owned by `e`, with every component zeroed.221kayan_table_add_row: proc(w: *kayan_world, t: *kayan_table, e: kayan_entity) -> u64 = {222 if (t*.count == t*.capacity) {223 kayan_table_grow(w, t);224 }225 row: u64 = t*.count;226 t*.count += 1;227 t*.entities[row] = e;228 for (id: u32 = 0; id < w*.component_count; id += 1) {229 if (kayan_table_has(t, id)) {230 kayan_column_zero_row(t*.columns[id]&, row);231 }232 }233 return row;234}235 236// 1.4's swap-remove, for every column: the last row is copied over `row`, and237// the slot of the entity that owned the last row is told where it went.238kayan_table_remove_row: proc(w: *kayan_world, t: *kayan_table, row: u64) -> void = {239 last: u64 = t*.count - 1;240 if (row != last) {241 for (id: u32 = 0; id < w*.component_count; id += 1) {242 if (kayan_table_has(t, id)) {243 kayan_column_copy_row(t*.columns[id]&, row, last);244 }245 }246 moved: kayan_entity = t*.entities[last];247 t*.entities[row] = moved;248 vec<kayan_slot>at(w*.slots&, cast(moved.index, u64))*.row = cast(row, u32);249 }250 t*.count = last;251}252 253// The typed view of one of the table's columns: T's rows, all of them in use.254// Asking for a component the table does not have is a bug, so it stops.255kayan_slice: proc<T>(t: *kayan_table) -> []T = {256 id: u32 = kayan_id_of<T>();257 if (!kayan_table_has(t, id)) {258 printfmt("kayan: the table has no column for component {}\n", id);259 exit(1);260 }261 return kayan_column_slice<T>(t*.columns[id]&, t*.count);262}263 264// A pointer to a table is good until the next table is made.265kayan_table_at: proc(w: *kayan_world, index: u32) -> *kayan_table = {266 return vec<kayan_table>at(w*.tables&, cast(index, u64));267}268 269// The table for a signature: found if it exists, made if it does not. This is270// where archetypes come from.271kayan_table_for: proc(w: *kayan_world, signature: u64) -> u32 = {272 found: *u32 = hashmap<u64, u32>get(w*.by_signature&, signature);273 if (found != null) {274 return found*;275 }276 index: u32 = cast(w*.tables.length, u32);277 _ = vec<kayan_table>append(w*.arena, w*.tables&, kayan_table_init(w, signature));278 _ = hashmap<u64, u32>put(w*.arena, w*.by_signature&, signature, index);279 return index;280}281 282// ---------------------------------------------------------------------------283// The world284// ---------------------------------------------------------------------------285 286kayan_world_init: proc(w: *kayan_world, arena: *memops_arena) -> void = {287 w* = {};288 w*.arena = arena;289}290 291kayan_alive: proc(w: *kayan_world, e: kayan_entity) -> b32 = {292 if (cast(e.index, u64) >= w*.slots.length) {293 return false;294 }295 return vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.generation == e.generation;296}297 298// A new entity with the components in `signature`, all zeroed, in the table299// for that signature.300kayan_spawn: proc(w: *kayan_world, signature: u64) -> kayan_entity = {301 e: kayan_entity = {};302 if (!vec<u32>is_empty(w*.free&)) {303 e.index = vec<u32>pop(w*.free&);304 } else {305 e.index = cast(w*.slots.length, u32);306 _ = vec<kayan_slot>append(w*.arena, w*.slots&, { .generation = 1 });307 }308 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));309 e.generation = slot*.generation;310 // kayan_table_for can make a table and move the others, so the table's311 // pointer is taken after it.312 table_index: u32 = kayan_table_for(w, signature);313 t: *kayan_table = kayan_table_at(w, table_index);314 slot*.table = table_index;315 slot*.row = cast(kayan_table_add_row(w, t, e), u32);316 w*.live += 1;317 return e;318}319 320kayan_despawn: proc(w: *kayan_world, e: kayan_entity) -> kayan_status = {321 if (!kayan_alive(w, e)) {322 return kayan_status.dead;323 }324 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));325 kayan_table_remove_row(w, kayan_table_at(w, slot*.table), cast(slot*.row, u64));326 slot*.generation += 1;327 _ = vec<u32>append(w*.arena, w*.free&, e.index);328 w*.live -= 1;329 return kayan_status.ok;330}331 332// The components a live entity has: its table's signature. 0 for a dead one.333kayan_signature: proc(w: *kayan_world, e: kayan_entity) -> u64 = {334 if (!kayan_alive(w, e)) {335 return 0;336 }337 return kayan_table_at(w, vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.table)*.signature;338}339 340// ---------------------------------------------------------------------------341// Moving between tables342// ---------------------------------------------------------------------------343 344// The table an entity of table `from` moves to when it gains component `id`.345// The first time, it is found by signature, which can make the table; the346// answer is written on both tables, as from's add edge and the destination's347// remove edge, so neither direction is looked up again.348kayan_table_with: proc(w: *kayan_world, from: u32, id: u32) -> u32 = {349 edge: u32 = kayan_table_at(w, from)*.add_edges[id];350 if (edge != 0) {351 return edge - 1;352 }353 to: u32 = kayan_table_for(w, kayan_table_at(w, from)*.signature | kayan_bit(id));354 // kayan_table_for can move the tables, so both pointers are taken after it.355 kayan_table_at(w, from)*.add_edges[id] = to + 1;356 kayan_table_at(w, to)*.remove_edges[id] = from + 1;357 return to;358}359 360// The same for losing component `id`.361kayan_table_without: proc(w: *kayan_world, from: u32, id: u32) -> u32 = {362 edge: u32 = kayan_table_at(w, from)*.remove_edges[id];363 if (edge != 0) {364 return edge - 1;365 }366 to: u32 = kayan_table_for(w, kayan_table_at(w, from)*.signature & ~kayan_bit(id));367 kayan_table_at(w, from)*.remove_edges[id] = to + 1;368 kayan_table_at(w, to)*.add_edges[id] = from + 1;369 return to;370}371 372// Move a live entity to table `to_index`. Components both tables have are373// copied, byte for byte; components only the new table has start zeroed;374// components only the old table has are dropped with its old row.375kayan_move: proc(w: *kayan_world, e: kayan_entity, to_index: u32) -> void = {376 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));377 from_index: u32 = slot*.table;378 if (from_index == to_index) {379 return;380 }381 from: *kayan_table = kayan_table_at(w, from_index);382 to: *kayan_table = kayan_table_at(w, to_index);383 from_row: u64 = cast(slot*.row, u64);384 to_row: u64 = kayan_table_add_row(w, to, e);385 for (id: u32 = 0; id < w*.component_count; id += 1) {386 if (kayan_table_has(from, id) && kayan_table_has(to, id)) {387 memcpy(kayan_column_row(to*.columns[id]&, to_row),388 kayan_column_row(from*.columns[id]&, from_row), from*.columns[id].size);389 }390 }391 kayan_table_remove_row(w, from, from_row);392 slot*.table = to_index;393 slot*.row = cast(to_row, u32);394}395 396// ---------------------------------------------------------------------------397// The byte layer, for tools398// ---------------------------------------------------------------------------399 400// e's component `id` as its bytes: an empty slice if e is dead or has no such401// component. The typed layer is for game code, which knows its types; this is402// for tools that work from reflect records instead.403kayan_get_bytes: proc(w: *kayan_world, e: kayan_entity, id: u32) -> []u8 = {404 none: []u8 = {};405 if (!kayan_alive(w, e) || id >= w*.component_count) {406 return none;407 }408 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));409 t: *kayan_table = kayan_table_at(w, slot*.table);410 if (!kayan_table_has(t, id)) {411 return none;412 }413 c: *kayan_column = t*.columns[id]&;414 row: u64 = cast(slot*.row, u64);415 all: []u8 = c*.data[0..t*.count * c*.size];416 return all[row * c*.size..(row + 1) * c*.size];417}418 419// ---------------------------------------------------------------------------420// The typed layer421// ---------------------------------------------------------------------------422 423// e's T, or null if e is dead or has no T. Handle, slot, table, row, typed424// view: the same path as 1.4's get_position, for any component. The pointer is425// good until the next structural change, so use it now and keep the handle.426kayan_get: proc<T>(w: *kayan_world, e: kayan_entity) -> *T = {427 if (!kayan_alive(w, e)) {428 return null;429 }430 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));431 t: *kayan_table = kayan_table_at(w, slot*.table);432 if (!kayan_table_has(t, kayan_id_of<T>())) {433 return null;434 }435 rows: []T = kayan_slice<T>(t);436 return rows[cast(slot*.row, u64)]&;437}438 439kayan_has: proc<T>(w: *kayan_world, e: kayan_entity) -> b32 = {440 return (kayan_signature(w, e) & kayan_mask_of<T>()) != 0;441}442 443// Write e's T. It must already have one: giving an entity a component it does444// not have changes its table, which is 2.6.445kayan_set: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> kayan_status = {446 if (!kayan_alive(w, e)) {447 return kayan_status.dead;448 }449 p: *T = kayan_get<T>(w, e);450 if (p == null) {451 return kayan_status.missing;452 }453 p* = value;454 return kayan_status.ok;455}456 457// Give e a T. If it already has one, this is kayan_set; if not, it moves to458// the table with T and then sets it.459kayan_add: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> kayan_status = {460 if (!kayan_alive(w, e)) {461 return kayan_status.dead;462 }463 if (!kayan_has<T>(w, e)) {464 from: u32 = vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.table;465 kayan_move(w, e, kayan_table_with(w, from, kayan_id_of<T>()));466 }467 return kayan_set<T>(w, e, value);468}469 470// Take e's T away, moving it to the table without T.471kayan_remove: proc<T>(w: *kayan_world, e: kayan_entity) -> kayan_status = {472 if (!kayan_alive(w, e)) {473 return kayan_status.dead;474 }475 if (!kayan_has<T>(w, e)) {476 return kayan_status.missing;477 }478 from: u32 = vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.table;479 kayan_move(w, e, kayan_table_without(w, from, kayan_id_of<T>()));480 return kayan_status.ok;481}482 483// The invariant, over every table: each row belongs to a live entity whose484// slot points back at that table and row, and the rows of all tables add up to485// the live count.486kayan_verify: proc(w: *kayan_world) -> u64 = {487 problems: u64 = 0;488 rows: u64 = 0;489 for (ti: u64 = 0; ti < w*.tables.length; ti += 1) {490 t: *kayan_table = kayan_table_at(w, cast(ti, u32));491 for (row: u64 = 0; row < t*.count; row += 1) {492 e: kayan_entity = t*.entities[row];493 if (!kayan_alive(w, e)) {494 printfmt("kayan_verify: table {} row {} holds dead entity {}:{}\n", ti, row, e.index, e.generation);495 problems += 1;496 continue;497 }498 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));499 if (cast(slot*.table, u64) != ti || cast(slot*.row, u64) != row) {500 printfmt("kayan_verify: {}:{} is in table {} row {}, its slot says table {} row {}\n",501 e.index, e.generation, ti, row, slot*.table, slot*.row);502 problems += 1;503 }504 }505 rows += t*.count;506 }507 if (rows != w*.live) {508 printfmt("kayan_verify: {} rows in tables, {} live entities\n", rows, w*.live);509 problems += 1;510 }511 return problems;512}513 514kayan_print_signature: proc(w: *kayan_world, signature: u64) -> void = {515 printf("{");516 first: b32 = true;517 for (id: u32 = 0; id < w*.component_count; id += 1) {518 if ((signature & kayan_bit(id)) != 0) {519 printf(first ? "%s" : " %s", w*.infos[id]*.name);520 first = false;521 }522 }523 printf("}");524}525 526kayan_print_table: proc(w: *kayan_world, ti: u32) -> void = {527 t: *kayan_table = kayan_table_at(w, ti);528 printf(" table %u ", ti);529 kayan_print_signature(w, t*.signature);530 printf(":");531 for (row: u64 = 0; row < t*.count; row += 1) {532 printf(" %u:%u", t*.entities[row].index, t*.entities[row].generation);533 }534 printf("\n");535}536 537kayan_print_tables: proc(w: *kayan_world) -> void = {538 for (ti: u32 = 0; ti < cast(w*.tables.length, u32); ti += 1) {539 kayan_print_table(w, ti);540 }541}542 543// ---------------------------------------------------------------------------544// The inspector: reflect records, no component types545// ---------------------------------------------------------------------------546 547kayan_indent: proc(depth: u32) -> void = {548 for (i: u32 = 0; i < depth; i += 1) {549 printf(" ");550 }551}552 553// One value of a builtin type, read from its bytes by its type's name.554kayan_print_builtin: proc(type_name: ^const char, at: []const u8) -> void = {555 if (strcmp(type_name, "f32") == 0) {556 printf("%.2f", cast(cast(at[0]&, *const f32)*, f64));557 } else if (strcmp(type_name, "u32") == 0) {558 printf("%u", cast(at[0]&, *const u32)*);559 } else if (strcmp(type_name, "i32") == 0) {560 printf("%d", cast(at[0]&, *const i32)*);561 } else {562 printf("<%s>", type_name);563 }564}565 566// Every field of a struct whose bytes are `base`, nested structs indented567// under their field. A field's bytes are cut from `base` by its offset and568// size, both from the reflect record.569kayan_print_fields: proc(type: *const reflect, base: []const u8, depth: u32) -> void = {570 fields: ^const reflect_field = reflect_fields(type);571 for (i: u64 = 0; i < type*.count; i += 1) {572 f: *const reflect_field = fields[i]&;573 if (reflect_field_has_attr(f, "hidden")) {574 continue;575 }576 at: []const u8 = base[f*.offset..f*.offset + f*.size];577 kayan_indent(depth);578 printf("%s:", f*.name);579 info: *const reflect = f*.info;580 if (info != null && info*.kind == reflect_kind_struct) {581 printf("\n");582 kayan_print_fields(info, at, depth + 1);583 } else {584 printf(" ");585 kayan_print_builtin(f*.type, at);586 printf("\n");587 }588 }589}590 591kayan_inspect: proc(w: *kayan_world, e: kayan_entity) -> void = {592 if (!kayan_alive(w, e)) {593 printfmt("{}:{} (dead)\n", e.index, e.generation);594 return;595 }596 printf("%u:%u ", e.index, e.generation);597 signature: u64 = kayan_signature(w, e);598 kayan_print_signature(w, signature);599 printf("\n");600 for (id: u32 = 0; id < w*.component_count; id += 1) {601 if ((signature & kayan_bit(id)) != 0) {602 info: *const reflect = w*.infos[id];603 printf(" %s\n", info*.name);604 bytes: []u8 = kayan_get_bytes(w, e, id);605 kayan_print_fields(info, bytes, 2);606 }607 }608}609 610// Every table: rows in use out of room, and per column its row size and its611// bytes in use out of bytes allocated. The memory a table holds and does not612// use is the cost of doubling on growth.613kayan_print_layout: proc(w: *kayan_world) -> void = {614 for (ti: u32 = 0; ti < cast(w*.tables.length, u32); ti += 1) {615 t: *kayan_table = kayan_table_at(w, ti);616 printf("table %u ", ti);617 kayan_print_signature(w, t*.signature);618 printf(": %u rows, room for %u\n", cast(t*.count, u32), cast(t*.capacity, u32));619 for (id: u32 = 0; id < w*.component_count; id += 1) {620 if (kayan_table_has(t, id)) {621 size: u64 = t*.columns[id].size;622 printf(" %-9s %2u bytes a row, %4u of %4u bytes in use\n", w*.infos[id]*.name, cast(size, u32),623 cast(size * t*.count, u32), cast(size * t*.capacity, u32));624 }625 }626 }627}628 629// ---------------------------------------------------------------------------630// Queries631// ---------------------------------------------------------------------------632 633// The tables a system runs over: every table whose signature has all of `all`634// and none of `none`, kept by index. Tables are only ever added, at the end of635// w.tables, so the query remembers how many it has checked and, brought up to636// date, looks at the new ones only.637kayan_query: struct = {638 all: u64;639 none: u64;640 tables: vec<u32>;641 checked: u64;642}643 644kayan_query_init: proc(all: u64, none: u64) -> kayan_query = {645 q: kayan_query = {};646 q.all = all;647 q.none = none;648 return q;649}650 651kayan_matches: proc(signature: u64, all: u64, none: u64) -> b32 = {652 return (signature & all) == all && (signature & none) == 0;653}654 655// Check the tables made since the query last looked. With none new, this is656// one comparison, so call it before every walk.657kayan_query_update: proc(w: *kayan_world, q: *kayan_query) -> void = {658 for (ti: u64 = q*.checked; ti < w*.tables.length; ti += 1) {659 if (kayan_matches(kayan_table_at(w, cast(ti, u32))*.signature, q*.all, q*.none)) {660 _ = vec<u32>append(w*.arena, q*.tables&, cast(ti, u32));661 }662 }663 q*.checked = w*.tables.length;664}665 666// The query's i-th table. Like any table pointer, good until a table is made.667kayan_query_table: proc(w: *kayan_world, q: *kayan_query, i: u64) -> *kayan_table = {668 return kayan_table_at(w, vec<u32>at(q*.tables&, i)*);669}670 671// How many entities the query matches: the rows of its tables.672kayan_query_count: proc(w: *kayan_world, q: *kayan_query) -> u64 = {673 kayan_query_update(w, q);674 n: u64 = 0;675 for (i: u64 = 0; i < q*.tables.length; i += 1) {676 n += kayan_query_table(w, q, i)*.count;677 }678 return n;679}680 681kayan_print_query: proc(w: *kayan_world, name: ^const char, q: *kayan_query) -> void = {682 kayan_query_update(w, q);683 printf("%s: all ", name);684 kayan_print_signature(w, q*.all);685 printf(", none ");686 kayan_print_signature(w, q*.none);687 printfmt("; tables: {}, entities: {}\n", q*.tables.length, kayan_query_count(w, q));688 for (i: u64 = 0; i < q*.tables.length; i += 1) {689 kayan_print_table(w, vec<u32>at(q*.tables&, i)*);690 }691}692 693// ---------------------------------------------------------------------------694// Systems695// ---------------------------------------------------------------------------696 697// What a system gets for each table it runs on: the world, the table, and the698// frame's time step. One struct rather than three parameters, so that adding699// to it later changes no system's signature.700kayan_iter: struct = {701 world: *kayan_world;702 table: *kayan_table;703 dt: f32;704}705 706kayan_system_proc: alias = *proc(it: *kayan_iter) -> void;707 708// A system: a name, the query that finds its tables, and the proc run on each709// of those tables that has rows. `tables` and `rows` count what it has run710// over.711kayan_system: struct = {712 name: ^const char;713 query: kayan_query;714 run: kayan_system_proc;715 tables: u64;716 rows: u64;717}718 719kayan_system_init: proc(name: ^const char, all: u64, none: u64, run: kayan_system_proc) -> kayan_system = {720 s: kayan_system = {};721 s.name = name;722 s.query = kayan_query_init(all, none);723 s.run = run;724 return s;725}726 727// One frame: every system in order, each over every table it matches that has728// rows. A system sees everything the systems before it wrote this frame.729kayan_progress: proc(w: *kayan_world, systems: []kayan_system, dt: f32) -> void = {730 for (s: u64 = 0; s < systems.length; s += 1) {731 sys: *kayan_system = systems[s]&;732 kayan_query_update(w, sys*.query&);733 for (i: u64 = 0; i < sys*.query.tables.length; i += 1) {734 it: kayan_iter = {};735 it.world = w;736 it.table = kayan_query_table(w, sys*.query&, i);737 it.dt = dt;738 if (it.table*.count > 0) {739 sys*.tables += 1;740 sys*.rows += it.table*.count;741 sys*.run(it&);742 }743 }744 }745}746 747kayan_print_schedule: proc(w: *kayan_world, systems: []kayan_system) -> void = {748 for (s: u64 = 0; s < systems.length; s += 1) {749 kayan_print_query(w, systems[s].name, systems[s].query&);750 }751}752 753// ---------------------------------------------------------------------------754// The program755// ---------------------------------------------------------------------------756 757// Kernels: plain loops over plain slices, with components for elements.758// noalias on what each one writes, as in lessons 1 to 7.759integrate: proc(ps: noalias []position, vs: []const velocity, dt: f32) -> void = {760 for (i: u64 = 0; i < ps.length; i += 1) {761 ps[i].at.x += vs[i].per_second.x * dt;762 ps[i].at.y += vs[i].per_second.y * dt;763 }764}765 766burn: proc(hs: noalias []health, fires: []const burning, dt: f32) -> void = {767 for (i: u64 = 0; i < hs.length; i += 1) {768 hs[i].points -= fires[i].per_second * dt;769 }770}771 772age: proc(ls: noalias []lifetime, dt: f32) -> void = {773 for (i: u64 = 0; i < ls.length; i += 1) {774 ls[i].seconds -= dt;775 }776}777 778// Systems: the table's columns, handed to a kernel.779move_system: proc(it: *kayan_iter) -> void = {780 integrate(kayan_slice<position>(it*.table), kayan_slice<velocity>(it*.table), it*.dt);781}782 783burn_system: proc(it: *kayan_iter) -> void = {784 burn(kayan_slice<health>(it*.table), kayan_slice<burning>(it*.table), it*.dt);785}786 787age_system: proc(it: *kayan_iter) -> void = {788 age(kayan_slice<lifetime>(it*.table), it*.dt);789}790 791spawn_turret: proc(w: *kayan_world, x: f32, y: f32) -> kayan_entity = {792 e: kayan_entity = kayan_spawn(w, kayan_mask_of<position>() | kayan_mask_of<turret>());793 _ = kayan_set<position>(w, e, { .at = { .x = x, .y = y } });794 _ = kayan_set<turret>(w, e, { .cooldown = 0.5, .timer = 0.0 });795 return e;796}797 798spawn_drone: proc(w: *kayan_world, x: f32, y: f32) -> kayan_entity = {799 e: kayan_entity = kayan_spawn(w, kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<health>());800 _ = kayan_set<position>(w, e, { .at = { .x = x, .y = y } });801 _ = kayan_set<velocity>(w, e, { .per_second = { .x = -1.0, .y = 0.0 } });802 _ = kayan_set<health>(w, e, { .points = 100.0, .max = 100.0 });803 return e;804}805 806spawn_bullet: proc(w: *kayan_world, x: f32, y: f32, seconds: f32) -> kayan_entity = {807 e: kayan_entity = kayan_spawn(w, kayan_mask_of<position>() | kayan_mask_of<velocity>()808 | kayan_mask_of<damage>() | kayan_mask_of<lifetime>());809 _ = kayan_set<position>(w, e, { .at = { .x = x, .y = y } });810 _ = kayan_set<velocity>(w, e, { .per_second = { .x = 8.0, .y = 0.0 } });811 _ = kayan_set<damage>(w, e, { .amount = 25.0 });812 _ = kayan_set<lifetime>(w, e, { .seconds = seconds });813 return e;814}815 816// Every entity, with the components this program cares about.817print_state: proc(w: *kayan_world) -> void = {818 for (ti: u32 = 0; ti < cast(w*.tables.length, u32); ti += 1) {819 t: *kayan_table = kayan_table_at(w, ti);820 for (row: u64 = 0; row < t*.count; row += 1) {821 e: kayan_entity = t*.entities[row];822 p: *position = kayan_get<position>(w, e);823 printf(" %u:%u at (%5.2f, %5.2f)", e.index, e.generation, cast(p*.at.x, f64), cast(p*.at.y, f64));824 h: *health = kayan_get<health>(w, e);825 if (h != null) {826 printf(" health %.0f", cast(h*.points, f64));827 }828 if (kayan_has<burning>(w, e)) {829 printf(", burning");830 }831 l: *lifetime = kayan_get<lifetime>(w, e);832 if (l != null) {833 printf(" lifetime %5.2f", cast(l*.seconds, f64));834 }835 printf("\n");836 }837 }838}839 840main: proc() -> i32 = {841 arena: memops_arena = {};842 memops_arena_initialize(arena&);843 w: kayan_world = {};844 kayan_world_init(w&, arena&);845 kayan_register(w&, kayan_id_of<position>(), position<>&);846 kayan_register(w&, kayan_id_of<velocity>(), velocity<>&);847 kayan_register(w&, kayan_id_of<health>(), health<>&);848 kayan_register(w&, kayan_id_of<damage>(), damage<>&);849 kayan_register(w&, kayan_id_of<lifetime>(), lifetime<>&);850 kayan_register(w&, kayan_id_of<turret>(), turret<>&);851 kayan_register(w&, kayan_id_of<burning>(), burning<>&);852 853 _ = spawn_turret(w&, 0.0, 0.0);854 _ = spawn_drone(w&, 20.0, 4.0);855 hot: kayan_entity = spawn_drone(w&, 24.0, 0.0);856 _ = spawn_drone(w&, 28.0, -4.0);857 _ = kayan_add<burning>(w&, hot, { .per_second = 8.0 });858 _ = spawn_bullet(w&, 1.0, 0.0, 0.5);859 _ = spawn_bullet(w&, 1.0, 4.0, 1.0);860 _ = spawn_bullet(w&, 1.0, -4.0, 1.5);861 862 // The schedule. The order is the data flow: a system that reads what863 // another writes runs after it.864 systems: [3]kayan_system = {};865 systems[0] = kayan_system_init("burn", kayan_mask_of<health>() | kayan_mask_of<burning>(), 0, burn_system);866 systems[1] = kayan_system_init("age", kayan_mask_of<lifetime>(), 0, age_system);867 systems[2] = kayan_system_init("move", kayan_mask_of<position>() | kayan_mask_of<velocity>(), 0, move_system);868 printfmt("the schedule, run in this order every frame:\n");869 kayan_print_schedule(w&, systems);870 printfmt("\nat the start:\n");871 print_state(w&);872 873 dt: f32 = 0.25;874 for (frame: u32 = 1; frame <= 8; frame += 1) {875 kayan_progress(w&, systems, dt);876 if (frame % 4 == 0) {877 printf("\nafter frame %u, %.2f seconds:\n", frame, cast(cast(frame, f32) * dt, f64));878 print_state(w&);879 }880 }881 882 printfmt("\n8 frames, one call per table per frame:\n");883 for (s: u64 = 0; s < 3; s += 1) {884 printf(" %-4s called %2u times, over %2u rows\n", systems[s].name, cast(systems[s].tables, u32), cast(systems[s].rows, u32));885 }886 887 // A system may change values, never structure, so nothing has removed the888 // bullets whose time is up.889 expired: u64 = 0;890 for (i: u64 = 0; i < systems[1].query.tables.length; i += 1) {891 ls: []lifetime = kayan_slice<lifetime>(kayan_query_table(w&, systems[1].query&, i));892 for (row: u64 = 0; row < ls.length; row += 1) {893 if (ls[row].seconds <= 0.0) {894 expired += 1;895 }896 }897 }898 printfmt("{} bullets are out of time and still in the world\n", expired);899 printfmt("kayan_verify: {} problems\n", kayan_verify(w&));900 return 0;901}What systems prints. tools/check.py fails the chapter if a single byte differs.
the schedule, run in this order every frame:
burn: all {health burning}, none {}; tables: 1, entities: 1
table 2 {position velocity health burning}: 2:1
age: all {lifetime}, none {}; tables: 1, entities: 3
table 3 {position velocity damage lifetime}: 4:1 5:1 6:1
move: all {position velocity}, none {}; tables: 3, entities: 6
table 1 {position velocity health}: 1:1 3:1
table 2 {position velocity health burning}: 2:1
table 3 {position velocity damage lifetime}: 4:1 5:1 6:1
at the start:
0:1 at ( 0.00, 0.00)
1:1 at (20.00, 4.00) health 100
3:1 at (28.00, -4.00) health 100
2:1 at (24.00, 0.00) health 100, burning
4:1 at ( 1.00, 0.00) lifetime 0.50
5:1 at ( 1.00, 4.00) lifetime 1.00
6:1 at ( 1.00, -4.00) lifetime 1.50
after frame 4, 1.00 seconds:
0:1 at ( 0.00, 0.00)
1:1 at (19.00, 4.00) health 100
3:1 at (27.00, -4.00) health 100
2:1 at (23.00, 0.00) health 92, burning
4:1 at ( 9.00, 0.00) lifetime -0.50
5:1 at ( 9.00, 4.00) lifetime 0.00
6:1 at ( 9.00, -4.00) lifetime 0.50
after frame 8, 2.00 seconds:
0:1 at ( 0.00, 0.00)
1:1 at (18.00, 4.00) health 100
3:1 at (26.00, -4.00) health 100
2:1 at (22.00, 0.00) health 84, burning
4:1 at (17.00, 0.00) lifetime -1.50
5:1 at (17.00, 4.00) lifetime -1.00
6:1 at (17.00, -4.00) lifetime -0.50
8 frames, one call per table per frame:
burn called 8 times, over 8 rows
age called 8 times, over 24 rows
move called 24 times, over 48 rows
3 bullets are out of time and still in the world
kayan_verify: 0 problems
Build an ECS · part 3, systems · 3.3
Commands
ecs/src/3.systems/3.3.commands/commands.rin
In the dojo: 5. a command buffer, 7. an arena, each on its own, to watch and to practise.
3.2's systems change values, never structure, so bullets whose time ran out stay in the world. The obvious fix is to despawn them from inside the system. It's wrong, and seeing exactly why is most of this chapter.
Step 1: the bug
expire_now_system: proc(it: *kayan_iter) -> void = {
t: *kayan_table = it*.table;
ls: []lifetime = kayan_slice<lifetime>(t);
for (row: u64 = 0; row < t*.count; row += 1) {
if (ls[row].seconds <= 0.0) {
_ = kayan_despawn(it*.world, t*.entities[row]);
}
}
}
Four bullets, three of them out of time:
1. an expire system that despawns as it goes
lifetimes: 0:1 0.00 1:1 1.00 2:1 0.00 3:1 0.00
after one frame:
lifetimes: 3:1 0.00 1:1 1.00
kayan_verify: 0 problems. The world is fine; the loop missed one.
Row by row:
row 0: 0:1 is out of time. Despawn: the last row, 3:1, moves into row 0.
row 1: 1:1 has time left.
row 2: 2:1 is out of time. Despawn: it's the last row, so nothing moves.
row 3: the table has 2 rows now. Stop.
3:1 moved into row 0, which the loop had already passed, so it was never looked at. The world is perfectly consistent (kayan_verify agrees). The loop is what's wrong: a despawn swap-removes from the table the loop is walking.
Walking backwards fixes this case (lesson 2 does it), because the row that moves in has already been looked at. But it doesn't fix the general problem:
- Adding or removing a component moves the entity to another table. If the query matches that table too and visits it later, the entity is processed twice.
- If that table is new, making it can grow
w.tablesand move every table. Thetthe loop is holding then points at the old copy, which the arena still owns, so nothing crashes and every write after that is lost. This is 2.4's bug, from inside a loop. - One system's despawn can land in a table another system is halfway through, if a system despawns entities it found through a second query.
Step 2: record now, apply later
So a system doesn't change structure. It records a command saying what should change, and the world applies every command after the frame's last system:
during the frame values change; tables, rows and pointers do not
after the frame spawns, despawns, adds and removes, in order
kayan_command_kind: enum = {
spawn,
despawn,
add,
remove,
}
kayan_command: struct = {
kind: kayan_command_kind;
entity: kayan_entity;
spawn: u32;
id: u32;
signature: u64;
value: []u8;
}
A command names the entity, and for an add or remove the component id. An add carries its value as bytes. The bytes, and the list of commands, live in a frame arena the world owns:
frame: memops_arena;
commands: vec<kayan_command>;
After the commands are applied, memops_arena_clear drops all of them at once, whatever they were.
Step 3: recording
The typed procs a system calls copy the value into the frame arena and append a command:
kayan_defer_value: proc<T>(w: *kayan_world, value: T) -> []u8 = {
data: ^u8 = cast(memops_arena_push(w*.frame&, sizeof(T), alignof(T)), ^u8);
memcpy(data, value&, sizeof(T));
return data[0..sizeof(T)];
}
kayan_defer_add: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> void = {
_ = kayan_defer(w, { .kind = kayan_command_kind.add, .entity = e, .id = kayan_id_of<T>(),
.value = kayan_defer_value<T>(w, value) });
}
The fixed expire system is the broken one with one word changed:
expire_system: proc(it: *kayan_iter) -> void = {
ls: []lifetime = kayan_slice<lifetime>(it*.table);
es: []kayan_entity = kayan_entities(it*.table);
for (row: u64 = 0; row < es.length; row += 1) {
if (ls[row].seconds <= 0.0) {
kayan_defer_despawn(it*.world, es[row]);
}
}
}
Now the table doesn't change while the loop runs, so the loop can take the entities as a slice up front, like any other column.
Step 4: applying, by id
Commands hold bytes and ids, not types. So applying an add needs 2.6's add by id:
kayan_add_id: proc(w: *kayan_world, e: kayan_entity, id: u32, value: []const u8) -> kayan_status = {
It's the same move-then-write as before, with the write a memcpy of the value's bytes into the row. Once that exists, the typed kayan_add<T> from 2.6 shrinks to a one-line wrapper. That's the pattern from the start of the course: byte fiddling written once, a type-safe generic layer on top.
kayan_add: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> kayan_status = {
bytes: ^const u8 = cast(value&, ^const u8);
return kayan_add_id(w, e, kayan_id_of<T>(), bytes[0..sizeof(T)]);
}
kayan_flush walks the list in order and applies each command. It's called at the end of kayan_progress:
kayan_flush(w);
}
Step 5: commands that arrive too late
By the time a command is applied, its entity may be gone. The usual case is two systems both asking to despawn the same thing. Applying in order handles that for free: the first despawn works, and the second finds a dead handle. Part 1's generations make that check one comparison:
if (status == kayan_status.ok) {
w*.applied += 1;
} else {
w*.skipped += 1;
}
Step 6: spawning from a system
A turret firing is a spawn in the middle of a frame. The new entity doesn't exist until the commands are applied, so there's no handle to give its components to. kayan_defer_spawn returns a pending spawn instead: the spawn command's place in the list. Components are given to that:
shot: kayan_pending = kayan_defer_spawn(it*.world, bullet_signature());
kayan_pending_set<position>(it*.world, shot, ps[row]);
kayan_pending_set<velocity>(it*.world, shot, { .per_second = { .x = 8.0, .y = 0.0 } });
kayan_pending_set<damage>(it*.world, shot, { .amount = 25.0 });
kayan_pending_set<lifetime>(it*.world, shot, { .seconds = 2.0 });
When the flush reaches the spawn, it makes the entity and writes the handle into the command. The adds after it look the handle up there. The bullet is spawned with its full signature, so the adds land in the table it's already in: no moves. And it doesn't move until the next frame, because no system ran over it this one.
Step 7: a system with two queries
A bullet that reaches a target damages it. The hit system walks the bullets (its own query) and, for each, the targets (a second query). It reaches the second query through a new field, ctx, which kayan passes through untouched:
targets: *kayan_query = cast(it*.ctx, *kayan_query);
The damage is a value, so it's written now. The bullet's despawn is structure, so it's recorded. Walking a second query in the middle of a frame is safe because of commands: no table is made or moved until the frame is over.
Run it
2. an expire system that records despawns
lifetimes: 0:1 0.00 1:1 1.00 2:1 0.00 3:1 0.00
despawn 0:1
despawn 2:1
despawn 3:1
after one frame:
lifetimes: 1:1 1.00
All three. Then the turret, with two targets 0.25 apart in its line of fire and a third out of it:
3. a turret, two targets in its line of fire, one out of it
frame 1
spawn {position velocity damage lifetime} -> 4:1
frame 2
frame 3
spawn {position velocity damage lifetime} -> 5:1
frame 4
despawn 4:1
despawn 4:1: skipped, dead
frame 5
spawn {position velocity damage lifetime} -> 4:2
...
health: 1:1 50, 2:1 50, 3:1 100
commands applied: 17, skipped: 2
In frame 4, bullet 4:1 is close enough to both targets. Both lose 25, and both hits record a despawn. The first applies; the second finds 4:1 already dead and is skipped. In frame 5 the next bullet gets number 4 back with generation 2: Part 1, working.
17 commands, for three bullets and two despawns: each spawn is five, the spawn and its four components.
Try
- In
expire_now_system, loop tols.lengthinstead oft*.count. Now all three get despawned. Work out why. (Hint: what's in the rows past the end of the table?) Is that loop correct? - Write a system that adds
burningdirectly (not deferred) to every target it walks. Count how many times it visits each entity. - Flush after each system instead of after the frame. What does that change about what later systems see? (flecs calls these sync points.)
1// 3.3 Commands2//3// 3.2's systems change values, never structure, so the bullets whose time ran4// out are still in the world. Despawning them from inside the system is the5// obvious fix, and it is wrong: the system is walking the very table a6// despawn swap-removes from, so the last row moves into a row the walk has7// already passed, and is never looked at. Adding or removing a component is8// worse: the entity moves to another table the query may visit later, and a9// new table can move every table, the one being walked included.10//11// So a system does not change structure. It records the change, a command,12// and the world applies every command after the frame's last system:13//14// during the frame values change; tables, rows and pointers do not15// after the frame spawns, despawns, adds and removes, in order16//17// The second half of this file is a command buffer: the record, the procs18// systems call to fill it, and the flush that empties it.19 20import "std/print.rin"21import "std/memops.rin"22import "std/vec.rin"23import "std/hashmap.rin"24 25vec2: struct = {26 x: f32;27 y: f32;28}29 30position: struct = { at: vec2; }31velocity: struct = { per_second: vec2; }32health: struct = { points: f32; max: f32; }33damage: struct = { amount: f32; }34lifetime: struct = { seconds: f32; }35// A field tagged "hidden" is bookkeeping the inspector does not show.36turret: struct = { cooldown: f32; timer: f32 @ "hidden"; }37burning: struct = { per_second: f32; }38 39component: enum = {40 position,41 velocity,42 health,43 damage,44 lifetime,45 turret,46 burning,47}48 49component_of: proc<position>() -> component = { return component.position; }50component_of: proc<velocity>() -> component = { return component.velocity; }51component_of: proc<health>() -> component = { return component.health; }52component_of: proc<damage>() -> component = { return component.damage; }53component_of: proc<lifetime>() -> component = { return component.lifetime; }54component_of: proc<turret>() -> component = { return component.turret; }55component_of: proc<burning>() -> component = { return component.burning; }56 57// ---------------------------------------------------------------------------58// Entities (Part 1)59// ---------------------------------------------------------------------------60 61kayan_entity: struct = {62 index: u32;63 generation: u32;64}65 66// `missing`: the entity is alive but has no such component.67kayan_status: enum[must_use] = {68 ok,69 dead,70 missing,71}72 73// Where an entity lives: which table, and which row of it.74kayan_slot: struct = {75 generation: u32;76 table: u32;77 row: u32;78}79 80// ---------------------------------------------------------------------------81// Columns (2.2)82// ---------------------------------------------------------------------------83 84kayan_column: struct = {85 data: ^u8;86 size: u64;87 align: u64;88}89 90kayan_column_init: proc(info: *const reflect) -> kayan_column = {91 c: kayan_column = {};92 c.size = info*.size;93 c.align = info*.align;94 return c;95}96 97kayan_column_row: proc(c: *kayan_column, row: u64) -> *u8 = {98 return c*.data[row * c*.size]&;99}100 101kayan_column_grow: proc(arena: *memops_arena, c: *kayan_column, count: u64, capacity: u64) -> void = {102 data: ^u8 = cast(memops_arena_push_zero(arena, c*.size * capacity, c*.align), ^u8);103 if (count > 0) {104 memcpy(data, c*.data, c*.size * count);105 }106 c*.data = data;107}108 109kayan_column_copy_row: proc(c: *kayan_column, to: u64, from: u64) -> void = {110 memcpy(kayan_column_row(c, to), kayan_column_row(c, from), c*.size);111}112 113kayan_column_zero_row: proc(c: *kayan_column, row: u64) -> void = {114 memset(kayan_column_row(c, row), 0, c*.size);115}116 117kayan_column_slice: proc<T>(c: *kayan_column, count: u64) -> []T = {118 if (c*.size != sizeof(T)) {119 printfmt("kayan: a column of {}-byte rows read as a {}-byte type\n", c*.size, sizeof(T));120 exit(1);121 }122 raw: ^T = cast(c*.data, ^T);123 return raw[0..count];124}125 126// ---------------------------------------------------------------------------127// The table128// ---------------------------------------------------------------------------129 130// `columns[id]` is in use when the signature has bit `id`; the others stay131// zeroed. Count and capacity are the table's, shared by its columns and by132// `entities`, so they can never disagree.133//134// The edges: `add_edges[id]` is the table this one's entities move to when135// they gain component `id`, and `remove_edges[id]` when they lose it. Each is136// the table's index plus one, so 0, which a new table starts with, means "not137// known yet".138kayan_table: struct = {139 signature: u64;140 count: u64;141 capacity: u64;142 entities: ^kayan_entity;143 columns: [64]kayan_column;144 add_edges: [64]u32;145 remove_edges: [64]u32;146}147 148// A structural change recorded during a frame and applied after it (3.3).149// `entity` is who it applies to; a spawn's is filled in when it is applied. An150// add to an entity spawned in the same frame has no entity to name yet, so it151// names its spawn instead: `spawn` is that command's index plus one.152kayan_command_kind: enum = {153 spawn,154 despawn,155 add,156 remove,157}158 159kayan_command: struct = {160 kind: kayan_command_kind;161 entity: kayan_entity;162 spawn: u32;163 id: u32;164 signature: u64;165 value: []u8;166}167 168// The world: the registry, the slots and free list, and every table, with a169// map from signature to table index so finding one is a lookup, not a search.170// Tables are referred to by index, never by pointer: making a table can grow171// `tables` and move all of them, and an index survives that.172kayan_world: struct = {173 arena: *memops_arena;174 infos: [64]*const reflect;175 component_count: u32;176 tables: vec<kayan_table>;177 by_signature: hashmap<u64, u32>;178 slots: vec<kayan_slot>;179 free: vec<u32>;180 live: u64;181 // The frame's commands, and their values, live in `frame`, which is182 // cleared once they are applied. `trace` prints each one as it is.183 frame: memops_arena;184 commands: vec<kayan_command>;185 applied: u64;186 skipped: u64;187 trace: b32;188}189 190kayan_bit: proc(id: u32) -> u64 = {191 return 1'u64 shl cast(id, u64);192}193 194kayan_id_of: proc<T>() -> u32 = {195 return cast(component_of<T>(), u32);196}197 198kayan_mask_of: proc<T>() -> u64 = {199 return kayan_bit(kayan_id_of<T>());200}201 202kayan_register: proc(w: *kayan_world, id: u32, info: *const reflect) -> void = {203 if (id >= 64) {204 printfmt("kayan: component id {} is past the 64 a signature can hold\n", id);205 exit(1);206 }207 w*.infos[id] = info;208 if (id + 1 > w*.component_count) {209 w*.component_count = id + 1;210 }211}212 213kayan_table_has: proc(t: *kayan_table, id: u32) -> b32 = {214 return (t*.signature & kayan_bit(id)) != 0;215}216 217// A table for a signature: one empty column per component in it, sized from218// the registry. Every component in the signature must be registered.219kayan_table_init: proc(w: *kayan_world, signature: u64) -> kayan_table = {220 t: kayan_table = {};221 t.signature = signature;222 for (id: u32 = 0; id < 64; id += 1) {223 if ((signature & kayan_bit(id)) != 0) {224 if (id >= w*.component_count || w*.infos[id] == null) {225 printfmt("kayan: signature {} uses component {}, which was never registered\n", signature, id);226 exit(1);227 }228 t.columns[id] = kayan_column_init(w*.infos[id]);229 }230 }231 return t;232}233 234// Double the room in `entities` and in every column at once.235kayan_table_grow: proc(w: *kayan_world, t: *kayan_table) -> void = {236 capacity: u64 = t*.capacity == 0 ? 4 : t*.capacity * 2;237 entities: ^kayan_entity = cast(memops_arena_push_zero(w*.arena,238 sizeof(kayan_entity) * capacity, alignof(kayan_entity)), ^kayan_entity);239 if (t*.count > 0) {240 memcpy(entities, t*.entities, sizeof(kayan_entity) * t*.count);241 }242 t*.entities = entities;243 for (id: u32 = 0; id < w*.component_count; id += 1) {244 if (kayan_table_has(t, id)) {245 kayan_column_grow(w*.arena, t*.columns[id]&, t*.count, capacity);246 }247 }248 t*.capacity = capacity;249}250 251// A new row at the end, owned by `e`, with every component zeroed.252kayan_table_add_row: proc(w: *kayan_world, t: *kayan_table, e: kayan_entity) -> u64 = {253 if (t*.count == t*.capacity) {254 kayan_table_grow(w, t);255 }256 row: u64 = t*.count;257 t*.count += 1;258 t*.entities[row] = e;259 for (id: u32 = 0; id < w*.component_count; id += 1) {260 if (kayan_table_has(t, id)) {261 kayan_column_zero_row(t*.columns[id]&, row);262 }263 }264 return row;265}266 267// 1.4's swap-remove, for every column: the last row is copied over `row`, and268// the slot of the entity that owned the last row is told where it went.269kayan_table_remove_row: proc(w: *kayan_world, t: *kayan_table, row: u64) -> void = {270 last: u64 = t*.count - 1;271 if (row != last) {272 for (id: u32 = 0; id < w*.component_count; id += 1) {273 if (kayan_table_has(t, id)) {274 kayan_column_copy_row(t*.columns[id]&, row, last);275 }276 }277 moved: kayan_entity = t*.entities[last];278 t*.entities[row] = moved;279 vec<kayan_slot>at(w*.slots&, cast(moved.index, u64))*.row = cast(row, u32);280 }281 t*.count = last;282}283 284// The typed view of one of the table's columns: T's rows, all of them in use.285// Asking for a component the table does not have is a bug, so it stops.286kayan_slice: proc<T>(t: *kayan_table) -> []T = {287 id: u32 = kayan_id_of<T>();288 if (!kayan_table_has(t, id)) {289 printfmt("kayan: the table has no column for component {}\n", id);290 exit(1);291 }292 return kayan_column_slice<T>(t*.columns[id]&, t*.count);293}294 295// Who owns each of the table's rows.296kayan_entities: proc(t: *kayan_table) -> []kayan_entity = {297 return t*.entities[0..t*.count];298}299 300// A pointer to a table is good until the next table is made.301kayan_table_at: proc(w: *kayan_world, index: u32) -> *kayan_table = {302 return vec<kayan_table>at(w*.tables&, cast(index, u64));303}304 305// The table for a signature: found if it exists, made if it does not. This is306// where archetypes come from.307kayan_table_for: proc(w: *kayan_world, signature: u64) -> u32 = {308 found: *u32 = hashmap<u64, u32>get(w*.by_signature&, signature);309 if (found != null) {310 return found*;311 }312 index: u32 = cast(w*.tables.length, u32);313 _ = vec<kayan_table>append(w*.arena, w*.tables&, kayan_table_init(w, signature));314 _ = hashmap<u64, u32>put(w*.arena, w*.by_signature&, signature, index);315 return index;316}317 318// ---------------------------------------------------------------------------319// The world320// ---------------------------------------------------------------------------321 322kayan_world_init: proc(w: *kayan_world, arena: *memops_arena) -> void = {323 w* = {};324 w*.arena = arena;325 // A frame's commands are small: 256 MB of address space is plenty.326 memops_arena_initialize(w*.frame&, 0x10000000);327}328 329kayan_alive: proc(w: *kayan_world, e: kayan_entity) -> b32 = {330 if (cast(e.index, u64) >= w*.slots.length) {331 return false;332 }333 return vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.generation == e.generation;334}335 336// A new entity with the components in `signature`, all zeroed, in the table337// for that signature.338kayan_spawn: proc(w: *kayan_world, signature: u64) -> kayan_entity = {339 e: kayan_entity = {};340 if (!vec<u32>is_empty(w*.free&)) {341 e.index = vec<u32>pop(w*.free&);342 } else {343 e.index = cast(w*.slots.length, u32);344 _ = vec<kayan_slot>append(w*.arena, w*.slots&, { .generation = 1 });345 }346 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));347 e.generation = slot*.generation;348 // kayan_table_for can make a table and move the others, so the table's349 // pointer is taken after it.350 table_index: u32 = kayan_table_for(w, signature);351 t: *kayan_table = kayan_table_at(w, table_index);352 slot*.table = table_index;353 slot*.row = cast(kayan_table_add_row(w, t, e), u32);354 w*.live += 1;355 return e;356}357 358kayan_despawn: proc(w: *kayan_world, e: kayan_entity) -> kayan_status = {359 if (!kayan_alive(w, e)) {360 return kayan_status.dead;361 }362 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));363 kayan_table_remove_row(w, kayan_table_at(w, slot*.table), cast(slot*.row, u64));364 slot*.generation += 1;365 _ = vec<u32>append(w*.arena, w*.free&, e.index);366 w*.live -= 1;367 return kayan_status.ok;368}369 370// The components a live entity has: its table's signature. 0 for a dead one.371kayan_signature: proc(w: *kayan_world, e: kayan_entity) -> u64 = {372 if (!kayan_alive(w, e)) {373 return 0;374 }375 return kayan_table_at(w, vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.table)*.signature;376}377 378// ---------------------------------------------------------------------------379// Moving between tables380// ---------------------------------------------------------------------------381 382// The table an entity of table `from` moves to when it gains component `id`.383// The first time, it is found by signature, which can make the table; the384// answer is written on both tables, as from's add edge and the destination's385// remove edge, so neither direction is looked up again.386kayan_table_with: proc(w: *kayan_world, from: u32, id: u32) -> u32 = {387 edge: u32 = kayan_table_at(w, from)*.add_edges[id];388 if (edge != 0) {389 return edge - 1;390 }391 to: u32 = kayan_table_for(w, kayan_table_at(w, from)*.signature | kayan_bit(id));392 // kayan_table_for can move the tables, so both pointers are taken after it.393 kayan_table_at(w, from)*.add_edges[id] = to + 1;394 kayan_table_at(w, to)*.remove_edges[id] = from + 1;395 return to;396}397 398// The same for losing component `id`.399kayan_table_without: proc(w: *kayan_world, from: u32, id: u32) -> u32 = {400 edge: u32 = kayan_table_at(w, from)*.remove_edges[id];401 if (edge != 0) {402 return edge - 1;403 }404 to: u32 = kayan_table_for(w, kayan_table_at(w, from)*.signature & ~kayan_bit(id));405 kayan_table_at(w, from)*.remove_edges[id] = to + 1;406 kayan_table_at(w, to)*.add_edges[id] = from + 1;407 return to;408}409 410// Move a live entity to table `to_index`. Components both tables have are411// copied, byte for byte; components only the new table has start zeroed;412// components only the old table has are dropped with its old row.413kayan_move: proc(w: *kayan_world, e: kayan_entity, to_index: u32) -> void = {414 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));415 from_index: u32 = slot*.table;416 if (from_index == to_index) {417 return;418 }419 from: *kayan_table = kayan_table_at(w, from_index);420 to: *kayan_table = kayan_table_at(w, to_index);421 from_row: u64 = cast(slot*.row, u64);422 to_row: u64 = kayan_table_add_row(w, to, e);423 for (id: u32 = 0; id < w*.component_count; id += 1) {424 if (kayan_table_has(from, id) && kayan_table_has(to, id)) {425 memcpy(kayan_column_row(to*.columns[id]&, to_row),426 kayan_column_row(from*.columns[id]&, from_row), from*.columns[id].size);427 }428 }429 kayan_table_remove_row(w, from, from_row);430 slot*.table = to_index;431 slot*.row = cast(to_row, u32);432}433 434// ---------------------------------------------------------------------------435// The byte layer, for tools436// ---------------------------------------------------------------------------437 438// e's component `id` as its bytes: an empty slice if e is dead or has no such439// component. The typed layer is for game code, which knows its types; this is440// for tools that work from reflect records instead.441kayan_get_bytes: proc(w: *kayan_world, e: kayan_entity, id: u32) -> []u8 = {442 none: []u8 = {};443 if (!kayan_alive(w, e) || id >= w*.component_count) {444 return none;445 }446 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));447 t: *kayan_table = kayan_table_at(w, slot*.table);448 if (!kayan_table_has(t, id)) {449 return none;450 }451 c: *kayan_column = t*.columns[id]&;452 row: u64 = cast(slot*.row, u64);453 all: []u8 = c*.data[0..t*.count * c*.size];454 return all[row * c*.size..(row + 1) * c*.size];455}456 457// Give e component `id`, with `value` for its bytes: 2.6's add, by id instead458// of by type, because a command holds its value as bytes.459kayan_add_id: proc(w: *kayan_world, e: kayan_entity, id: u32, value: []const u8) -> kayan_status = {460 if (!kayan_alive(w, e)) {461 return kayan_status.dead;462 }463 from: u32 = vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.table;464 if (!kayan_table_has(kayan_table_at(w, from), id)) {465 kayan_move(w, e, kayan_table_with(w, from, id));466 }467 bytes: []u8 = kayan_get_bytes(w, e, id);468 if (value.length != bytes.length) {469 printfmt("kayan: a {}-byte value for a component of {} bytes\n", value.length, bytes.length);470 exit(1);471 }472 memcpy(bytes[0]&, value[0]&, bytes.length);473 return kayan_status.ok;474}475 476kayan_remove_id: proc(w: *kayan_world, e: kayan_entity, id: u32) -> kayan_status = {477 if (!kayan_alive(w, e)) {478 return kayan_status.dead;479 }480 from: u32 = vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.table;481 if (!kayan_table_has(kayan_table_at(w, from), id)) {482 return kayan_status.missing;483 }484 kayan_move(w, e, kayan_table_without(w, from, id));485 return kayan_status.ok;486}487 488// ---------------------------------------------------------------------------489// The typed layer490// ---------------------------------------------------------------------------491 492// e's T, or null if e is dead or has no T. Handle, slot, table, row, typed493// view: the same path as 1.4's get_position, for any component. The pointer is494// good until the next structural change, so use it now and keep the handle.495kayan_get: proc<T>(w: *kayan_world, e: kayan_entity) -> *T = {496 if (!kayan_alive(w, e)) {497 return null;498 }499 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));500 t: *kayan_table = kayan_table_at(w, slot*.table);501 if (!kayan_table_has(t, kayan_id_of<T>())) {502 return null;503 }504 rows: []T = kayan_slice<T>(t);505 return rows[cast(slot*.row, u64)]&;506}507 508kayan_has: proc<T>(w: *kayan_world, e: kayan_entity) -> b32 = {509 return (kayan_signature(w, e) & kayan_mask_of<T>()) != 0;510}511 512// Write e's T. It must already have one: giving an entity a component it does513// not have changes its table, which is 2.6.514kayan_set: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> kayan_status = {515 if (!kayan_alive(w, e)) {516 return kayan_status.dead;517 }518 p: *T = kayan_get<T>(w, e);519 if (p == null) {520 return kayan_status.missing;521 }522 p* = value;523 return kayan_status.ok;524}525 526// Give e a T, or take its T away: the byte procs, with the type turned into an527// id and the value into its bytes.528kayan_add: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> kayan_status = {529 bytes: ^const u8 = cast(value&, ^const u8);530 return kayan_add_id(w, e, kayan_id_of<T>(), bytes[0..sizeof(T)]);531}532 533kayan_remove: proc<T>(w: *kayan_world, e: kayan_entity) -> kayan_status = {534 return kayan_remove_id(w, e, kayan_id_of<T>());535}536 537// The invariant, over every table: each row belongs to a live entity whose538// slot points back at that table and row, and the rows of all tables add up to539// the live count.540kayan_verify: proc(w: *kayan_world) -> u64 = {541 problems: u64 = 0;542 rows: u64 = 0;543 for (ti: u64 = 0; ti < w*.tables.length; ti += 1) {544 t: *kayan_table = kayan_table_at(w, cast(ti, u32));545 for (row: u64 = 0; row < t*.count; row += 1) {546 e: kayan_entity = t*.entities[row];547 if (!kayan_alive(w, e)) {548 printfmt("kayan_verify: table {} row {} holds dead entity {}:{}\n", ti, row, e.index, e.generation);549 problems += 1;550 continue;551 }552 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));553 if (cast(slot*.table, u64) != ti || cast(slot*.row, u64) != row) {554 printfmt("kayan_verify: {}:{} is in table {} row {}, its slot says table {} row {}\n",555 e.index, e.generation, ti, row, slot*.table, slot*.row);556 problems += 1;557 }558 }559 rows += t*.count;560 }561 if (rows != w*.live) {562 printfmt("kayan_verify: {} rows in tables, {} live entities\n", rows, w*.live);563 problems += 1;564 }565 return problems;566}567 568kayan_print_signature: proc(w: *kayan_world, signature: u64) -> void = {569 printf("{");570 first: b32 = true;571 for (id: u32 = 0; id < w*.component_count; id += 1) {572 if ((signature & kayan_bit(id)) != 0) {573 printf(first ? "%s" : " %s", w*.infos[id]*.name);574 first = false;575 }576 }577 printf("}");578}579 580kayan_print_table: proc(w: *kayan_world, ti: u32) -> void = {581 t: *kayan_table = kayan_table_at(w, ti);582 printf(" table %u ", ti);583 kayan_print_signature(w, t*.signature);584 printf(":");585 for (row: u64 = 0; row < t*.count; row += 1) {586 printf(" %u:%u", t*.entities[row].index, t*.entities[row].generation);587 }588 printf("\n");589}590 591kayan_print_tables: proc(w: *kayan_world) -> void = {592 for (ti: u32 = 0; ti < cast(w*.tables.length, u32); ti += 1) {593 kayan_print_table(w, ti);594 }595}596 597// ---------------------------------------------------------------------------598// The inspector: reflect records, no component types599// ---------------------------------------------------------------------------600 601kayan_indent: proc(depth: u32) -> void = {602 for (i: u32 = 0; i < depth; i += 1) {603 printf(" ");604 }605}606 607// One value of a builtin type, read from its bytes by its type's name.608kayan_print_builtin: proc(type_name: ^const char, at: []const u8) -> void = {609 if (strcmp(type_name, "f32") == 0) {610 printf("%.2f", cast(cast(at[0]&, *const f32)*, f64));611 } else if (strcmp(type_name, "u32") == 0) {612 printf("%u", cast(at[0]&, *const u32)*);613 } else if (strcmp(type_name, "i32") == 0) {614 printf("%d", cast(at[0]&, *const i32)*);615 } else {616 printf("<%s>", type_name);617 }618}619 620// Every field of a struct whose bytes are `base`, nested structs indented621// under their field. A field's bytes are cut from `base` by its offset and622// size, both from the reflect record.623kayan_print_fields: proc(type: *const reflect, base: []const u8, depth: u32) -> void = {624 fields: ^const reflect_field = reflect_fields(type);625 for (i: u64 = 0; i < type*.count; i += 1) {626 f: *const reflect_field = fields[i]&;627 if (reflect_field_has_attr(f, "hidden")) {628 continue;629 }630 at: []const u8 = base[f*.offset..f*.offset + f*.size];631 kayan_indent(depth);632 printf("%s:", f*.name);633 info: *const reflect = f*.info;634 if (info != null && info*.kind == reflect_kind_struct) {635 printf("\n");636 kayan_print_fields(info, at, depth + 1);637 } else {638 printf(" ");639 kayan_print_builtin(f*.type, at);640 printf("\n");641 }642 }643}644 645kayan_inspect: proc(w: *kayan_world, e: kayan_entity) -> void = {646 if (!kayan_alive(w, e)) {647 printfmt("{}:{} (dead)\n", e.index, e.generation);648 return;649 }650 printf("%u:%u ", e.index, e.generation);651 signature: u64 = kayan_signature(w, e);652 kayan_print_signature(w, signature);653 printf("\n");654 for (id: u32 = 0; id < w*.component_count; id += 1) {655 if ((signature & kayan_bit(id)) != 0) {656 info: *const reflect = w*.infos[id];657 printf(" %s\n", info*.name);658 bytes: []u8 = kayan_get_bytes(w, e, id);659 kayan_print_fields(info, bytes, 2);660 }661 }662}663 664// Every table: rows in use out of room, and per column its row size and its665// bytes in use out of bytes allocated. The memory a table holds and does not666// use is the cost of doubling on growth.667kayan_print_layout: proc(w: *kayan_world) -> void = {668 for (ti: u32 = 0; ti < cast(w*.tables.length, u32); ti += 1) {669 t: *kayan_table = kayan_table_at(w, ti);670 printf("table %u ", ti);671 kayan_print_signature(w, t*.signature);672 printf(": %u rows, room for %u\n", cast(t*.count, u32), cast(t*.capacity, u32));673 for (id: u32 = 0; id < w*.component_count; id += 1) {674 if (kayan_table_has(t, id)) {675 size: u64 = t*.columns[id].size;676 printf(" %-9s %2u bytes a row, %4u of %4u bytes in use\n", w*.infos[id]*.name, cast(size, u32),677 cast(size * t*.count, u32), cast(size * t*.capacity, u32));678 }679 }680 }681}682 683// ---------------------------------------------------------------------------684// Queries685// ---------------------------------------------------------------------------686 687// The tables a system runs over: every table whose signature has all of `all`688// and none of `none`, kept by index. Tables are only ever added, at the end of689// w.tables, so the query remembers how many it has checked and, brought up to690// date, looks at the new ones only.691kayan_query: struct = {692 all: u64;693 none: u64;694 tables: vec<u32>;695 checked: u64;696}697 698kayan_query_init: proc(all: u64, none: u64) -> kayan_query = {699 q: kayan_query = {};700 q.all = all;701 q.none = none;702 return q;703}704 705kayan_matches: proc(signature: u64, all: u64, none: u64) -> b32 = {706 return (signature & all) == all && (signature & none) == 0;707}708 709// Check the tables made since the query last looked. With none new, this is710// one comparison, so call it before every walk.711kayan_query_update: proc(w: *kayan_world, q: *kayan_query) -> void = {712 for (ti: u64 = q*.checked; ti < w*.tables.length; ti += 1) {713 if (kayan_matches(kayan_table_at(w, cast(ti, u32))*.signature, q*.all, q*.none)) {714 _ = vec<u32>append(w*.arena, q*.tables&, cast(ti, u32));715 }716 }717 q*.checked = w*.tables.length;718}719 720// The query's i-th table. Like any table pointer, good until a table is made.721kayan_query_table: proc(w: *kayan_world, q: *kayan_query, i: u64) -> *kayan_table = {722 return kayan_table_at(w, vec<u32>at(q*.tables&, i)*);723}724 725// How many entities the query matches: the rows of its tables.726kayan_query_count: proc(w: *kayan_world, q: *kayan_query) -> u64 = {727 kayan_query_update(w, q);728 n: u64 = 0;729 for (i: u64 = 0; i < q*.tables.length; i += 1) {730 n += kayan_query_table(w, q, i)*.count;731 }732 return n;733}734 735kayan_print_query: proc(w: *kayan_world, name: ^const char, q: *kayan_query) -> void = {736 kayan_query_update(w, q);737 printf("%s: all ", name);738 kayan_print_signature(w, q*.all);739 printf(", none ");740 kayan_print_signature(w, q*.none);741 printfmt("; tables: {}, entities: {}\n", q*.tables.length, kayan_query_count(w, q));742 for (i: u64 = 0; i < q*.tables.length; i += 1) {743 kayan_print_table(w, vec<u32>at(q*.tables&, i)*);744 }745}746 747// ---------------------------------------------------------------------------748// Commands749// ---------------------------------------------------------------------------750 751// A spawn that has been recorded and not applied. There is no entity yet, so752// this is the spawn command's place in the list; give it components with753// kayan_pending_set.754kayan_pending: struct = {755 command: u32;756}757 758kayan_defer: proc(w: *kayan_world, c: kayan_command) -> u32 = {759 index: u32 = cast(w*.commands.length, u32);760 _ = vec<kayan_command>append(w*.frame&, w*.commands&, c);761 return index;762}763 764// A copy of `value`, as bytes, in the frame arena.765kayan_defer_value: proc<T>(w: *kayan_world, value: T) -> []u8 = {766 data: ^u8 = cast(memops_arena_push(w*.frame&, sizeof(T), alignof(T)), ^u8);767 memcpy(data, value&, sizeof(T));768 return data[0..sizeof(T)];769}770 771kayan_defer_spawn: proc(w: *kayan_world, signature: u64) -> kayan_pending = {772 p: kayan_pending = {};773 p.command = kayan_defer(w, { .kind = kayan_command_kind.spawn, .signature = signature });774 return p;775}776 777kayan_pending_set: proc<T>(w: *kayan_world, p: kayan_pending, value: T) -> void = {778 _ = kayan_defer(w, { .kind = kayan_command_kind.add, .spawn = p.command + 1, .id = kayan_id_of<T>(),779 .value = kayan_defer_value<T>(w, value) });780}781 782kayan_defer_despawn: proc(w: *kayan_world, e: kayan_entity) -> void = {783 _ = kayan_defer(w, { .kind = kayan_command_kind.despawn, .entity = e });784}785 786kayan_defer_add: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> void = {787 _ = kayan_defer(w, { .kind = kayan_command_kind.add, .entity = e, .id = kayan_id_of<T>(),788 .value = kayan_defer_value<T>(w, value) });789}790 791kayan_defer_remove: proc<T>(w: *kayan_world, e: kayan_entity) -> void = {792 _ = kayan_defer(w, { .kind = kayan_command_kind.remove, .entity = e, .id = kayan_id_of<T>() });793}794 795// One applied command. The components given to a new entity are part of its796// spawn line, so they are not printed on their own.797kayan_print_command: proc(w: *kayan_world, c: *kayan_command, status: kayan_status) -> void = {798 if (c*.spawn != 0 && status == kayan_status.ok) {799 return;800 }801 printfmt(" {} ", c*.kind);802 if (c*.kind == kayan_command_kind.spawn) {803 kayan_print_signature(w, c*.signature);804 printf(" -> ");805 } else if (c*.kind != kayan_command_kind.despawn) {806 printf("%s, ", w*.infos[c*.id]*.name);807 }808 printf("%u:%u", c*.entity.index, c*.entity.generation);809 if (status != kayan_status.ok) {810 printfmt(": skipped, {}", status);811 }812 printf("\n");813}814 815// Apply the frame's commands in the order they were recorded, then let them816// all go at once. A command for an entity that died after it was recorded817// (despawned by an earlier command, most likely) does nothing, and is counted.818kayan_flush: proc(w: *kayan_world) -> void = {819 for (i: u64 = 0; i < w*.commands.length; i += 1) {820 c: *kayan_command = vec<kayan_command>at(w*.commands&, i);821 if (c*.spawn != 0) {822 c*.entity = vec<kayan_command>at(w*.commands&, cast(c*.spawn - 1, u64))*.entity;823 }824 status: kayan_status = kayan_status.ok;825 if (c*.kind == kayan_command_kind.spawn) {826 c*.entity = kayan_spawn(w, c*.signature);827 } else if (c*.kind == kayan_command_kind.despawn) {828 status = kayan_despawn(w, c*.entity);829 } else if (c*.kind == kayan_command_kind.add) {830 status = kayan_add_id(w, c*.entity, c*.id, c*.value);831 } else {832 status = kayan_remove_id(w, c*.entity, c*.id);833 }834 if (status == kayan_status.ok) {835 w*.applied += 1;836 } else {837 w*.skipped += 1;838 }839 if (w*.trace) {840 kayan_print_command(w, c, status);841 }842 }843 w*.commands = {};844 memops_arena_clear(w*.frame&);845}846 847// ---------------------------------------------------------------------------848// Systems849// ---------------------------------------------------------------------------850 851// What a system gets for each table it runs on: the world, the table, and the852// frame's time step. One struct rather than three parameters, so that adding853// to it later changes no system's signature.854kayan_iter: struct = {855 world: *kayan_world;856 table: *kayan_table;857 dt: f32;858 ctx: *void;859}860 861kayan_system_proc: alias = *proc(it: *kayan_iter) -> void;862 863// A system: a name, the query that finds its tables, and the proc run on each864// of those tables that has rows. `tables` and `rows` count what it has run865// over. `ctx` is handed to the proc as is: whatever the system needs beyond866// its own tables, such as another query, or the game's state.867kayan_system: struct = {868 name: ^const char;869 query: kayan_query;870 run: kayan_system_proc;871 ctx: *void;872 tables: u64;873 rows: u64;874}875 876kayan_system_init: proc(name: ^const char, all: u64, none: u64, run: kayan_system_proc, ctx: *void) -> kayan_system = {877 s: kayan_system = {};878 s.name = name;879 s.query = kayan_query_init(all, none);880 s.run = run;881 s.ctx = ctx;882 return s;883}884 885// One frame: every system in order, each over every table it matches that has886// rows, then the commands they recorded. A system sees every value the887// systems before it wrote this frame, and no structural change until the888// next.889kayan_progress: proc(w: *kayan_world, systems: []kayan_system, dt: f32) -> void = {890 for (s: u64 = 0; s < systems.length; s += 1) {891 sys: *kayan_system = systems[s]&;892 kayan_query_update(w, sys*.query&);893 for (i: u64 = 0; i < sys*.query.tables.length; i += 1) {894 it: kayan_iter = {};895 it.world = w;896 it.table = kayan_query_table(w, sys*.query&, i);897 it.dt = dt;898 it.ctx = sys*.ctx;899 if (it.table*.count > 0) {900 sys*.tables += 1;901 sys*.rows += it.table*.count;902 sys*.run(it&);903 }904 }905 }906 kayan_flush(w);907}908 909kayan_print_schedule: proc(w: *kayan_world, systems: []kayan_system) -> void = {910 for (s: u64 = 0; s < systems.length; s += 1) {911 kayan_print_query(w, systems[s].name, systems[s].query&);912 }913}914 915// ---------------------------------------------------------------------------916// The program917// ---------------------------------------------------------------------------918 919bullet_signature: proc() -> u64 = {920 return kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<damage>() | kayan_mask_of<lifetime>();921}922 923spawn_bullet: proc(w: *kayan_world, x: f32, seconds: f32) -> kayan_entity = {924 e: kayan_entity = kayan_spawn(w, bullet_signature());925 _ = kayan_set<position>(w, e, { .at = { .x = x, .y = 0.0 } });926 _ = kayan_set<velocity>(w, e, { .per_second = { .x = 8.0, .y = 0.0 } });927 _ = kayan_set<damage>(w, e, { .amount = 25.0 });928 _ = kayan_set<lifetime>(w, e, { .seconds = seconds });929 return e;930}931 932spawn_target: proc(w: *kayan_world, x: f32, y: f32) -> kayan_entity = {933 e: kayan_entity = kayan_spawn(w, kayan_mask_of<position>() | kayan_mask_of<health>());934 _ = kayan_set<position>(w, e, { .at = { .x = x, .y = y } });935 _ = kayan_set<health>(w, e, { .points = 100.0, .max = 100.0 });936 return e;937}938 939// The wrong way: despawn as the loop goes. Row `row` is swap-removed, the last940// row moves into it, and the loop moves on to row + 1.941expire_now_system: proc(it: *kayan_iter) -> void = {942 t: *kayan_table = it*.table;943 ls: []lifetime = kayan_slice<lifetime>(t);944 for (row: u64 = 0; row < t*.count; row += 1) {945 if (ls[row].seconds <= 0.0) {946 _ = kayan_despawn(it*.world, t*.entities[row]);947 }948 }949}950 951// The right way: record the despawn. The table does not change until the952// frame is over, so every row is looked at once.953expire_system: proc(it: *kayan_iter) -> void = {954 ls: []lifetime = kayan_slice<lifetime>(it*.table);955 es: []kayan_entity = kayan_entities(it*.table);956 for (row: u64 = 0; row < es.length; row += 1) {957 if (ls[row].seconds <= 0.0) {958 kayan_defer_despawn(it*.world, es[row]);959 }960 }961}962 963age_system: proc(it: *kayan_iter) -> void = {964 ls: []lifetime = kayan_slice<lifetime>(it*.table);965 for (row: u64 = 0; row < ls.length; row += 1) {966 ls[row].seconds -= it*.dt;967 }968}969 970move_system: proc(it: *kayan_iter) -> void = {971 ps: []position = kayan_slice<position>(it*.table);972 vs: []velocity = kayan_slice<velocity>(it*.table);973 for (row: u64 = 0; row < ps.length; row += 1) {974 ps[row].at.x += vs[row].per_second.x * it*.dt;975 ps[row].at.y += vs[row].per_second.y * it*.dt;976 }977}978 979// A turret whose timer is up spawns a bullet where it stands. The bullet does980// not exist until the frame ends, so it first moves next frame.981turret_system: proc(it: *kayan_iter) -> void = {982 ps: []position = kayan_slice<position>(it*.table);983 ts: []turret = kayan_slice<turret>(it*.table);984 for (row: u64 = 0; row < ts.length; row += 1) {985 if (ts[row].timer <= 0.0) {986 shot: kayan_pending = kayan_defer_spawn(it*.world, bullet_signature());987 kayan_pending_set<position>(it*.world, shot, ps[row]);988 kayan_pending_set<velocity>(it*.world, shot, { .per_second = { .x = 8.0, .y = 0.0 } });989 kayan_pending_set<damage>(it*.world, shot, { .amount = 25.0 });990 kayan_pending_set<lifetime>(it*.world, shot, { .seconds = 2.0 });991 ts[row].timer += ts[row].cooldown;992 }993 ts[row].timer -= it*.dt;994 }995}996 997// A bullet within half a unit of a target hits it. The damage is a value, so998// it is written now; the bullet's despawn is structure, so it is recorded. The999// targets are another query, which the system gets through its ctx. During a1000// frame no table is made or moved, so walking a second query is safe.1001hit_system: proc(it: *kayan_iter) -> void = {1002 w: *kayan_world = it*.world;1003 targets: *kayan_query = cast(it*.ctx, *kayan_query);1004 ps: []position = kayan_slice<position>(it*.table);1005 ds: []damage = kayan_slice<damage>(it*.table);1006 es: []kayan_entity = kayan_entities(it*.table);1007 kayan_query_update(w, targets);1008 for (b: u64 = 0; b < es.length; b += 1) {1009 for (i: u64 = 0; i < targets*.tables.length; i += 1) {1010 t: *kayan_table = kayan_query_table(w, targets, i);1011 tps: []position = kayan_slice<position>(t);1012 ths: []health = kayan_slice<health>(t);1013 for (row: u64 = 0; row < tps.length; row += 1) {1014 dx: f32 = tps[row].at.x - ps[b].at.x;1015 dy: f32 = tps[row].at.y - ps[b].at.y;1016 if (dx * dx + dy * dy < 0.25) {1017 ths[row].points -= ds[b].amount;1018 kayan_defer_despawn(w, es[b]);1019 }1020 }1021 }1022 }1023}1024 1025make_world: proc(w: *kayan_world, arena: *memops_arena) -> void = {1026 kayan_world_init(w, arena);1027 kayan_register(w, kayan_id_of<position>(), position<>&);1028 kayan_register(w, kayan_id_of<velocity>(), velocity<>&);1029 kayan_register(w, kayan_id_of<health>(), health<>&);1030 kayan_register(w, kayan_id_of<damage>(), damage<>&);1031 kayan_register(w, kayan_id_of<lifetime>(), lifetime<>&);1032 kayan_register(w, kayan_id_of<turret>(), turret<>&);1033 kayan_register(w, kayan_id_of<burning>(), burning<>&);1034}1035 1036// Four bullets, three of them out of time.1037four_bullets: proc(w: *kayan_world) -> void = {1038 _ = spawn_bullet(w, 1.0, 0.0);1039 _ = spawn_bullet(w, 2.0, 1.0);1040 _ = spawn_bullet(w, 3.0, 0.0);1041 _ = spawn_bullet(w, 4.0, 0.0);1042}1043 1044print_lifetimes: proc(w: *kayan_world) -> void = {1045 t: *kayan_table = kayan_table_at(w, kayan_table_for(w, bullet_signature()));1046 ls: []lifetime = kayan_slice<lifetime>(t);1047 printf(" lifetimes:");1048 for (row: u64 = 0; row < t*.count; row += 1) {1049 printf(" %u:%u %.2f", t*.entities[row].index, t*.entities[row].generation, cast(ls[row].seconds, f64));1050 }1051 printf("\n");1052}1053 1054main: proc() -> i32 = {1055 arena: memops_arena = {};1056 memops_arena_initialize(arena&);1057 lifetime_only: u64 = kayan_mask_of<lifetime>();1058 1059 printfmt("1. an expire system that despawns as it goes\n");1060 a: kayan_world = {};1061 make_world(a&, arena&);1062 four_bullets(a&);1063 print_lifetimes(a&);1064 wrong: [1]kayan_system = {};1065 wrong[0] = kayan_system_init("expire", lifetime_only, 0, expire_now_system, null);1066 kayan_progress(a&, wrong, 0.25);1067 printfmt("after one frame:\n");1068 print_lifetimes(a&);1069 printfmt("kayan_verify: {} problems. The world is fine; the loop missed one.\n", kayan_verify(a&));1070 1071 printfmt("\n2. an expire system that records despawns\n");1072 b: kayan_world = {};1073 make_world(b&, arena&);1074 b.trace = true;1075 four_bullets(b&);1076 print_lifetimes(b&);1077 right: [1]kayan_system = {};1078 right[0] = kayan_system_init("expire", lifetime_only, 0, expire_system, null);1079 kayan_progress(b&, right, 0.25);1080 printfmt("after one frame:\n");1081 print_lifetimes(b&);1082 1083 printfmt("\n3. a turret, two targets in its line of fire, one out of it\n");1084 c: kayan_world = {};1085 make_world(c&, arena&);1086 c.trace = true;1087 gun: kayan_entity = kayan_spawn(c&, kayan_mask_of<position>() | kayan_mask_of<turret>());1088 _ = kayan_set<turret>(c&, gun, { .cooldown = 0.5, .timer = 0.0 });1089 near: kayan_entity = spawn_target(c&, 4.0, 0.0);1090 far: kayan_entity = spawn_target(c&, 4.25, 0.0);1091 aside: kayan_entity = spawn_target(c&, 4.0, 3.0);1092 targets: kayan_query = kayan_query_init(kayan_mask_of<position>() | kayan_mask_of<health>(), 0);1093 systems: [5]kayan_system = {};1094 systems[0] = kayan_system_init("turret", kayan_mask_of<position>() | kayan_mask_of<turret>(), 0, turret_system, null);1095 systems[1] = kayan_system_init("age", lifetime_only, 0, age_system, null);1096 systems[2] = kayan_system_init("expire", lifetime_only, 0, expire_system, null);1097 systems[3] = kayan_system_init("hit", kayan_mask_of<position>() | kayan_mask_of<damage>(), 0, hit_system, targets&);1098 systems[4] = kayan_system_init("move", kayan_mask_of<position>() | kayan_mask_of<velocity>(), 0, move_system, null);1099 for (frame: u32 = 1; frame <= 6; frame += 1) {1100 printfmt("frame {}\n", frame);1101 kayan_progress(c&, systems, 0.25);1102 }1103 printf("health: %u:%u %.0f, %u:%u %.0f, %u:%u %.0f\n",1104 near.index, near.generation, cast(kayan_get<health>(c&, near)*.points, f64),1105 far.index, far.generation, cast(kayan_get<health>(c&, far)*.points, f64),1106 aside.index, aside.generation, cast(kayan_get<health>(c&, aside)*.points, f64));1107 printfmt("commands applied: {}, skipped: {}\n", c.applied, c.skipped);1108 kayan_print_tables(c&);1109 printfmt("kayan_verify: {} problems\n", kayan_verify(c&));1110 return 0;1111}What commands prints. tools/check.py fails the chapter if a single byte differs.
1. an expire system that despawns as it goes
lifetimes: 0:1 0.00 1:1 1.00 2:1 0.00 3:1 0.00
after one frame:
lifetimes: 3:1 0.00 1:1 1.00
kayan_verify: 0 problems. The world is fine; the loop missed one.
2. an expire system that records despawns
lifetimes: 0:1 0.00 1:1 1.00 2:1 0.00 3:1 0.00
despawn 0:1
despawn 2:1
despawn 3:1
after one frame:
lifetimes: 1:1 1.00
3. a turret, two targets in its line of fire, one out of it
frame 1
spawn {position velocity damage lifetime} -> 4:1
frame 2
frame 3
spawn {position velocity damage lifetime} -> 5:1
frame 4
despawn 4:1
despawn 4:1: skipped, dead
frame 5
spawn {position velocity damage lifetime} -> 4:2
frame 6
despawn 5:1
despawn 5:1: skipped, dead
health: 1:1 50, 2:1 50, 3:1 100
commands applied: 17, skipped: 2
table 0 {position turret}: 0:1
table 1 {position health}: 1:1 2:1 3:1
table 2 {position velocity damage lifetime}: 4:2
kayan_verify: 0 problems
Build an ECS · part 3, systems · 3.4
A game
ecs/src/3.systems/3.4.game/game.rin
In the dojo: this chapter puts the building blocks together; the dojo has each one on its own, to watch and to practise.
Everything together. Three lanes, a turret at the left end of each, and drones coming from the right in waves:
- a turret fires down its lane while there's a drone in it;
- a bullet that reaches a drone damages it;
- a drone below half health catches fire;
- a drone with no health left is destroyed;
- a drone that gets to the left end has reached the base.
The top of the file is kayan, exactly as 3.3 left it. The game is the bottom: components, kernels, systems, a schedule and a level script. Every rule of the game is a system over a query. Every structural change a system makes is a command.
The pieces
The components are the same seven from 2.1. A drone is a signature:
drone_signature: proc() -> u64 = {
return kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<health>();
}
A burning drone is that plus burning, which puts it in another table. There is no drone type anywhere in the program. A drone is whatever lives in a table with these columns.
What the systems share (the drones query, the clock, the score) is one struct, handed to the systems that need it through ctx:
game: struct = {
drones: kayan_query;
time: f32;
fired: u32;
hits: u32;
fires: u32;
kills: u32;
breaches: u32;
came: u32;
}
The schedule
systems: [9]kayan_system = {};
systems[0] = kayan_system_init("turret", shooting, 0, turret_system, g&);
systems[1] = kayan_system_init("age", kayan_mask_of<lifetime>(), 0, age_system, null);
systems[2] = kayan_system_init("expire", kayan_mask_of<lifetime>(), 0, expire_system, null);
systems[3] = kayan_system_init("move", movers, 0, move_system, null);
systems[4] = kayan_system_init("hit", bullets, 0, hit_system, g&);
systems[5] = kayan_system_init("ignite", kayan_mask_of<health>(), kayan_mask_of<burning>(), ignite_system, g&);
systems[6] = kayan_system_init("burn", kayan_mask_of<health>() | kayan_mask_of<burning>(), 0, burn_system, null);
systems[7] = kayan_system_init("die", kayan_mask_of<health>(), 0, die_system, g&);
systems[8] = kayan_system_init("breach", kayan_mask_of<position>() | kayan_mask_of<health>(), 0, breach_system, g&);
Read it as the frame, top to bottom. Each line's position is a data-flow decision (3.2):
movebeforehit: bullets hit where they are, not where they were.hitbeforeignite,burnanddie: a drone takes damage, catches fire and is destroyed in the same frame its health drops.diebeforebreach: only a drone that survived the frame can reach the base.breachalso checkshealth > 0itself, because both run in the same frame anddie's despawn hasn't been applied yet.ignitehasnone {burning}: a drone catches fire once. The add is a command, so next frame it's in the burning table and the query no longer matches it.
The rules
Each one is a few lines. The ones that only change values call kernels:
burn_system: proc(it: *kayan_iter) -> void = {
burn(kayan_slice<health>(it*.table), kayan_slice<burning>(it*.table), it*.dt);
}
The ones that change structure record commands:
if (hs[row].points < 0.5 * hs[row].max && hs[row].points > 0.0) {
kayan_defer_add<burning>(it*.world, es[row], { .per_second = 10.0 });
g*.fires += 1;
}
The turret is the most involved. It asks the drones query whether anything is ahead of it in its lane, then spawns a bullet with a pending spawn (3.3):
if (ts[row].timer <= 0.0 && drone_ahead(it*.world, g, ps[row].at.x, ps[row].at.y)) {
hit is 3.3's hit system, with one extra thought about speed. A bullet moves a unit a frame and a fast drone a quarter, so they close by at most 1.25 a frame. A hit window of ±0.75 is 1.5 wide, so no bullet can pass through a drone between two frames:
if (dx * dx < 0.5625 && dy * dy < 0.25) {
The level
The waves aren't a system. They're the level script, and it runs between frames, in the game loop, where spawning directly is safe:
for (frame: u32 = 0; frame < frames; frame += 1) {
g.time = cast(frame, f32) * dt;
if (frame % 32 == 0 && frame / 32 < 8) {
send_wave(w&, g&, frame / 32);
}
kayan_progress(w&, systems, dt);
problems += kayan_verify(w&);
kayan_verify runs after every frame: 512 checks of Part 1's invariant over the whole world.
Drawing by table
The picture of the lanes walks a query of everything with a position. What an entity is drawn as comes from its table's signature, decided once per table, not per entity:
glyph_of: proc(signature: u64) -> i32 = {
if ((signature & kayan_mask_of<turret>()) != 0) {
return 'T';
}
if ((signature & kayan_mask_of<burning>()) != 0) {
return '*';
}
if ((signature & kayan_mask_of<health>()) != 0) {
return 'd';
}
return '-';
}
In an OOP engine this would be a virtual draw() per object. Here it's a property of a table, and the inner loop is a plain loop over positions.
Exact on purpose
Every speed and time step is a power of two: dt is 1/16, drones move 2 or 4 units a second, bullets 16, fire burns 10 a second (0.625 a frame). So every position, health and timer is exactly representable, and no rounding ever happens. That's why a Debug build, a Release build, gcc, clang and a CPU that fuses multiply-adds all print the same game. Real games give that up for nicer numbers. Rounding differences usually don't matter, but they do the day you want a replay or a lockstep network game.
Run it
0.00s wave 1: 3:1, 2 units a second
2.00s wave 2: 7:1 8:1, 2 units a second
4.00s 12 entities in 4 tables
2 T......-...........-........d....
1 T..........-...........-*........
0 T......-...........-........d....
4.00s wave 3: 9:2 10:2 14:1, 2 units a second
4.00s 3:1 destroyed
...
28.00s 6 entities in 4 tables
2 T.*..............................
1 T.*..............................
0 T.*..............................
28.12s 15:4 destroyed
28.31s 4:6 destroyed
28.31s 5:7 destroyed
...
32.00 seconds, 512 frames; kayan_verify after every one: 0 problems
drones: 28 came, 27 destroyed, 1 reached the base, 0 still flying
bullets: 110 fired, 111 hits; fires started: 28
entities: at most 27 alive at once; commands: 716 applied, 1 skipped
The last three drones burn down one unit short of the base. There's one more hit than bullets: once, a bullet reached two drones in the same frame. That's also the one skipped command, its second despawn.
the schedule, and what each system ran over:
turret called 512 times, over 1536 rows
age called 409 times, over 1622 rows
expire called 409 times, over 1622 rows
move called 1018 times, over 5051 rows
hit called 409 times, over 1622 rows
ignite called 443 times, over 3145 rows
burn called 166 times, over 284 rows
die called 609 times, over 3429 rows
breach called 609 times, over 3429 rows
And the whole game ran on four tables:
the tables at the end:
table 0 {position turret}: 0:1 1:1 2:1
table 1 {position velocity health}:
table 2 {position velocity damage lifetime}:
table 3 {position velocity health burning}:
What you built
Part 1 entities handles, generations, slots, swap-remove, kayan_verify
Part 2 components ids, columns, tables, archetypes, typed access,
moving between tables, the archetype graph, reflection
Part 3 systems queries, kernels and schedules, commands, a game
That's a working archetype ECS, about 650 lines of kayan without comments. It's the design flecs is built on (tables, the archetype graph, cached queries, command buffers), and Bevy and Unity DOTS are close cousins, minus their years of features.
Things worth adding next, roughly in order of how much they teach:
- Relationships:
child_of(parent)as a component, flecs' big idea. - Change detection: a per-column version number, so a system can skip tables nothing wrote to.
- Parallel systems: systems that don't write what others read can run at the same time. The schedule already holds what you need to decide that.
- Removing empty tables, and the queries' table lists with them.
- Serialization: the reflect records already describe every byte; write the tables out and read them back.
Try
- Add a fourth lane. What changes? (Only the level:
lanes, the picture.) - Make bullets pierce: they don't despawn on a hit, but each drone can only be hit by a bullet once. Where does "once" live? (A component on the bullet? On the drone?)
- Add a
shieldcomponent that absorbs the next hit, then removes itself. Which systems change? - Print
kayan_inspectof a drone the frame it catches fire.
1// 3.4 A game2//3// Everything together. Three lanes, a turret at the left end of each, and4// drones coming from the right in waves. A turret fires down its lane while a5// drone is in it; a bullet that reaches a drone damages it; a drone below half6// health catches fire, and one with no health left is destroyed. A drone that7// gets to the left end has reached the base.8//9// kayan entities, tables, queries, systems and commands: 1.1 to 3.3,10// unchanged11// the game components, kernels, systems and a schedule: the end of this12// file13//14// Every rule of the game is a system over a query, and every structural change15// a system makes goes through a command. The level script that sends the waves16// runs between frames, where spawning directly is safe.17//18// Every speed and time step is a power of two, so the arithmetic is exact and19// a Debug and a Release build print the same game.20 21import "std/print.rin"22import "std/memops.rin"23import "std/vec.rin"24import "std/hashmap.rin"25 26vec2: struct = {27 x: f32;28 y: f32;29}30 31position: struct = { at: vec2; }32velocity: struct = { per_second: vec2; }33health: struct = { points: f32; max: f32; }34damage: struct = { amount: f32; }35lifetime: struct = { seconds: f32; }36// A field tagged "hidden" is bookkeeping the inspector does not show.37turret: struct = { cooldown: f32; timer: f32 @ "hidden"; }38burning: struct = { per_second: f32; }39 40component: enum = {41 position,42 velocity,43 health,44 damage,45 lifetime,46 turret,47 burning,48}49 50component_of: proc<position>() -> component = { return component.position; }51component_of: proc<velocity>() -> component = { return component.velocity; }52component_of: proc<health>() -> component = { return component.health; }53component_of: proc<damage>() -> component = { return component.damage; }54component_of: proc<lifetime>() -> component = { return component.lifetime; }55component_of: proc<turret>() -> component = { return component.turret; }56component_of: proc<burning>() -> component = { return component.burning; }57 58// ---------------------------------------------------------------------------59// Entities (Part 1)60// ---------------------------------------------------------------------------61 62kayan_entity: struct = {63 index: u32;64 generation: u32;65}66 67// `missing`: the entity is alive but has no such component.68kayan_status: enum[must_use] = {69 ok,70 dead,71 missing,72}73 74// Where an entity lives: which table, and which row of it.75kayan_slot: struct = {76 generation: u32;77 table: u32;78 row: u32;79}80 81// ---------------------------------------------------------------------------82// Columns (2.2)83// ---------------------------------------------------------------------------84 85kayan_column: struct = {86 data: ^u8;87 size: u64;88 align: u64;89}90 91kayan_column_init: proc(info: *const reflect) -> kayan_column = {92 c: kayan_column = {};93 c.size = info*.size;94 c.align = info*.align;95 return c;96}97 98kayan_column_row: proc(c: *kayan_column, row: u64) -> *u8 = {99 return c*.data[row * c*.size]&;100}101 102kayan_column_grow: proc(arena: *memops_arena, c: *kayan_column, count: u64, capacity: u64) -> void = {103 data: ^u8 = cast(memops_arena_push_zero(arena, c*.size * capacity, c*.align), ^u8);104 if (count > 0) {105 memcpy(data, c*.data, c*.size * count);106 }107 c*.data = data;108}109 110kayan_column_copy_row: proc(c: *kayan_column, to: u64, from: u64) -> void = {111 memcpy(kayan_column_row(c, to), kayan_column_row(c, from), c*.size);112}113 114kayan_column_zero_row: proc(c: *kayan_column, row: u64) -> void = {115 memset(kayan_column_row(c, row), 0, c*.size);116}117 118kayan_column_slice: proc<T>(c: *kayan_column, count: u64) -> []T = {119 if (c*.size != sizeof(T)) {120 printfmt("kayan: a column of {}-byte rows read as a {}-byte type\n", c*.size, sizeof(T));121 exit(1);122 }123 raw: ^T = cast(c*.data, ^T);124 return raw[0..count];125}126 127// ---------------------------------------------------------------------------128// The table129// ---------------------------------------------------------------------------130 131// `columns[id]` is in use when the signature has bit `id`; the others stay132// zeroed. Count and capacity are the table's, shared by its columns and by133// `entities`, so they can never disagree.134//135// The edges: `add_edges[id]` is the table this one's entities move to when136// they gain component `id`, and `remove_edges[id]` when they lose it. Each is137// the table's index plus one, so 0, which a new table starts with, means "not138// known yet".139kayan_table: struct = {140 signature: u64;141 count: u64;142 capacity: u64;143 entities: ^kayan_entity;144 columns: [64]kayan_column;145 add_edges: [64]u32;146 remove_edges: [64]u32;147}148 149// A structural change recorded during a frame and applied after it (3.3).150// `entity` is who it applies to; a spawn's is filled in when it is applied. An151// add to an entity spawned in the same frame has no entity to name yet, so it152// names its spawn instead: `spawn` is that command's index plus one.153kayan_command_kind: enum = {154 spawn,155 despawn,156 add,157 remove,158}159 160kayan_command: struct = {161 kind: kayan_command_kind;162 entity: kayan_entity;163 spawn: u32;164 id: u32;165 signature: u64;166 value: []u8;167}168 169// The world: the registry, the slots and free list, and every table, with a170// map from signature to table index so finding one is a lookup, not a search.171// Tables are referred to by index, never by pointer: making a table can grow172// `tables` and move all of them, and an index survives that.173kayan_world: struct = {174 arena: *memops_arena;175 infos: [64]*const reflect;176 component_count: u32;177 tables: vec<kayan_table>;178 by_signature: hashmap<u64, u32>;179 slots: vec<kayan_slot>;180 free: vec<u32>;181 live: u64;182 // The frame's commands, and their values, live in `frame`, which is183 // cleared once they are applied. `trace` prints each one as it is.184 frame: memops_arena;185 commands: vec<kayan_command>;186 applied: u64;187 skipped: u64;188 trace: b32;189}190 191kayan_bit: proc(id: u32) -> u64 = {192 return 1'u64 shl cast(id, u64);193}194 195kayan_id_of: proc<T>() -> u32 = {196 return cast(component_of<T>(), u32);197}198 199kayan_mask_of: proc<T>() -> u64 = {200 return kayan_bit(kayan_id_of<T>());201}202 203kayan_register: proc(w: *kayan_world, id: u32, info: *const reflect) -> void = {204 if (id >= 64) {205 printfmt("kayan: component id {} is past the 64 a signature can hold\n", id);206 exit(1);207 }208 w*.infos[id] = info;209 if (id + 1 > w*.component_count) {210 w*.component_count = id + 1;211 }212}213 214kayan_table_has: proc(t: *kayan_table, id: u32) -> b32 = {215 return (t*.signature & kayan_bit(id)) != 0;216}217 218// A table for a signature: one empty column per component in it, sized from219// the registry. Every component in the signature must be registered.220kayan_table_init: proc(w: *kayan_world, signature: u64) -> kayan_table = {221 t: kayan_table = {};222 t.signature = signature;223 for (id: u32 = 0; id < 64; id += 1) {224 if ((signature & kayan_bit(id)) != 0) {225 if (id >= w*.component_count || w*.infos[id] == null) {226 printfmt("kayan: signature {} uses component {}, which was never registered\n", signature, id);227 exit(1);228 }229 t.columns[id] = kayan_column_init(w*.infos[id]);230 }231 }232 return t;233}234 235// Double the room in `entities` and in every column at once.236kayan_table_grow: proc(w: *kayan_world, t: *kayan_table) -> void = {237 capacity: u64 = t*.capacity == 0 ? 4 : t*.capacity * 2;238 entities: ^kayan_entity = cast(memops_arena_push_zero(w*.arena,239 sizeof(kayan_entity) * capacity, alignof(kayan_entity)), ^kayan_entity);240 if (t*.count > 0) {241 memcpy(entities, t*.entities, sizeof(kayan_entity) * t*.count);242 }243 t*.entities = entities;244 for (id: u32 = 0; id < w*.component_count; id += 1) {245 if (kayan_table_has(t, id)) {246 kayan_column_grow(w*.arena, t*.columns[id]&, t*.count, capacity);247 }248 }249 t*.capacity = capacity;250}251 252// A new row at the end, owned by `e`, with every component zeroed.253kayan_table_add_row: proc(w: *kayan_world, t: *kayan_table, e: kayan_entity) -> u64 = {254 if (t*.count == t*.capacity) {255 kayan_table_grow(w, t);256 }257 row: u64 = t*.count;258 t*.count += 1;259 t*.entities[row] = e;260 for (id: u32 = 0; id < w*.component_count; id += 1) {261 if (kayan_table_has(t, id)) {262 kayan_column_zero_row(t*.columns[id]&, row);263 }264 }265 return row;266}267 268// 1.4's swap-remove, for every column: the last row is copied over `row`, and269// the slot of the entity that owned the last row is told where it went.270kayan_table_remove_row: proc(w: *kayan_world, t: *kayan_table, row: u64) -> void = {271 last: u64 = t*.count - 1;272 if (row != last) {273 for (id: u32 = 0; id < w*.component_count; id += 1) {274 if (kayan_table_has(t, id)) {275 kayan_column_copy_row(t*.columns[id]&, row, last);276 }277 }278 moved: kayan_entity = t*.entities[last];279 t*.entities[row] = moved;280 vec<kayan_slot>at(w*.slots&, cast(moved.index, u64))*.row = cast(row, u32);281 }282 t*.count = last;283}284 285// The typed view of one of the table's columns: T's rows, all of them in use.286// Asking for a component the table does not have is a bug, so it stops.287kayan_slice: proc<T>(t: *kayan_table) -> []T = {288 id: u32 = kayan_id_of<T>();289 if (!kayan_table_has(t, id)) {290 printfmt("kayan: the table has no column for component {}\n", id);291 exit(1);292 }293 return kayan_column_slice<T>(t*.columns[id]&, t*.count);294}295 296// Who owns each of the table's rows.297kayan_entities: proc(t: *kayan_table) -> []kayan_entity = {298 return t*.entities[0..t*.count];299}300 301// A pointer to a table is good until the next table is made.302kayan_table_at: proc(w: *kayan_world, index: u32) -> *kayan_table = {303 return vec<kayan_table>at(w*.tables&, cast(index, u64));304}305 306// The table for a signature: found if it exists, made if it does not. This is307// where archetypes come from.308kayan_table_for: proc(w: *kayan_world, signature: u64) -> u32 = {309 found: *u32 = hashmap<u64, u32>get(w*.by_signature&, signature);310 if (found != null) {311 return found*;312 }313 index: u32 = cast(w*.tables.length, u32);314 _ = vec<kayan_table>append(w*.arena, w*.tables&, kayan_table_init(w, signature));315 _ = hashmap<u64, u32>put(w*.arena, w*.by_signature&, signature, index);316 return index;317}318 319// ---------------------------------------------------------------------------320// The world321// ---------------------------------------------------------------------------322 323kayan_world_init: proc(w: *kayan_world, arena: *memops_arena) -> void = {324 w* = {};325 w*.arena = arena;326 // A frame's commands are small: 256 MB of address space is plenty.327 memops_arena_initialize(w*.frame&, 0x10000000);328}329 330kayan_alive: proc(w: *kayan_world, e: kayan_entity) -> b32 = {331 if (cast(e.index, u64) >= w*.slots.length) {332 return false;333 }334 return vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.generation == e.generation;335}336 337// A new entity with the components in `signature`, all zeroed, in the table338// for that signature.339kayan_spawn: proc(w: *kayan_world, signature: u64) -> kayan_entity = {340 e: kayan_entity = {};341 if (!vec<u32>is_empty(w*.free&)) {342 e.index = vec<u32>pop(w*.free&);343 } else {344 e.index = cast(w*.slots.length, u32);345 _ = vec<kayan_slot>append(w*.arena, w*.slots&, { .generation = 1 });346 }347 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));348 e.generation = slot*.generation;349 // kayan_table_for can make a table and move the others, so the table's350 // pointer is taken after it.351 table_index: u32 = kayan_table_for(w, signature);352 t: *kayan_table = kayan_table_at(w, table_index);353 slot*.table = table_index;354 slot*.row = cast(kayan_table_add_row(w, t, e), u32);355 w*.live += 1;356 return e;357}358 359kayan_despawn: proc(w: *kayan_world, e: kayan_entity) -> kayan_status = {360 if (!kayan_alive(w, e)) {361 return kayan_status.dead;362 }363 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));364 kayan_table_remove_row(w, kayan_table_at(w, slot*.table), cast(slot*.row, u64));365 slot*.generation += 1;366 _ = vec<u32>append(w*.arena, w*.free&, e.index);367 w*.live -= 1;368 return kayan_status.ok;369}370 371// The components a live entity has: its table's signature. 0 for a dead one.372kayan_signature: proc(w: *kayan_world, e: kayan_entity) -> u64 = {373 if (!kayan_alive(w, e)) {374 return 0;375 }376 return kayan_table_at(w, vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.table)*.signature;377}378 379// ---------------------------------------------------------------------------380// Moving between tables381// ---------------------------------------------------------------------------382 383// The table an entity of table `from` moves to when it gains component `id`.384// The first time, it is found by signature, which can make the table; the385// answer is written on both tables, as from's add edge and the destination's386// remove edge, so neither direction is looked up again.387kayan_table_with: proc(w: *kayan_world, from: u32, id: u32) -> u32 = {388 edge: u32 = kayan_table_at(w, from)*.add_edges[id];389 if (edge != 0) {390 return edge - 1;391 }392 to: u32 = kayan_table_for(w, kayan_table_at(w, from)*.signature | kayan_bit(id));393 // kayan_table_for can move the tables, so both pointers are taken after it.394 kayan_table_at(w, from)*.add_edges[id] = to + 1;395 kayan_table_at(w, to)*.remove_edges[id] = from + 1;396 return to;397}398 399// The same for losing component `id`.400kayan_table_without: proc(w: *kayan_world, from: u32, id: u32) -> u32 = {401 edge: u32 = kayan_table_at(w, from)*.remove_edges[id];402 if (edge != 0) {403 return edge - 1;404 }405 to: u32 = kayan_table_for(w, kayan_table_at(w, from)*.signature & ~kayan_bit(id));406 kayan_table_at(w, from)*.remove_edges[id] = to + 1;407 kayan_table_at(w, to)*.add_edges[id] = from + 1;408 return to;409}410 411// Move a live entity to table `to_index`. Components both tables have are412// copied, byte for byte; components only the new table has start zeroed;413// components only the old table has are dropped with its old row.414kayan_move: proc(w: *kayan_world, e: kayan_entity, to_index: u32) -> void = {415 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));416 from_index: u32 = slot*.table;417 if (from_index == to_index) {418 return;419 }420 from: *kayan_table = kayan_table_at(w, from_index);421 to: *kayan_table = kayan_table_at(w, to_index);422 from_row: u64 = cast(slot*.row, u64);423 to_row: u64 = kayan_table_add_row(w, to, e);424 for (id: u32 = 0; id < w*.component_count; id += 1) {425 if (kayan_table_has(from, id) && kayan_table_has(to, id)) {426 memcpy(kayan_column_row(to*.columns[id]&, to_row),427 kayan_column_row(from*.columns[id]&, from_row), from*.columns[id].size);428 }429 }430 kayan_table_remove_row(w, from, from_row);431 slot*.table = to_index;432 slot*.row = cast(to_row, u32);433}434 435// ---------------------------------------------------------------------------436// The byte layer, for tools437// ---------------------------------------------------------------------------438 439// e's component `id` as its bytes: an empty slice if e is dead or has no such440// component. The typed layer is for game code, which knows its types; this is441// for tools that work from reflect records instead.442kayan_get_bytes: proc(w: *kayan_world, e: kayan_entity, id: u32) -> []u8 = {443 none: []u8 = {};444 if (!kayan_alive(w, e) || id >= w*.component_count) {445 return none;446 }447 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));448 t: *kayan_table = kayan_table_at(w, slot*.table);449 if (!kayan_table_has(t, id)) {450 return none;451 }452 c: *kayan_column = t*.columns[id]&;453 row: u64 = cast(slot*.row, u64);454 all: []u8 = c*.data[0..t*.count * c*.size];455 return all[row * c*.size..(row + 1) * c*.size];456}457 458// Give e component `id`, with `value` for its bytes: 2.6's add, by id instead459// of by type, because a command holds its value as bytes.460kayan_add_id: proc(w: *kayan_world, e: kayan_entity, id: u32, value: []const u8) -> kayan_status = {461 if (!kayan_alive(w, e)) {462 return kayan_status.dead;463 }464 from: u32 = vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.table;465 if (!kayan_table_has(kayan_table_at(w, from), id)) {466 kayan_move(w, e, kayan_table_with(w, from, id));467 }468 bytes: []u8 = kayan_get_bytes(w, e, id);469 if (value.length != bytes.length) {470 printfmt("kayan: a {}-byte value for a component of {} bytes\n", value.length, bytes.length);471 exit(1);472 }473 memcpy(bytes[0]&, value[0]&, bytes.length);474 return kayan_status.ok;475}476 477kayan_remove_id: proc(w: *kayan_world, e: kayan_entity, id: u32) -> kayan_status = {478 if (!kayan_alive(w, e)) {479 return kayan_status.dead;480 }481 from: u32 = vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.table;482 if (!kayan_table_has(kayan_table_at(w, from), id)) {483 return kayan_status.missing;484 }485 kayan_move(w, e, kayan_table_without(w, from, id));486 return kayan_status.ok;487}488 489// ---------------------------------------------------------------------------490// The typed layer491// ---------------------------------------------------------------------------492 493// e's T, or null if e is dead or has no T. Handle, slot, table, row, typed494// view: the same path as 1.4's get_position, for any component. The pointer is495// good until the next structural change, so use it now and keep the handle.496kayan_get: proc<T>(w: *kayan_world, e: kayan_entity) -> *T = {497 if (!kayan_alive(w, e)) {498 return null;499 }500 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));501 t: *kayan_table = kayan_table_at(w, slot*.table);502 if (!kayan_table_has(t, kayan_id_of<T>())) {503 return null;504 }505 rows: []T = kayan_slice<T>(t);506 return rows[cast(slot*.row, u64)]&;507}508 509kayan_has: proc<T>(w: *kayan_world, e: kayan_entity) -> b32 = {510 return (kayan_signature(w, e) & kayan_mask_of<T>()) != 0;511}512 513// Write e's T. It must already have one: giving an entity a component it does514// not have changes its table, which is 2.6.515kayan_set: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> kayan_status = {516 if (!kayan_alive(w, e)) {517 return kayan_status.dead;518 }519 p: *T = kayan_get<T>(w, e);520 if (p == null) {521 return kayan_status.missing;522 }523 p* = value;524 return kayan_status.ok;525}526 527// Give e a T, or take its T away: the byte procs, with the type turned into an528// id and the value into its bytes.529kayan_add: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> kayan_status = {530 bytes: ^const u8 = cast(value&, ^const u8);531 return kayan_add_id(w, e, kayan_id_of<T>(), bytes[0..sizeof(T)]);532}533 534kayan_remove: proc<T>(w: *kayan_world, e: kayan_entity) -> kayan_status = {535 return kayan_remove_id(w, e, kayan_id_of<T>());536}537 538// The invariant, over every table: each row belongs to a live entity whose539// slot points back at that table and row, and the rows of all tables add up to540// the live count.541kayan_verify: proc(w: *kayan_world) -> u64 = {542 problems: u64 = 0;543 rows: u64 = 0;544 for (ti: u64 = 0; ti < w*.tables.length; ti += 1) {545 t: *kayan_table = kayan_table_at(w, cast(ti, u32));546 for (row: u64 = 0; row < t*.count; row += 1) {547 e: kayan_entity = t*.entities[row];548 if (!kayan_alive(w, e)) {549 printfmt("kayan_verify: table {} row {} holds dead entity {}:{}\n", ti, row, e.index, e.generation);550 problems += 1;551 continue;552 }553 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));554 if (cast(slot*.table, u64) != ti || cast(slot*.row, u64) != row) {555 printfmt("kayan_verify: {}:{} is in table {} row {}, its slot says table {} row {}\n",556 e.index, e.generation, ti, row, slot*.table, slot*.row);557 problems += 1;558 }559 }560 rows += t*.count;561 }562 if (rows != w*.live) {563 printfmt("kayan_verify: {} rows in tables, {} live entities\n", rows, w*.live);564 problems += 1;565 }566 return problems;567}568 569kayan_print_signature: proc(w: *kayan_world, signature: u64) -> void = {570 printf("{");571 first: b32 = true;572 for (id: u32 = 0; id < w*.component_count; id += 1) {573 if ((signature & kayan_bit(id)) != 0) {574 printf(first ? "%s" : " %s", w*.infos[id]*.name);575 first = false;576 }577 }578 printf("}");579}580 581kayan_print_table: proc(w: *kayan_world, ti: u32) -> void = {582 t: *kayan_table = kayan_table_at(w, ti);583 printf(" table %u ", ti);584 kayan_print_signature(w, t*.signature);585 printf(":");586 for (row: u64 = 0; row < t*.count; row += 1) {587 printf(" %u:%u", t*.entities[row].index, t*.entities[row].generation);588 }589 printf("\n");590}591 592kayan_print_tables: proc(w: *kayan_world) -> void = {593 for (ti: u32 = 0; ti < cast(w*.tables.length, u32); ti += 1) {594 kayan_print_table(w, ti);595 }596}597 598// ---------------------------------------------------------------------------599// The inspector: reflect records, no component types600// ---------------------------------------------------------------------------601 602kayan_indent: proc(depth: u32) -> void = {603 for (i: u32 = 0; i < depth; i += 1) {604 printf(" ");605 }606}607 608// One value of a builtin type, read from its bytes by its type's name.609kayan_print_builtin: proc(type_name: ^const char, at: []const u8) -> void = {610 if (strcmp(type_name, "f32") == 0) {611 printf("%.2f", cast(cast(at[0]&, *const f32)*, f64));612 } else if (strcmp(type_name, "u32") == 0) {613 printf("%u", cast(at[0]&, *const u32)*);614 } else if (strcmp(type_name, "i32") == 0) {615 printf("%d", cast(at[0]&, *const i32)*);616 } else {617 printf("<%s>", type_name);618 }619}620 621// Every field of a struct whose bytes are `base`, nested structs indented622// under their field. A field's bytes are cut from `base` by its offset and623// size, both from the reflect record.624kayan_print_fields: proc(type: *const reflect, base: []const u8, depth: u32) -> void = {625 fields: ^const reflect_field = reflect_fields(type);626 for (i: u64 = 0; i < type*.count; i += 1) {627 f: *const reflect_field = fields[i]&;628 if (reflect_field_has_attr(f, "hidden")) {629 continue;630 }631 at: []const u8 = base[f*.offset..f*.offset + f*.size];632 kayan_indent(depth);633 printf("%s:", f*.name);634 info: *const reflect = f*.info;635 if (info != null && info*.kind == reflect_kind_struct) {636 printf("\n");637 kayan_print_fields(info, at, depth + 1);638 } else {639 printf(" ");640 kayan_print_builtin(f*.type, at);641 printf("\n");642 }643 }644}645 646kayan_inspect: proc(w: *kayan_world, e: kayan_entity) -> void = {647 if (!kayan_alive(w, e)) {648 printfmt("{}:{} (dead)\n", e.index, e.generation);649 return;650 }651 printf("%u:%u ", e.index, e.generation);652 signature: u64 = kayan_signature(w, e);653 kayan_print_signature(w, signature);654 printf("\n");655 for (id: u32 = 0; id < w*.component_count; id += 1) {656 if ((signature & kayan_bit(id)) != 0) {657 info: *const reflect = w*.infos[id];658 printf(" %s\n", info*.name);659 bytes: []u8 = kayan_get_bytes(w, e, id);660 kayan_print_fields(info, bytes, 2);661 }662 }663}664 665// Every table: rows in use out of room, and per column its row size and its666// bytes in use out of bytes allocated. The memory a table holds and does not667// use is the cost of doubling on growth.668kayan_print_layout: proc(w: *kayan_world) -> void = {669 for (ti: u32 = 0; ti < cast(w*.tables.length, u32); ti += 1) {670 t: *kayan_table = kayan_table_at(w, ti);671 printf("table %u ", ti);672 kayan_print_signature(w, t*.signature);673 printf(": %u rows, room for %u\n", cast(t*.count, u32), cast(t*.capacity, u32));674 for (id: u32 = 0; id < w*.component_count; id += 1) {675 if (kayan_table_has(t, id)) {676 size: u64 = t*.columns[id].size;677 printf(" %-9s %2u bytes a row, %4u of %4u bytes in use\n", w*.infos[id]*.name, cast(size, u32),678 cast(size * t*.count, u32), cast(size * t*.capacity, u32));679 }680 }681 }682}683 684// ---------------------------------------------------------------------------685// Queries686// ---------------------------------------------------------------------------687 688// The tables a system runs over: every table whose signature has all of `all`689// and none of `none`, kept by index. Tables are only ever added, at the end of690// w.tables, so the query remembers how many it has checked and, brought up to691// date, looks at the new ones only.692kayan_query: struct = {693 all: u64;694 none: u64;695 tables: vec<u32>;696 checked: u64;697}698 699kayan_query_init: proc(all: u64, none: u64) -> kayan_query = {700 q: kayan_query = {};701 q.all = all;702 q.none = none;703 return q;704}705 706kayan_matches: proc(signature: u64, all: u64, none: u64) -> b32 = {707 return (signature & all) == all && (signature & none) == 0;708}709 710// Check the tables made since the query last looked. With none new, this is711// one comparison, so call it before every walk.712kayan_query_update: proc(w: *kayan_world, q: *kayan_query) -> void = {713 for (ti: u64 = q*.checked; ti < w*.tables.length; ti += 1) {714 if (kayan_matches(kayan_table_at(w, cast(ti, u32))*.signature, q*.all, q*.none)) {715 _ = vec<u32>append(w*.arena, q*.tables&, cast(ti, u32));716 }717 }718 q*.checked = w*.tables.length;719}720 721// The query's i-th table. Like any table pointer, good until a table is made.722kayan_query_table: proc(w: *kayan_world, q: *kayan_query, i: u64) -> *kayan_table = {723 return kayan_table_at(w, vec<u32>at(q*.tables&, i)*);724}725 726// How many entities the query matches: the rows of its tables.727kayan_query_count: proc(w: *kayan_world, q: *kayan_query) -> u64 = {728 kayan_query_update(w, q);729 n: u64 = 0;730 for (i: u64 = 0; i < q*.tables.length; i += 1) {731 n += kayan_query_table(w, q, i)*.count;732 }733 return n;734}735 736kayan_print_query: proc(w: *kayan_world, name: ^const char, q: *kayan_query) -> void = {737 kayan_query_update(w, q);738 printf("%s: all ", name);739 kayan_print_signature(w, q*.all);740 printf(", none ");741 kayan_print_signature(w, q*.none);742 printfmt("; tables: {}, entities: {}\n", q*.tables.length, kayan_query_count(w, q));743 for (i: u64 = 0; i < q*.tables.length; i += 1) {744 kayan_print_table(w, vec<u32>at(q*.tables&, i)*);745 }746}747 748// ---------------------------------------------------------------------------749// Commands750// ---------------------------------------------------------------------------751 752// A spawn that has been recorded and not applied. There is no entity yet, so753// this is the spawn command's place in the list; give it components with754// kayan_pending_set.755kayan_pending: struct = {756 command: u32;757}758 759kayan_defer: proc(w: *kayan_world, c: kayan_command) -> u32 = {760 index: u32 = cast(w*.commands.length, u32);761 _ = vec<kayan_command>append(w*.frame&, w*.commands&, c);762 return index;763}764 765// A copy of `value`, as bytes, in the frame arena.766kayan_defer_value: proc<T>(w: *kayan_world, value: T) -> []u8 = {767 data: ^u8 = cast(memops_arena_push(w*.frame&, sizeof(T), alignof(T)), ^u8);768 memcpy(data, value&, sizeof(T));769 return data[0..sizeof(T)];770}771 772kayan_defer_spawn: proc(w: *kayan_world, signature: u64) -> kayan_pending = {773 p: kayan_pending = {};774 p.command = kayan_defer(w, { .kind = kayan_command_kind.spawn, .signature = signature });775 return p;776}777 778kayan_pending_set: proc<T>(w: *kayan_world, p: kayan_pending, value: T) -> void = {779 _ = kayan_defer(w, { .kind = kayan_command_kind.add, .spawn = p.command + 1, .id = kayan_id_of<T>(),780 .value = kayan_defer_value<T>(w, value) });781}782 783kayan_defer_despawn: proc(w: *kayan_world, e: kayan_entity) -> void = {784 _ = kayan_defer(w, { .kind = kayan_command_kind.despawn, .entity = e });785}786 787kayan_defer_add: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> void = {788 _ = kayan_defer(w, { .kind = kayan_command_kind.add, .entity = e, .id = kayan_id_of<T>(),789 .value = kayan_defer_value<T>(w, value) });790}791 792kayan_defer_remove: proc<T>(w: *kayan_world, e: kayan_entity) -> void = {793 _ = kayan_defer(w, { .kind = kayan_command_kind.remove, .entity = e, .id = kayan_id_of<T>() });794}795 796// One applied command. The components given to a new entity are part of its797// spawn line, so they are not printed on their own.798kayan_print_command: proc(w: *kayan_world, c: *kayan_command, status: kayan_status) -> void = {799 if (c*.spawn != 0 && status == kayan_status.ok) {800 return;801 }802 printfmt(" {} ", c*.kind);803 if (c*.kind == kayan_command_kind.spawn) {804 kayan_print_signature(w, c*.signature);805 printf(" -> ");806 } else if (c*.kind != kayan_command_kind.despawn) {807 printf("%s, ", w*.infos[c*.id]*.name);808 }809 printf("%u:%u", c*.entity.index, c*.entity.generation);810 if (status != kayan_status.ok) {811 printfmt(": skipped, {}", status);812 }813 printf("\n");814}815 816// Apply the frame's commands in the order they were recorded, then let them817// all go at once. A command for an entity that died after it was recorded818// (despawned by an earlier command, most likely) does nothing, and is counted.819kayan_flush: proc(w: *kayan_world) -> void = {820 for (i: u64 = 0; i < w*.commands.length; i += 1) {821 c: *kayan_command = vec<kayan_command>at(w*.commands&, i);822 if (c*.spawn != 0) {823 c*.entity = vec<kayan_command>at(w*.commands&, cast(c*.spawn - 1, u64))*.entity;824 }825 status: kayan_status = kayan_status.ok;826 if (c*.kind == kayan_command_kind.spawn) {827 c*.entity = kayan_spawn(w, c*.signature);828 } else if (c*.kind == kayan_command_kind.despawn) {829 status = kayan_despawn(w, c*.entity);830 } else if (c*.kind == kayan_command_kind.add) {831 status = kayan_add_id(w, c*.entity, c*.id, c*.value);832 } else {833 status = kayan_remove_id(w, c*.entity, c*.id);834 }835 if (status == kayan_status.ok) {836 w*.applied += 1;837 } else {838 w*.skipped += 1;839 }840 if (w*.trace) {841 kayan_print_command(w, c, status);842 }843 }844 w*.commands = {};845 memops_arena_clear(w*.frame&);846}847 848// ---------------------------------------------------------------------------849// Systems850// ---------------------------------------------------------------------------851 852// What a system gets for each table it runs on: the world, the table, and the853// frame's time step. One struct rather than three parameters, so that adding854// to it later changes no system's signature.855kayan_iter: struct = {856 world: *kayan_world;857 table: *kayan_table;858 dt: f32;859 ctx: *void;860}861 862kayan_system_proc: alias = *proc(it: *kayan_iter) -> void;863 864// A system: a name, the query that finds its tables, and the proc run on each865// of those tables that has rows. `tables` and `rows` count what it has run866// over. `ctx` is handed to the proc as is: whatever the system needs beyond867// its own tables, such as another query, or the game's state.868kayan_system: struct = {869 name: ^const char;870 query: kayan_query;871 run: kayan_system_proc;872 ctx: *void;873 tables: u64;874 rows: u64;875}876 877kayan_system_init: proc(name: ^const char, all: u64, none: u64, run: kayan_system_proc, ctx: *void) -> kayan_system = {878 s: kayan_system = {};879 s.name = name;880 s.query = kayan_query_init(all, none);881 s.run = run;882 s.ctx = ctx;883 return s;884}885 886// One frame: every system in order, each over every table it matches that has887// rows, then the commands they recorded. A system sees every value the888// systems before it wrote this frame, and no structural change until the889// next.890kayan_progress: proc(w: *kayan_world, systems: []kayan_system, dt: f32) -> void = {891 for (s: u64 = 0; s < systems.length; s += 1) {892 sys: *kayan_system = systems[s]&;893 kayan_query_update(w, sys*.query&);894 for (i: u64 = 0; i < sys*.query.tables.length; i += 1) {895 it: kayan_iter = {};896 it.world = w;897 it.table = kayan_query_table(w, sys*.query&, i);898 it.dt = dt;899 it.ctx = sys*.ctx;900 if (it.table*.count > 0) {901 sys*.tables += 1;902 sys*.rows += it.table*.count;903 sys*.run(it&);904 }905 }906 }907 kayan_flush(w);908}909 910kayan_print_schedule: proc(w: *kayan_world, systems: []kayan_system) -> void = {911 for (s: u64 = 0; s < systems.length; s += 1) {912 kayan_print_query(w, systems[s].name, systems[s].query&);913 }914}915 916// ---------------------------------------------------------------------------917// The program918// ---------------------------------------------------------------------------919 920// ---------------------------------------------------------------------------921// The game922// ---------------------------------------------------------------------------923 924// What the game's systems share: the drones, for the systems that look for925// them, and the score. Systems get it through their ctx.926game: struct = {927 drones: kayan_query;928 time: f32;929 fired: u32;930 hits: u32;931 fires: u32;932 kills: u32;933 breaches: u32;934 came: u32;935}936 937// Kernels: plain loops over plain slices, as in 3.2.938integrate: proc(ps: noalias []position, vs: []const velocity, dt: f32) -> void = {939 for (i: u64 = 0; i < ps.length; i += 1) {940 ps[i].at.x += vs[i].per_second.x * dt;941 ps[i].at.y += vs[i].per_second.y * dt;942 }943}944 945burn: proc(hs: noalias []health, fires: []const burning, dt: f32) -> void = {946 for (i: u64 = 0; i < hs.length; i += 1) {947 hs[i].points -= fires[i].per_second * dt;948 }949}950 951age: proc(ls: noalias []lifetime, dt: f32) -> void = {952 for (i: u64 = 0; i < ls.length; i += 1) {953 ls[i].seconds -= dt;954 }955}956 957bullet_signature: proc() -> u64 = {958 return kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<damage>() | kayan_mask_of<lifetime>();959}960 961drone_signature: proc() -> u64 = {962 return kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<health>();963}964 965// Is there a drone in the lane at height y, ahead of x?966drone_ahead: proc(w: *kayan_world, g: *game, x: f32, y: f32) -> b32 = {967 for (i: u64 = 0; i < g*.drones.tables.length; i += 1) {968 ps: []position = kayan_slice<position>(kayan_query_table(w, g*.drones&, i));969 for (row: u64 = 0; row < ps.length; row += 1) {970 dy: f32 = ps[row].at.y - y;971 if (dy * dy < 0.25 && ps[row].at.x > x) {972 return true;973 }974 }975 }976 return false;977}978 979// A turret whose timer is up fires, if it has something to fire at.980turret_system: proc(it: *kayan_iter) -> void = {981 g: *game = cast(it*.ctx, *game);982 ps: []position = kayan_slice<position>(it*.table);983 ts: []turret = kayan_slice<turret>(it*.table);984 kayan_query_update(it*.world, g*.drones&);985 for (row: u64 = 0; row < ts.length; row += 1) {986 if (ts[row].timer <= 0.0 && drone_ahead(it*.world, g, ps[row].at.x, ps[row].at.y)) {987 shot: kayan_pending = kayan_defer_spawn(it*.world, bullet_signature());988 kayan_pending_set<position>(it*.world, shot, ps[row]);989 kayan_pending_set<velocity>(it*.world, shot, { .per_second = { .x = 16.0, .y = 0.0 } });990 kayan_pending_set<damage>(it*.world, shot, { .amount = 25.0 });991 kayan_pending_set<lifetime>(it*.world, shot, { .seconds = 2.0 });992 ts[row].timer = ts[row].cooldown;993 g*.fired += 1;994 }995 if (ts[row].timer > 0.0) {996 ts[row].timer -= it*.dt;997 }998 }999}1000 1001age_system: proc(it: *kayan_iter) -> void = {1002 age(kayan_slice<lifetime>(it*.table), it*.dt);1003}1004 1005expire_system: proc(it: *kayan_iter) -> void = {1006 ls: []lifetime = kayan_slice<lifetime>(it*.table);1007 es: []kayan_entity = kayan_entities(it*.table);1008 for (row: u64 = 0; row < es.length; row += 1) {1009 if (ls[row].seconds <= 0.0) {1010 kayan_defer_despawn(it*.world, es[row]);1011 }1012 }1013}1014 1015move_system: proc(it: *kayan_iter) -> void = {1016 integrate(kayan_slice<position>(it*.table), kayan_slice<velocity>(it*.table), it*.dt);1017}1018 1019// A bullet within three quarters of a unit of a drone in its lane hits it. A1020// bullet moves a unit a frame and a drone at most a quarter, so none passes1021// through a drone between two frames without being this close in one of them.1022hit_system: proc(it: *kayan_iter) -> void = {1023 w: *kayan_world = it*.world;1024 g: *game = cast(it*.ctx, *game);1025 ps: []position = kayan_slice<position>(it*.table);1026 ds: []damage = kayan_slice<damage>(it*.table);1027 es: []kayan_entity = kayan_entities(it*.table);1028 kayan_query_update(w, g*.drones&);1029 for (b: u64 = 0; b < es.length; b += 1) {1030 for (i: u64 = 0; i < g*.drones.tables.length; i += 1) {1031 t: *kayan_table = kayan_query_table(w, g*.drones&, i);1032 tps: []position = kayan_slice<position>(t);1033 ths: []health = kayan_slice<health>(t);1034 for (row: u64 = 0; row < tps.length; row += 1) {1035 dx: f32 = tps[row].at.x - ps[b].at.x;1036 dy: f32 = tps[row].at.y - ps[b].at.y;1037 if (dx * dx < 0.5625 && dy * dy < 0.25) {1038 ths[row].points -= ds[b].amount;1039 kayan_defer_despawn(w, es[b]);1040 g*.hits += 1;1041 }1042 }1043 }1044 }1045}1046 1047// Below half health, a drone catches fire: a component added, so a command.1048ignite_system: proc(it: *kayan_iter) -> void = {1049 g: *game = cast(it*.ctx, *game);1050 hs: []health = kayan_slice<health>(it*.table);1051 es: []kayan_entity = kayan_entities(it*.table);1052 for (row: u64 = 0; row < es.length; row += 1) {1053 if (hs[row].points < 0.5 * hs[row].max && hs[row].points > 0.0) {1054 kayan_defer_add<burning>(it*.world, es[row], { .per_second = 10.0 });1055 g*.fires += 1;1056 }1057 }1058}1059 1060burn_system: proc(it: *kayan_iter) -> void = {1061 burn(kayan_slice<health>(it*.table), kayan_slice<burning>(it*.table), it*.dt);1062}1063 1064die_system: proc(it: *kayan_iter) -> void = {1065 g: *game = cast(it*.ctx, *game);1066 hs: []health = kayan_slice<health>(it*.table);1067 es: []kayan_entity = kayan_entities(it*.table);1068 for (row: u64 = 0; row < es.length; row += 1) {1069 if (hs[row].points <= 0.0) {1070 kayan_defer_despawn(it*.world, es[row]);1071 g*.kills += 1;1072 printf("%6.2fs %u:%u destroyed\n", cast(g*.time, f64), es[row].index, es[row].generation);1073 }1074 }1075}1076 1077// A drone still alive at the left end has reached the base.1078breach_system: proc(it: *kayan_iter) -> void = {1079 g: *game = cast(it*.ctx, *game);1080 ps: []position = kayan_slice<position>(it*.table);1081 hs: []health = kayan_slice<health>(it*.table);1082 es: []kayan_entity = kayan_entities(it*.table);1083 for (row: u64 = 0; row < es.length; row += 1) {1084 if (ps[row].at.x <= 0.0 && hs[row].points > 0.0) {1085 kayan_defer_despawn(it*.world, es[row]);1086 g*.breaches += 1;1087 printf("%6.2fs %u:%u reached the base in lane %.0f\n", cast(g*.time, f64),1088 es[row].index, es[row].generation, cast(ps[row].at.y, f64));1089 }1090 }1091}1092 1093// ---------------------------------------------------------------------------1094// The level: waves, and a picture of the lanes1095// ---------------------------------------------------------------------------1096 1097lanes: u32 = 3;1098arena_end: f32 = 32.0;1099 1100spawn_turret: proc(w: *kayan_world, lane: u32) -> kayan_entity = {1101 e: kayan_entity = kayan_spawn(w, kayan_mask_of<position>() | kayan_mask_of<turret>());1102 _ = kayan_set<position>(w, e, { .at = { .x = 0.0, .y = cast(lane, f32) } });1103 // Each lane's turret starts a quarter second after the one below it.1104 _ = kayan_set<turret>(w, e, { .cooldown = 0.75, .timer = 0.25 * cast(lane, f32) });1105 return e;1106}1107 1108spawn_drone: proc(w: *kayan_world, x: f32, lane: u32, speed: f32) -> kayan_entity = {1109 e: kayan_entity = kayan_spawn(w, drone_signature());1110 _ = kayan_set<position>(w, e, { .at = { .x = x, .y = cast(lane, f32) } });1111 _ = kayan_set<velocity>(w, e, { .per_second = { .x = -speed, .y = 0.0 } });1112 _ = kayan_set<health>(w, e, { .points = 100.0, .max = 100.0 });1113 return e;1114}1115 1116// Wave k: `wave_rows[k]` drones, two units apart, in each lane whose bit is1117// set in `wave_lanes[k]`, all at one speed.1118wave_lanes: [8]u32 = { 2, 5, 7, 2, 7, 5, 7, 7 };1119wave_rows: [8]u32 = { 1, 1, 1, 3, 1, 2, 2, 2 };1120wave_speed: [8]f32 = { 2.0, 2.0, 2.0, 2.0, 4.0, 4.0, 2.0, 4.0 };1121 1122send_wave: proc(w: *kayan_world, g: *game, k: u32) -> void = {1123 printf("%6.2fs wave %u:", cast(g*.time, f64), k + 1);1124 for (lane: u32 = 0; lane < lanes; lane += 1) {1125 if ((wave_lanes[k] & (1'u32 shl lane)) != 0) {1126 for (r: u32 = 0; r < wave_rows[k]; r += 1) {1127 e: kayan_entity = spawn_drone(w, arena_end + 2.0 * cast(r, f32), lane, wave_speed[k]);1128 printf(" %u:%u", e.index, e.generation);1129 g*.came += 1;1130 }1131 }1132 }1133 printf(", %.0f units a second\n", cast(wave_speed[k], f64));1134}1135 1136// What an entity is drawn as follows from its table: one check per table,1137// not per entity.1138glyph_of: proc(signature: u64) -> i32 = {1139 if ((signature & kayan_mask_of<turret>()) != 0) {1140 return 'T';1141 }1142 if ((signature & kayan_mask_of<burning>()) != 0) {1143 return '*';1144 }1145 if ((signature & kayan_mask_of<health>()) != 0) {1146 return 'd';1147 }1148 return '-';1149}1150 1151// The lanes as text, lane 2 at the top. Every entity with a position, by way1152// of a query.1153draw: proc(w: *kayan_world, g: *game, everything: *kayan_query) -> void = {1154 cells: [3][33]i32 = {};1155 for (lane: u32 = 0; lane < lanes; lane += 1) {1156 for (x: u32 = 0; x < 33; x += 1) {1157 cells[lane][x] = '.';1158 }1159 }1160 kayan_query_update(w, everything);1161 for (i: u64 = 0; i < everything*.tables.length; i += 1) {1162 t: *kayan_table = kayan_query_table(w, everything, i);1163 glyph: i32 = glyph_of(t*.signature);1164 ps: []position = kayan_slice<position>(t);1165 for (row: u64 = 0; row < ps.length; row += 1) {1166 x: i32 = cast(ps[row].at.x, i32);1167 lane: i32 = cast(ps[row].at.y, i32);1168 if (x >= 0 && x <= 32 && lane >= 0 && lane < 3) {1169 cells[lane][x] = glyph;1170 }1171 }1172 }1173 printf("%6.2fs %u entities in %u tables\n", cast(g*.time, f64), cast(w*.live, u32), cast(w*.tables.length, u32));1174 for (lane: u32 = lanes; lane > 0; lane -= 1) {1175 printf(" %u ", lane - 1);1176 for (x: u32 = 0; x < 33; x += 1) {1177 printf("%c", cells[lane - 1][x]);1178 }1179 printf("\n");1180 }1181}1182 1183main: proc() -> i32 = {1184 arena: memops_arena = {};1185 memops_arena_initialize(arena&);1186 w: kayan_world = {};1187 kayan_world_init(w&, arena&);1188 kayan_register(w&, kayan_id_of<position>(), position<>&);1189 kayan_register(w&, kayan_id_of<velocity>(), velocity<>&);1190 kayan_register(w&, kayan_id_of<health>(), health<>&);1191 kayan_register(w&, kayan_id_of<damage>(), damage<>&);1192 kayan_register(w&, kayan_id_of<lifetime>(), lifetime<>&);1193 kayan_register(w&, kayan_id_of<turret>(), turret<>&);1194 kayan_register(w&, kayan_id_of<burning>(), burning<>&);1195 1196 g: game = {};1197 g.drones = kayan_query_init(kayan_mask_of<position>() | kayan_mask_of<health>(), 0);1198 everything: kayan_query = kayan_query_init(kayan_mask_of<position>(), 0);1199 for (lane: u32 = 0; lane < lanes; lane += 1) {1200 _ = spawn_turret(w&, lane);1201 }1202 1203 // The schedule. Each system reads what the ones before it wrote: bullets1204 // hit where they are after moving, a drone catches fire and is destroyed1205 // in the frame its health drops, and only a drone that survived the frame1206 // can reach the base.1207 shooting: u64 = kayan_mask_of<position>() | kayan_mask_of<turret>();1208 bullets: u64 = kayan_mask_of<position>() | kayan_mask_of<damage>();1209 movers: u64 = kayan_mask_of<position>() | kayan_mask_of<velocity>();1210 systems: [9]kayan_system = {};1211 systems[0] = kayan_system_init("turret", shooting, 0, turret_system, g&);1212 systems[1] = kayan_system_init("age", kayan_mask_of<lifetime>(), 0, age_system, null);1213 systems[2] = kayan_system_init("expire", kayan_mask_of<lifetime>(), 0, expire_system, null);1214 systems[3] = kayan_system_init("move", movers, 0, move_system, null);1215 systems[4] = kayan_system_init("hit", bullets, 0, hit_system, g&);1216 systems[5] = kayan_system_init("ignite", kayan_mask_of<health>(), kayan_mask_of<burning>(), ignite_system, g&);1217 systems[6] = kayan_system_init("burn", kayan_mask_of<health>() | kayan_mask_of<burning>(), 0, burn_system, null);1218 systems[7] = kayan_system_init("die", kayan_mask_of<health>(), 0, die_system, g&);1219 systems[8] = kayan_system_init("breach", kayan_mask_of<position>() | kayan_mask_of<health>(), 0, breach_system, g&);1220 1221 dt: f32 = 0.0625;1222 frames: u32 = 512;1223 problems: u64 = 0;1224 most: u64 = 0;1225 for (frame: u32 = 0; frame < frames; frame += 1) {1226 g.time = cast(frame, f32) * dt;1227 if (frame % 32 == 0 && frame / 32 < 8) {1228 send_wave(w&, g&, frame / 32);1229 }1230 kayan_progress(w&, systems, dt);1231 problems += kayan_verify(w&);1232 most = w.live > most ? w.live : most;1233 if (frame % 64 == 63) {1234 g.time = cast(frame + 1, f32) * dt;1235 draw(w&, g&, everything&);1236 }1237 }1238 1239 kayan_query_update(w&, g.drones&);1240 flying: u64 = kayan_query_count(w&, g.drones&);1241 printf("\n%.2f seconds, %u frames; kayan_verify after every one: %u problems\n",1242 cast(cast(frames, f32) * dt, f64), frames, cast(problems, u32));1243 printf("drones: %u came, %u destroyed, %u reached the base, %u still flying\n", g.came, g.kills, g.breaches, cast(flying, u32));1244 printf("bullets: %u fired, %u hits; fires started: %u\n", g.fired, g.hits, g.fires);1245 printf("entities: at most %u alive at once; commands: %u applied, %u skipped\n",1246 cast(most, u32), cast(w.applied, u32), cast(w.skipped, u32));1247 printfmt("\nthe schedule, and what each system ran over:\n");1248 for (s: u64 = 0; s < 9; s += 1) {1249 printf(" %-6s called %4u times, over %5u rows\n", systems[s].name, cast(systems[s].tables, u32), cast(systems[s].rows, u32));1250 }1251 printfmt("\nthe tables at the end:\n");1252 kayan_print_tables(w&);1253 return 0;1254}What game prints. tools/check.py fails the chapter if a single byte differs.
0.00s wave 1: 3:1, 2 units a second
2.00s wave 2: 7:1 8:1, 2 units a second
4.00s 12 entities in 4 tables
2 T......-...........-........d....
1 T..........-...........-*........
0 T......-...........-........d....
4.00s wave 3: 9:2 10:2 14:1, 2 units a second
4.00s 3:1 destroyed
5.75s 8:1 destroyed
5.75s 7:1 destroyed
6.00s wave 4: 7:2 8:2 3:3, 2 units a second
7.12s 10:2 destroyed
8.00s 15 entities in 4 tables
2 T..........-...........-d........
1 T..-...........-...........-d.d.d
0 T..........-...........-d........
8.00s wave 5: 13:3 16:2 17:1, 4 units a second
8.62s 14:1 destroyed
8.62s 9:2 destroyed
10.00s wave 6: 18:2 19:2 6:5 11:4, 4 units a second
10.00s 7:2 destroyed
11.38s 17:1 destroyed
11.38s 13:3 destroyed
11.81s 16:2 destroyed
12.00s 14 entities in 4 tables
2 T..-...........-........d.d......
1 T......-..............d.d........
0 T..-...........-........d.d......
12.00s wave 7: 16:3 5:7 14:4 15:4 7:3 4:6, 2 units a second
14.00s wave 8: 12:7 20:2 21:1 22:1 23:1 24:1, 4 units a second
14.19s 6:5 destroyed
14.19s 18:2 destroyed
14.75s 8:2 destroyed
16.00s 21 entities in 4 tables
2 T......-..d.............d.d......
1 T..........-....d.......d.d......
0 T......-..d.............d.d......
16.69s 11:4 destroyed
16.69s 19:2 destroyed
17.56s 3:3 destroyed
19.81s 23:1 destroyed
19.81s 12:7 destroyed
20.00s 14 entities in 4 tables
2 T.........d.....d.d..............
1 T..-....*.d.....d.d..............
0 T.........d.....d.d..............
20.19s 21:1 destroyed
22.31s 20:2 destroyed
22.31s 24:1 destroyed
22.44s 22:1 reached the base in lane 1
24.00s 12 entities in 4 tables
2 T..-....d.d......................
1 T......-d.d......................
0 T..-....d.d......................
25.31s 14:4 destroyed
25.56s 16:3 destroyed
25.56s 7:3 destroyed
28.00s 6 entities in 4 tables
2 T.*..............................
1 T.*..............................
0 T.*..............................
28.12s 15:4 destroyed
28.31s 4:6 destroyed
28.31s 5:7 destroyed
32.00s 3 entities in 4 tables
2 T................................
1 T................................
0 T................................
32.00 seconds, 512 frames; kayan_verify after every one: 0 problems
drones: 28 came, 27 destroyed, 1 reached the base, 0 still flying
bullets: 110 fired, 111 hits; fires started: 28
entities: at most 27 alive at once; commands: 716 applied, 1 skipped
the schedule, and what each system ran over:
turret called 512 times, over 1536 rows
age called 409 times, over 1622 rows
expire called 409 times, over 1622 rows
move called 1018 times, over 5051 rows
hit called 409 times, over 1622 rows
ignite called 443 times, over 3145 rows
burn called 166 times, over 284 rows
die called 609 times, over 3429 rows
breach called 609 times, over 3429 rows
the tables at the end:
table 0 {position turret}: 0:1 1:1 2:1
table 1 {position velocity health}:
table 2 {position velocity damage lifetime}:
table 3 {position velocity health burning}:
The dojo · thirteen building blocks
The dojo
dojo/README.md
An ECS is mostly a few small building blocks put together. None of them is specific to an ECS: a renderer's texture handles are a slot map, a particle system is swap-remove over packed arrays, undo and network replication are command buffers. The course meets each block for the first time in the middle of building something bigger, which is why its chapters can feel dense. The dojo takes the blocks out and gives each one two halves:
- a playground, to understand it: a small program that runs the block step by step and draws its insides, with one switch at the top that breaks it the way it most often breaks. Step through it in a debugger, flip the switch, watch what goes wrong.
- a kata, to own it: you write the block yourself, and a harness checks it. Reading and stepping through code feels like learning, but writing it from a blank page, getting it wrong, and doing it again a few days later is what makes it yours.
| block | what it is | in the course | |
|---|---|---|---|
| 1 | free list and generations | handles that are safe to keep after what they name is gone | 1.2, 1.3 |
| 2 | swap-remove and the back-reference | values packed with no holes, found by id: a sparse set | 1.4, 2.3, 2.6 |
| 3 | the slot map | 1 and 2 together: handles into packed values | 1.4, 2.4 |
| 4 | a type-erased column | an array of any type as bytes, viewed as a typed slice | 2.2, 2.3, 2.5 |
| 5 | a command buffer | structural changes recorded now and applied later | 3.3, 3.4 |
| 6 | bitsets | a set of components in a u64: has, all of, none of | 2.1, 3.1 |
| 7 | an arena | push, push, clear: memory for things that die together | every chapter |
| 8 | index, not pointer | links that stay right when their storage moves | 2.2, 2.4, 2.5 |
| 9 | interning | find or make, by key: a hash table that hands out dense ids | 2.4 |
| 10 | memoized edges | a slow lookup done once, then remembered on the edge | 2.6 |
| 11 | a reflection walk | reading, writing and comparing any value through its type's record | 2.7 |
| 12 | a watermark | looking only at what was added since last time | 3.1 |
| 13 | a dispatch table | behaviour as data: the schedule as a list of functions | 3.2 |
Do 1 to 5 in order the first time: 3 is 1 and 2 put together, and 5 applies its commands to something like 3. 6 to 13 come in the order the course meets them and stand on their own, except that 9, 10 and 12 work on 6's signatures. Any of them can come before the course, alongside it, or after it.
Each block
dojo/3.slot_map/
README.md the block, its invariant, the levels, the variations
play_slot_map.rin the playground
harness.rin fixed: a scenario, then a fuzz run against a model
level1.rin ... 3.rin where a kata starts
solution.rin the reference; don't peek
kata.rin yours: made by `start`, ignored by git
The harness doesn't know how your kata works, only what it must do. First it runs a short scenario, the playground's steps, and prints each as it passes. Then it runs 20,000 random operations from a seed against a simple model of the right answers, and checks after each one. A failure says which operation, which seed, and how to stop just before it in a debugger.
Levels, so the reps get harder:
- One function in an otherwise finished file: the one with the idea in it, usually the one people forget.
- Every function body, with the fields and signatures given.
- A blank page, with only a list of what the harness calls.
- Variations: the same harness, different insides, listed in each block's README.
Each README gives a target time for levels 1 to 3.
Watching a playground
python tools/dojo.py play slot_map
A playground runs one scenario, one operation per line of main, and prints what changed after each, so it works without a debugger. In one, put a breakpoint on a line of main and step over the operations. Where the block has slots there are only a few of them, in fixed arrays, so the watch window shows the whole state. From block 6 on, a verify proc runs after every operation, checks the block's invariant, and says what broke; a breakpoint inside its failure branch stops at the first operation that breaks it.
The switch is a global b32, read each time it's used, so besides editing it and running again, you can flip it mid-run: stop where the playground's header says, set it to 0 in the watch window, and continue. rin writes C, and the global keeps its name, so in gdb:
(gdb) break handles_free
(gdb) run
(gdb) set var bump_on_free = 0
(gdb) continue
Doing reps
python tools/dojo.py the blocks, your reps, and what is due
python tools/dojo.py play slot_map the playground
python tools/dojo.py start slot_map 2 a fresh kata.rin from level 2; the clock starts
python tools/dojo.py slot_map build the harness with your kata.rin and run it
A block can be named by its number, its name or the start of its name: 3, slot_map and slot are the same. While the kata fails, fix it and run it again. When it passes, the time since start goes into dojo/log.md, and the block is due again 1, 3, 7, 14 and then 30 days after each pass. python tools/dojo.py with no arguments says what is due. Spacing reps out is what makes them stick, much more than doing many in one day.
start won't overwrite a kata.rin you've changed unless you add --force. Your log is a plain markdown table; commit it or don't.
When a kata fails, the harness says where:
FAIL at operation 64 (seed 1): row 4 holds 43's value 37043, but get(43) gives 43020
debug it: build/dojo/swap_remove/harness --seed 1 --stop 64
with a breakpoint on dojo_stop_here, which runs just before that operation
Run that binary under a debugger with a breakpoint on dojo_stop_here. It stops just before the failing operation, with your structure in the state that leads to the failure. If the harness crashes instead, on a vec index past the end say, the runner runs it again with --trace 1 and prints the operation that was running. --seed N reruns a different fuzz run, and --sanitize builds with AddressSanitizer and UBSan, which block 4's column of raw bytes is exactly what they're for.
How it's built
Every kata file is a whole program. Yours, the solution and each level import what they use, then the harness, which has main:
import "std/memops.rin"
import "std/vec.rin"
import "std/option.rin"
import "../dojo.rin"
import "harness.rin"
So any of them builds on its own, rin compile dojo/3.slot_map/solution.rin for instance, and rin check or your editor finds every name it uses. Only the harness doesn't build alone: it calls the procs the kata defines. A file imported twice is read once, so the kata and the harness both importing dojo.rin is fine. dojo/dojo.rin holds what every harness shares: dojo_handle and dojo_status, the run's settings, the random number generator, and the reporting. The runner builds kata.rin the way tools/check.py does, rin compile and then the C compiler, with the warnings for unused parameters and locals off, since a kata in progress is full of them.
python tools/check.py dojo keeps the dojo honest: every solution must build on its own, pass its harness and print its harness.expected, level 1 must fail (its gap must be a gap), level 2 must compile, and the playgrounds must print their .expected. The harnesses were also checked against broken solutions, the mistakes each block is about, and catch each one.
The same dojo exists in Rust, in rust-dod. The random number generator is the same, so a seed is the same run in both, and the playgrounds print the same thing, except where block 4's column gets its memory: an arena in rin, a Vec in Rust. The katas differ where the languages do: block 7's arena hands out offsets in Rust and addresses in rin, and in block 8 only rin can keep a pointer across a move, so only rin's switch does; each README says how.
The dojo · block 1
Free list and generations
dojo/1.free_list/play_free_list.rin
Handles name things that come and go: entities, textures, sounds, network connections. A handle is two numbers, index:generation, and two small ideas make it safe to keep:
- A free list. A freed index is reused before a new one is made, so the arrays an index points into stay as long as the most things ever alive at once, not the most things ever made.
- A generation per index. Freeing bumps the index's generation, so every handle made for the old life stops matching. A reused index can't be mistaken for whatever had it before.
The invariant: a handle is alive exactly when its generation is the one its index holds now. A free index already holds its next generation, which no handle has yet, so "free" needs no flag of its own. Every index is either alive or on the free list, once. Generations start at 1, so a zeroed handle, {}, is never alive.
In the course: 1.2 builds the free list and meets its bug, 1.3 adds generations, and every later chapter keeps both.
Watch it: the playground
python tools/dojo.py play free_list
It allocates four handles, frees two, reuses one, and draws both arrays after each step:
free b, then d: their generations go up now, before anyone reuses them
generations 0:1 1:2 2:1 3:2
free list [1 3]
alloc e: alloc takes from the end of the free list, so index 3, in its next life
e 3:2
Step through it: break on handles_alloc and handles_free, and watch generations and free change. Then flip the switch at the top, bump_on_free, to 0 and run it again. The freed b is still alive, the reused index comes back with the old generation so d and e are the same handle, and freeing d again frees e and puts index 3 on the free list twice. That's the bug from 1.2, in four lines of output.
Do reps: the kata
You write handles, an allocator of handles:
handles_alloc: proc(h: *handles) -> dojo_handle = {
handles_free: proc(h: *handles, e: dojo_handle) -> dojo_status = {
handles_alive: proc(h: *handles, e: dojo_handle) -> b32 = {
plus handles_init, handles_live (how many are alive) and handles_slots (how many indices have been handed out). dojo_handle and dojo_status are given, in dojo/dojo.rin, and the arrays are vec<u32>s from the arena handles_init is given.
| level | you write | target |
|---|---|---|
| 1 | handles_free, in an otherwise finished file | 2:00 |
| 2 | every proc body; the fields and signatures are given | 8:00 |
| 3 | the whole file, from a blank page | 10:00 |
python tools/dojo.py start free_list 2 kata.rin from level 2, and the clock starts
python tools/dojo.py free_list your kata.rin against the harness
The harness first runs a short scenario, the playground's steps, and prints each as it passes. Then it runs 20,000 random allocs and frees from a seed, against a model that remembers, per index, the last generation handed out and whether the index is free. It checks what your allocator does, not how: a generation is never handed out twice for an index, a freed index is reused before a new one is made, and handles_alive agrees with the model for every handle, live and dead. The seeds give the same runs as rust-dod's dojo.
Level 4, variations. The same harness, different insides:
- Keep the free list inside the dead slots: no
freevec, but anext_freefield in each slot that is only meaningful while the slot is free, and afirst_freeinhandles. What marks the end of the list? - 16-bit generations in the slots, stored as
u16, that wrap from 65535 back to 1, never 0. What does wrapping give up? - Reuse in first-in, first-out order, so an index rests as long as possible before it comes back. Why might a game want that?
When it fails
FAIL at operation 266 (seed 1): after free, alive() is still true for 43:1
debug it: build/dojo/free_list/harness --seed 1 --stop 266
with a breakpoint on dojo_stop_here, which runs just before that operation
python tools/dojo.py free_list built that harness with your kata. Run it under your debugger with those arguments and a breakpoint on dojo_stop_here (gdb --args build/dojo/free_list/harness --seed 1 --stop 266, then break dojo_stop_here, run). It stops just before the operation that fails; step out and into your code.
If the harness crashes instead, on an index past the end of a vec say, the runner runs it again with --trace 1 and prints the operation that was running.
The playground. The kata's harness is dojo/1.free_list/harness.rin, and the levels start from dojo/1.free_list/level1.rin to level3.rin.
1// Dojo 1, the playground: a free list and generations.2//3// Handles name things that come and go: entities, textures, sounds, network4// connections. Two small ideas make them safe to keep:5//6// * a free list: freed indices are reused before new ones are made, so the7// arrays an index points into stay as small as the most things alive;8// * a generation per index: freeing bumps it, so every handle made for the9// old life stops matching, and a reused index can't be mistaken for the10// thing that had it before.11//12// This program runs the allocator step by step and draws its two arrays after13// each step. To see it in a debugger, break on `handles_alloc` and14// `handles_free` and watch `generations` and `free` change.15//16// The switch: set bump_on_free to 0 and run it again. The reused index comes17// back with the old generation, and a dead handle is alive again. It's a18// global, read each time it's used, so a debugger can flip it mid-run too:19// stop on handles_free, set it to 0 in the watch window, and continue.20 21import "std/print.rin"22 23bump_on_free: b32 = 1;24 25handle: struct = {26 index: u32;27 generation: u32;28}29 30handles: struct = {31 // The life each index is in now, or will be in next if it is free.32 generations: [8]u32;33 count: u32;34 // Freed indices, a stack: the last one freed is the first one reused.35 free: [8]u32;36 free_count: u32;37}38 39handles_alloc: proc(h: *handles) -> handle = {40 index: u32 = 0;41 if (h*.free_count > 0) {42 h*.free_count -= 1;43 index = h*.free[h*.free_count];44 } else {45 index = h*.count;46 h*.generations[index] = 1;47 h*.count += 1;48 }49 return { .index = index, .generation = h*.generations[index] };50}51 52handles_alive: proc(h: *handles, e: handle) -> b32 = {53 return e.index < h*.count && h*.generations[e.index] == e.generation;54}55 56handles_free: proc(h: *handles, e: handle) -> b32 = {57 if (!handles_alive(h, e)) {58 return 0;59 }60 if (bump_on_free) {61 h*.generations[e.index] += 1;62 }63 h*.free[h*.free_count] = e.index;64 h*.free_count += 1;65 return 1;66}67 68// Each index with the generation it holds, then the free list.69draw: proc(h: *handles) -> void = {70 printfmt(" generations");71 for (i: u32 = 0; i < h*.count; i += 1) {72 printfmt(" {}:{}", i, h*.generations[i]);73 }74 printfmt("\n free list [");75 for (i: u32 = 0; i < h*.free_count; i += 1) {76 if (i > 0) {77 printfmt(" ");78 }79 printfmt("{}", h*.free[i]);80 }81 printfmt("]\n");82}83 84says: proc(alive: b32) -> ^const char = {85 return alive ? "alive" : "dead";86}87 88main: proc() -> i32 = {89 h: handles = {};90 printfmt("alloc a, b, c, d: four new indices, each in its first life\n");91 a: handle = handles_alloc(h&);92 b: handle = handles_alloc(h&);93 c: handle = handles_alloc(h&);94 d: handle = handles_alloc(h&);95 printfmt(" a {}:{} b {}:{} c {}:{} d {}:{}\n", a.index, a.generation, b.index, b.generation,96 c.index, c.generation, d.index, d.generation);97 draw(h&);98 99 printfmt("free b, then d: their generations go up now, before anyone reuses them\n");100 _ = handles_free(h&, b);101 _ = handles_free(h&, d);102 draw(h&);103 104 printfmt("alloc e: alloc takes from the end of the free list, so index 3, in its next life\n");105 e: handle = handles_alloc(h&);106 printfmt(" e {}:{}\n", e.index, e.generation);107 draw(h&);108 109 printfmt("the old handles\n");110 printfmt(" d {}:{} is {}\n", d.index, d.generation, says(handles_alive(h&, d)));111 printfmt(" e {}:{} is {}\n", e.index, e.generation, says(handles_alive(h&, e)));112 printfmt(" b {}:{} is {}\n", b.index, b.generation, says(handles_alive(h&, b)));113 if (d.index == e.index && d.generation == e.generation) {114 printfmt(" d and e are the same handle: whoever kept d now reaches e\n");115 } else {116 printfmt(" d and e share index {} but not a generation, so d can never reach e\n", e.index);117 }118 119 printfmt("free d again\n");120 if (handles_free(h&, d)) {121 printfmt(" freed: d named e, so e is gone\n");122 } else {123 printfmt(" refused: d is dead, nothing changes\n");124 }125 draw(h&);126 127 printfmt("alloc f, g: index 1 first, then a new one\n");128 f: handle = handles_alloc(h&);129 g: handle = handles_alloc(h&);130 printfmt(" f {}:{} g {}:{}\n", f.index, f.generation, g.index, g.generation);131 draw(h&);132 printfmt(" c {}:{} was never freed: still {}\n", c.index, c.generation, says(handles_alive(h&, c)));133 return 0;134}What play_free_list prints. tools/check.py fails the playground if a single byte differs.
alloc a, b, c, d: four new indices, each in its first life
a 0:1 b 1:1 c 2:1 d 3:1
generations 0:1 1:1 2:1 3:1
free list []
free b, then d: their generations go up now, before anyone reuses them
generations 0:1 1:2 2:1 3:2
free list [1 3]
alloc e: alloc takes from the end of the free list, so index 3, in its next life
e 3:2
generations 0:1 1:2 2:1 3:2
free list [1]
the old handles
d 3:1 is dead
e 3:2 is alive
b 1:1 is dead
d and e share index 3 but not a generation, so d can never reach e
free d again
refused: d is dead, nothing changes
generations 0:1 1:2 2:1 3:2
free list [1]
alloc f, g: index 1 first, then a new one
f 1:2 g 4:1
generations 0:1 1:2 2:1 3:2 4:1
free list []
c 2:1 was never freed: still alive
The dojo · block 2
Swap-remove and the back-reference
dojo/2.swap_remove/play_swap_remove.rin
Systems loop over values, so values are kept packed: one after another, no holes, nothing to skip. Removing one from the middle would leave a hole, so the last value moves into it instead. That's swap-remove: one copy, however many values there are, at the price of order.
Values move, so an id can't be the position of its value. Two arrays keep track:
- sparse, by id:
rows[id]says which row id's value is in now; - packed, by row:
ids[row]is the id that owns the row. This is the back-reference: when the last row moves into a hole, it says whose entry inrowsto fix.
The whole structure is what EnTT calls a sparse set.
The invariant: the rows are packed, and the two arrays point at each other: for every row, rows[ids[row]] names that row, and an id with no value has no row.
In the course: 1.4 packs entities into rows this way, 2.3 does it for every column of a table, and 2.6 does it when an entity moves between tables.
Watch it: the playground
python tools/dojo.py play swap_remove
It inserts four values, removes one from the middle, and draws both sides after each step:
remove 7, in row 1: the last row, 5's, moves into the hole
row 0 1 2
ids 3 5 1
values 30 50 10
rows 1->2 3->0 5->1 (id -> row)
Step through it: break on sparse_set_remove, and watch ids and rows as the last row moves. Then flip the switch at the top, fix_back_reference, to 0 and run it again. Id 5's row still says 3, past the end, so get 5 finds nothing. Then insert 9 puts 9's value in row 3, and get 5 returns 90, another id's value, with nothing to say it's wrong. That's the bug every packed array invites, and the reason the back-reference exists.
Do reps: the kata
You write sparse_set. The heart of it is sparse_set_remove:
sparse_set_remove: proc(s: *sparse_set, id: u32) -> option<u64> = {
if (sparse_set_get(s, id) == null) {
return option<u64>none();
}
entry: *u32 = vec<u32>at(s*.rows&, cast(id, u64));
row: u64 = cast(entry* - 1, u64);
entry* = 0;
value: u64 = vec<u64>at(s*.values&, row)*;
last_value: u64 = vec<u64>pop(s*.values&);
last_id: u32 = vec<u32>pop(s*.ids&);
if (row < s*.values.length) {
vec<u64>at(s*.values&, row)* = last_value;
vec<u32>at(s*.ids&, row)* = last_id;
vec<u32>at(s*.rows&, cast(last_id, u64))* = cast(row, u32) + 1;
}
return option<u64>some(value);
}
That's the reference, so don't read it before level 1 if you'd rather not know. vec has no swap-remove of its own: the last element is popped, and written over the hole unless the hole was the last row. The full API:
sparse_set_insert: proc(s: *sparse_set, id: u32, value: u64) -> option<u64> = {
sparse_set_get: proc(s: *sparse_set, id: u32) -> *u64 = {
plus sparse_set_init, sparse_set_len, sparse_set_values (the packed values, in row order) and sparse_set_ids (ids[row] owns values[row]).
| level | you write | target |
|---|---|---|
| 1 | sparse_set_remove, in an otherwise finished file | 2:00 |
| 2 | every proc body; the fields and signatures are given | 8:00 |
| 3 | the whole file, from a blank page | 10:00 |
python tools/dojo.py start swap_remove 2
python tools/dojo.py swap_remove
The harness runs the playground's steps as a scenario, then 20,000 random inserts and removes over a small range of ids, so the same ids come and go many times, against a plain array from id to value. Every 64 operations it checks the packed rows themselves: as many as the model has, and each row's owner holding exactly that row's value. The seeds give the same runs as rust-dod's dojo.
rin notes. Where Rust's kata returns Option<T> and has get_mut, this one uses std's option<u64> for what insert and remove give back (the old value, or none), and one sparse_set_get that returns a pointer to the value, null for none, to read or to change, the way kayan_get<T> does in the course. rows[id] holds the row plus one, so a new entry, 0, means "no row", and growing rows is appending zeros.
Level 4, variations. The same harness, different insides:
- Pages:
rowssplit into pages of 64 ids, each page made only when an id in it gets a value: avec<^u32>of pages, null for a page not made yet. A big id no longer means a bigrows. This is what EnTT does. - A sentinel:
rows[id]holds the row itself, with0xFFFFFFFFfor "no row". What does growingrowscost now? - A flag: a second
vec<b32>,has_row, besiderows. Compare the memory with the plus-one encoding.
And one that fails on purpose: a remove that keeps order, shifting every later row down and fixing each one's back-reference. The scenario stops it at the first ids line. Why does the harness insist on swap-remove's order, and what does keeping order cost?
When it fails
A forgotten back-reference often shows up as a crash in a sparse_set_get that is right, not in the sparse_set_remove that is wrong: get follows the stale row past the end, and vec<u64>at stops the program.
rin runtime: vec index out of bounds
the harness crashed in the scenario, after the last step printed above
The last step printed is the last that passed, so the bug is in what ran after it. In the fuzz run, the runner runs the harness again with --trace 1 and prints the operation that was running when it crashed. A failed check gives the operation and the seed:
FAIL at operation 64 (seed 1): row 4 holds 43's value 37043, but get(43) gives 43020
debug it: build/dojo/swap_remove/harness --seed 1 --stop 64
with a breakpoint on dojo_stop_here, which runs just before that operation
Run the harness under your debugger with those arguments and a breakpoint on dojo_stop_here (gdb --args build/dojo/swap_remove/harness --seed 1 --stop 64, then break dojo_stop_here, run). It stops just before the operation that fails; step out and into your code.
The playground. The kata's harness is dojo/2.swap_remove/harness.rin, and the levels start from dojo/2.swap_remove/level1.rin to level3.rin.
1// Dojo 2, the playground: swap-remove and the back-reference.2//3// Systems loop over values, so values are kept packed: one after another, no4// holes, nothing to skip. Removing one from the middle would leave a hole, so5// the last value moves into it instead. That's swap-remove: one copy, however6// many values there are. The price is that values move, so an id can't be the7// position of its value. Two arrays keep track:8//9// * sparse, by id: rows[id] is the row id's value is in now;10// * packed, by row: ids[row] is the id that owns the row, the11// back-reference. When the last row moves, it says whose entry in `rows`12// to fix.13//14// This is what EnTT calls a sparse set, and what the ECS course's tables do15// in 1.4 and 2.3.16//17// This program inserts four values, removes one from the middle, and draws18// both sides after each step. To see it in a debugger, break on19// `sparse_set_remove` and watch `ids` and `rows` as the last row moves.20//21// The switch: set fix_back_reference to 0 and run it again. The moved id's row22// still points where its value used to be, and the next insert writes another23// id's value there. It's a global, read each time it's used, so a debugger can24// flip it mid-run too: stop on sparse_set_remove, set it to 0 in the watch25// window, and continue.26 27import "std/print.rin"28 29fix_back_reference: b32 = 1;30 31sparse_set: struct = {32 values: [8]u64;33 ids: [8]u32;34 count: u32;35 // rows[id] is one more than id's row, so 0 means id has no value.36 rows: [16]u32;37}38 39sparse_set_insert: proc(s: *sparse_set, id: u32, value: u64) -> void = {40 s*.rows[id] = s*.count + 1;41 s*.values[s*.count] = value;42 s*.ids[s*.count] = id;43 s*.count += 1;44}45 46sparse_set_remove: proc(s: *sparse_set, id: u32) -> void = {47 if (s*.rows[id] == 0) {48 return;49 }50 row: u32 = s*.rows[id] - 1;51 s*.rows[id] = 0;52 last: u32 = s*.count - 1;53 s*.values[row] = s*.values[last];54 s*.ids[row] = s*.ids[last];55 s*.count = last;56 if (row < last && fix_back_reference) {57 s*.rows[s*.ids[row]] = row + 1;58 }59}60 61// id's value through its row, or 0 for nothing. A row past the end reads62// nothing; a row that is wrong but in range reads someone else's value.63sparse_set_get: proc(s: *sparse_set, id: u32) -> u64 = {64 if (s*.rows[id] == 0 || s*.rows[id] - 1 >= s*.count) {65 return 0;66 }67 return s*.values[s*.rows[id] - 1];68}69 70draw: proc(s: *sparse_set) -> void = {71 printfmt(" row ");72 for (r: u32 = 0; r < s*.count; r += 1) {73 printf("%5u", r);74 }75 printfmt("\n ids ");76 for (r: u32 = 0; r < s*.count; r += 1) {77 printf("%5u", s*.ids[r]);78 }79 printfmt("\n values ");80 for (r: u32 = 0; r < s*.count; r += 1) {81 printf("%5u", cast(s*.values[r], u32));82 }83 printfmt("\n rows ");84 first: b32 = 1;85 for (id: u32 = 0; id < 16; id += 1) {86 if (s*.rows[id] != 0) {87 if (!first) {88 printfmt(" ");89 }90 printfmt("{}->{}", id, s*.rows[id] - 1);91 first = 0;92 }93 }94 printfmt(" (id -> row)\n");95}96 97read: proc(s: *sparse_set, id: u32) -> void = {98 value: u64 = sparse_set_get(s, id);99 if (value == 0) {100 printfmt("nothing");101 } else {102 printfmt("{}", value);103 }104}105 106main: proc() -> i32 = {107 s: sparse_set = {};108 printfmt("insert 3, 7, 1, 5, each with a value of ten times its id\n");109 ids: [4]u32 = { 3, 7, 1, 5 };110 for (i: u32 = 0; i < 4; i += 1) {111 sparse_set_insert(s&, ids[i], cast(ids[i], u64) * 10);112 }113 draw(s&);114 115 printfmt("remove 7, in row 1: the last row, 5's, moves into the hole\n");116 sparse_set_remove(s&, 7);117 draw(s&);118 printfmt(" get 5: ");119 read(s&, 5);120 printfmt("\n get 7: ");121 read(s&, 7);122 printfmt("\n");123 124 printfmt("insert 9: a new row at the end\n");125 sparse_set_insert(s&, 9, 90);126 draw(s&);127 printfmt(" get 5: ");128 read(s&, 5);129 printfmt("\n get 9: ");130 read(s&, 9);131 printfmt("\n");132 133 printfmt("remove 3, in row 0: the last row, 9's, moves into it\n");134 sparse_set_remove(s&, 3);135 draw(s&);136 printfmt(" get 9: ");137 read(s&, 9);138 printfmt(" get 5: ");139 read(s&, 5);140 printfmt(" get 1: ");141 read(s&, 1);142 printfmt("\n");143 return 0;144}What play_swap_remove prints. tools/check.py fails the playground if a single byte differs.
insert 3, 7, 1, 5, each with a value of ten times its id
row 0 1 2 3
ids 3 7 1 5
values 30 70 10 50
rows 1->2 3->0 5->3 7->1 (id -> row)
remove 7, in row 1: the last row, 5's, moves into the hole
row 0 1 2
ids 3 5 1
values 30 50 10
rows 1->2 3->0 5->1 (id -> row)
get 5: 50
get 7: nothing
insert 9: a new row at the end
row 0 1 2 3
ids 3 5 1 9
values 30 50 10 90
rows 1->2 3->0 5->1 9->3 (id -> row)
get 5: 50
get 9: 90
remove 3, in row 0: the last row, 9's, moves into it
row 0 1 2
ids 9 5 1
values 90 50 10
rows 1->2 5->1 9->0 (id -> row)
get 9: 90 get 5: 50 get 1: 10
The dojo · block 3
The slot map
dojo/3.slot_map/play_slot_map.rin
Dojos 1 and 2, put together. Values live packed in rows, so a loop over them has no holes (dojo 2). The rest of the program keeps handles, which survive rows moving and say when their value is gone (dojo 1). In between is a slot per index:
handle 1:2 -> slots[1] { generation 2, row 0 } -> values[0]
A lookup is two steps: the handle's generation must be the slot's, and the slot says the row. A remove is both dojos' work: bump the generation and free the index, then swap-remove the row and fix the slot of whoever moved. Game engines keep textures, meshes and sounds this way, and it is the ECS's entity index.
The invariant: dojo 1's and dojo 2's together. A handle is alive exactly when its generation is its slot's; the rows are packed; and for every row, the slot of the row's handle points back at that row.
In the course: 1.4 is a slot map with an entity's components as its value, and every chapter after keeps it, with a table number added to the slot in 2.4.
Watch it: the playground
python tools/dojo.py play slot_map
It keeps three textures, removes one, reuses its index, and draws the slots and the rows after each step:
remove stone: free index 1 in its next life, and water moves into row 1
slots 0: generation 1, row 0 | 1: generation 2, free | 2: generation 1, row 1
rows 0:1 grass | 2:1 water
Step through it: break on slot_map_remove, and watch the slot, the free list and the rows change in that order. Then flip the switch at the top, check_generation, to 0 and run it again. The removed stone still finds slot 1, and reads whatever is in slot 1's row now: water, then lava. In a game, that's a sound that plays the wrong sample, or a mesh drawn with another mesh's texture.
Do reps: the kata
You write slot_map. Its two-step lookup:
slot_map_get: proc(m: *slot_map, h: dojo_handle) -> *u64 = {
if (!slot_map_contains(m, h)) {
return null;
}
row: u32 = vec<slot_map_slot>at(m*.slots&, cast(h.index, u64))*.row;
return vec<u64>at(m*.values&, cast(row, u64));
}
and the API around it:
slot_map_insert: proc(m: *slot_map, value: u64) -> dojo_handle = {
slot_map_remove: proc(m: *slot_map, h: dojo_handle) -> option<u64> = {
plus slot_map_init, slot_map_contains, slot_map_len, slot_map_values (the packed values, in row order) and slot_map_handles (handles[row] owns values[row]). dojo_handle comes from dojo/dojo.rin, as in dojo 1.
| level | you write | target |
|---|---|---|
| 1 | slot_map_remove, in an otherwise finished file | 3:00 |
| 2 | every proc body; the fields and signatures are given | 12:00 |
| 3 | the whole file, from a blank page | 15:00 |
python tools/dojo.py start slot_map 2
python tools/dojo.py slot_map
Level 3 is the one to come back to. Once you can write a slot map from a blank page in fifteen minutes, dojos 1 and 2 are in your hands too.
The harness runs the playground's steps as a scenario, then 20,000 random inserts, removes, reads and writes against a model that keeps, per index, the last generation handed out, whether the index is free, and its value. It checks dojo 1's rules for the handles (a generation never handed out twice for an index, a freed index reused before a new one is made, a removed handle never alive again) and dojo 2's for the rows (packed, each row's handle holding that row's value). The seeds give the same runs as rust-dod's dojo.
rin notes. As in dojo 2, slot_map_remove gives back std's option<u64> and slot_map_get returns a pointer, null for a dead handle, to read or to change the value; Rust's kata has get and get_mut. A zeroed handle, {}, is never alive, since generations start at 1, and the scenario checks that too.
Level 4, variations. The same harness, different insides:
- Free list inside the slots: a
next_free: u32in each slot, only meaningful while the slot is free, and afirst_freeinslot_map. Nofreevec. What marks the end of the list? - One look at the slot:
slot_map_removewithoutslot_map_contains: fetch the slot once, check its generation, and use it. How many times does the reference look the slot up? - **
slot_map_retain(m, keep)**, withkeepaproc(h: dojo_handle, value: u64) -> b32: remove every value it rejects, in one pass, keeping the rows packed. Which way should the loop walk? (Not checked by the harness; write your own test.)
When it fails
FAIL at operation 233 (seed 1): remove(33:1) returned 214, the model says 210
debug it: build/dojo/slot_map/harness --seed 1 --stop 233
with a breakpoint on dojo_stop_here, which runs just before that operation
The wrong value usually means a slot pointing at the wrong row: an earlier remove moved a row and didn't fix its slot. Run the harness under your debugger with those arguments and a breakpoint on dojo_stop_here, which runs just before that operation, then look at the slot of the handle being removed.
If the harness crashes instead, on a row past the end of values say, the runner runs it again with --trace 1 and prints the operation that was running, or says the crash came in the scenario, after the last step printed.
The playground. The kata's harness is dojo/3.slot_map/harness.rin, and the levels start from dojo/3.slot_map/level1.rin to level3.rin.
1// Dojo 3, the playground: a slot map.2//3// Dojos 1 and 2, put together. Values live packed in rows, so a loop over them4// has no holes (dojo 2). The rest of the program keeps handles, which survive5// rows moving and say when their value is gone (dojo 1). In between is a slot6// per index:7//8// handle 1:2 -> slots[1] { generation 2, row 0 } -> values[0]9//10// A lookup is two steps: the handle's generation must be the slot's, and the11// slot says the row. A remove is both dojos' work: bump the generation and12// free the index, then swap-remove the row and fix the slot of whoever moved.13// This is the ECS course's entity index (1.4), with one value per entity, and14// what game engines use for textures, meshes and sounds.15//16// This program inserts three values, removes one, reuses its index, and17// draws the slots and the rows after each step. To see it in a debugger,18// break on `slot_map_remove` and watch `slots`, `handles` and `values`.19//20// The switch: set check_generation to 0 and run it again. A removed handle21// still finds a slot, and reads whatever value is in that slot's row now. It's22// a global, read each time it's used, so a debugger can flip it mid-run too:23// stop on slot_map_remove, set it to 0 in the watch window, and continue.24 25import "std/print.rin"26 27check_generation: b32 = 1;28 29handle: struct = {30 index: u32;31 generation: u32;32}33 34slot: struct = {35 generation: u32;36 row: u32;37}38 39slot_map: struct = {40 slots: [8]slot;41 slot_count: u32;42 free: [8]u32;43 free_count: u32;44 values: [8]^const char;45 handles: [8]handle;46 count: u32;47}48 49slot_map_insert: proc(m: *slot_map, value: ^const char) -> handle = {50 index: u32 = 0;51 if (m*.free_count > 0) {52 m*.free_count -= 1;53 index = m*.free[m*.free_count];54 } else {55 index = m*.slot_count;56 m*.slots[index].generation = 1;57 m*.slot_count += 1;58 }59 m*.slots[index].row = m*.count;60 h: handle = { .index = index, .generation = m*.slots[index].generation };61 m*.values[m*.count] = value;62 m*.handles[m*.count] = h;63 m*.count += 1;64 return h;65}66 67slot_map_contains: proc(m: *slot_map, h: handle) -> b32 = {68 if (h.index >= m*.slot_count) {69 return 0;70 }71 return !check_generation || m*.slots[h.index].generation == h.generation;72}73 74// The value through the handle's slot, or null. A row past the end reads75// nothing.76slot_map_get: proc(m: *slot_map, h: handle) -> ^const char = {77 if (!slot_map_contains(m, h) || m*.slots[h.index].row >= m*.count) {78 return null;79 }80 return m*.values[m*.slots[h.index].row];81}82 83slot_map_remove: proc(m: *slot_map, h: handle) -> void = {84 if (!slot_map_contains(m, h)) {85 return;86 }87 row: u32 = m*.slots[h.index].row;88 m*.slots[h.index].generation += 1;89 m*.free[m*.free_count] = h.index;90 m*.free_count += 1;91 last: u32 = m*.count - 1;92 m*.values[row] = m*.values[last];93 m*.handles[row] = m*.handles[last];94 m*.count = last;95 if (row < last) {96 m*.slots[m*.handles[row].index].row = row;97 }98}99 100is_free: proc(m: *slot_map, index: u32) -> b32 = {101 for (i: u32 = 0; i < m*.free_count; i += 1) {102 if (m*.free[i] == index) {103 return 1;104 }105 }106 return 0;107}108 109draw: proc(m: *slot_map) -> void = {110 printfmt(" slots ");111 for (i: u32 = 0; i < m*.slot_count; i += 1) {112 if (i > 0) {113 printfmt(" | ");114 }115 if (is_free(m, i)) {116 printfmt("{}: generation {}, free", i, m*.slots[i].generation);117 } else {118 printfmt("{}: generation {}, row {}", i, m*.slots[i].generation, m*.slots[i].row);119 }120 }121 printfmt("\n rows ");122 for (r: u32 = 0; r < m*.count; r += 1) {123 if (r > 0) {124 printfmt(" | ");125 }126 printfmt("{}:{} {}", m*.handles[r].index, m*.handles[r].generation, m*.values[r]);127 }128 printfmt("\n");129}130 131read: proc(m: *slot_map, name: ^const char, h: handle) -> void = {132 value: ^const char = slot_map_get(m, h);133 if (value == null) {134 value = "nothing: it is gone";135 }136 printfmt("{} {}:{}: {}", name, h.index, h.generation, value);137}138 139main: proc() -> i32 = {140 m: slot_map = {};141 printfmt("insert three textures\n");142 grass: handle = slot_map_insert(m&, "grass");143 stone: handle = slot_map_insert(m&, "stone");144 water: handle = slot_map_insert(m&, "water");145 printfmt(" grass {}:{} stone {}:{} water {}:{}\n", grass.index, grass.generation,146 stone.index, stone.generation, water.index, water.generation);147 draw(m&);148 149 printfmt("remove stone: free index 1 in its next life, and water moves into row 1\n");150 slot_map_remove(m&, stone);151 draw(m&);152 printfmt(" ");153 read(m&, "water", water);154 printfmt("\n ");155 read(m&, "stone", stone);156 printfmt("\n");157 158 printfmt("insert lava: index 1 again, generation 2, in a new row at the end\n");159 lava: handle = slot_map_insert(m&, "lava");160 printfmt(" lava {}:{}\n", lava.index, lava.generation);161 draw(m&);162 printfmt(" ");163 read(m&, "stone", stone);164 printfmt("\n ");165 read(m&, "lava", lava);166 printfmt("\n");167 168 printfmt("remove grass: lava, in the last row, moves into row 0\n");169 slot_map_remove(m&, grass);170 draw(m&);171 printfmt(" ");172 read(m&, "lava", lava);173 printfmt(" ");174 read(m&, "water", water);175 printfmt(" ");176 read(m&, "grass", grass);177 printfmt("\n");178 return 0;179}What play_slot_map prints. tools/check.py fails the playground if a single byte differs.
insert three textures
grass 0:1 stone 1:1 water 2:1
slots 0: generation 1, row 0 | 1: generation 1, row 1 | 2: generation 1, row 2
rows 0:1 grass | 1:1 stone | 2:1 water
remove stone: free index 1 in its next life, and water moves into row 1
slots 0: generation 1, row 0 | 1: generation 2, free | 2: generation 1, row 1
rows 0:1 grass | 2:1 water
water 2:1: water
stone 1:1: nothing: it is gone
insert lava: index 1 again, generation 2, in a new row at the end
lava 1:2
slots 0: generation 1, row 0 | 1: generation 2, row 2 | 2: generation 1, row 1
rows 0:1 grass | 2:1 water | 1:2 lava
stone 1:1: nothing: it is gone
lava 1:2: lava
remove grass: lava, in the last row, moves into row 0
slots 0: generation 2, free | 1: generation 2, row 0 | 2: generation 1, row 1
rows 1:2 lava | 2:1 water
lava 1:2: lava water 2:1: water grass 0:1: nothing: it is gone
The dojo · block 4
A type-erased column
dojo/4.column/play_column.rin
A column is one component's rows, kept as bytes. It doesn't know the type it holds, only how many bytes make a row and how they must be aligned. That is enough to grow, to zero a new row, and to move the last row into a hole, which is everything a table does to its columns. The type comes back only at the edge: a typed view turns the bytes into a []T, for a T of the column's size and alignment.
Two things make it work:
- Aligned storage. The bytes come from the arena aligned to the column's alignment, so every row of the type lands where that type may live.
- Growing by a factor. A full column doubles, so the copies that growing costs add up to about one per push, however many rows there are.
The invariant: the column holds count rows of size bytes each, packed from the start of a block aligned to align, with room for capacity rows. A typed view is only handed out for a T of exactly size bytes, aligned to exactly align. The type must also be plain data, a struct of numbers with no pointers in it, since a column copies rows as bytes; that part is on you.
In the course: 2.2 builds the column, 2.3 puts columns in a table, and 2.5 is about what happens to a pointer into a column that grows.
Watch it: the playground
python tools/dojo.py play column
It fills a column of vec2s and draws each row's bytes next to the value they are:
push 3 zeroed rows, and write each through the typed view
capacity 4, 3 rows in use, 32 bytes of storage
row 0 00 00 80 3f 00 00 00 40 as a vec2 (1, 2)
row 1 00 00 40 40 00 00 80 40 as a vec2 (3, 4)
Then it grows the column, swap-removes from it, and asks for the rows as two wrong types. Step through it: break on column_grow and watch the rows move to the new block. The old block stays in the arena, unused, so a slice taken before the grow still points somewhere valid, and anything written through it is lost without a word.
The switch at the top, grow_by_doubling, picks how a full column grows. Set it to 0, so the column grows by one row, and run it again. The last line is the point: the same 1000 pushes copy 1,020 rows when the column doubles, and 499,494 when it grows by one.
1000 pushes into a u32 column, growing by doubling
9 grows, 1020 rows copied to make room
Do reps: the kata
You write column. Its bytes come from the arena, aligned to the column's alignment:
data: ^u8 = cast(memops_arena_push_zero(c*.arena, c*.size * capacity, c*.align), ^u8);
and the typed view is a generic proc that checks the type before it casts:
column_slice: proc<T>(c: *column) -> []T = {
plus column_init(c, arena, size, align), column_len, column_capacity, column_push_zeroed (a zeroed row at the end, and its row number), column_swap_remove(c, row), and column_row(c, row), a row's bytes as a []u8.
| level | you write | target |
|---|---|---|
| 1 | column_slice, the typed view, in an otherwise finished file | 3:00 |
| 2 | every proc body; the fields and signatures are given | 15:00 |
| 3 | the whole file, from a blank page | 20:00 |
python tools/dojo.py start column 2 kata.rin from level 2, and the clock starts
python tools/dojo.py column your kata.rin against the harness
python tools/dojo.py column --sanitize the same with AddressSanitizer and UBSan
The harness first runs a short scenario: three vec3s written through column_slice and read back as bytes, a swap-remove, a column aligned to 16, and a column of tags, zero-byte rows. Then it runs 20,000 random pushes, writes and swap-removes, shared between four row types: 3 bytes aligned to 1, a u32, a 12-byte vec3, and a 16-byte vec4 aligned to 16. Each is checked against a vec of that type, through column_slice and through column_row, along with the alignment of every view, and that a full column grows by at least half. The seeds give the same runs as rust-dod's dojo.
--sanitize is worth the wait here. A row copied one byte too far, or a memcpy from a block that was never allocated, often works on your machine. AddressSanitizer and UBSan don't let it happen by luck. (They need the C compiler's sanitizer runtime; with clang that may be missing, so try CC=gcc.)
What the harness can't check is a refusal: a column read as the wrong type should stop the program with a message, and so should a row past the rows in use, and a harness that stops can't go on to the next step. Try them by hand in a scratch program: read a vec3 column as a u32, then as a struct of twelve u8s (the right size, aligned to 1), then ask for column_row(c&, 2) of a two-row column. Each should stop with a message that says what was wrong. rust-dod's harness checks these, because Rust can catch a panic.
Level 4, variations. The same harness, different insides:
- Grow by 1.5 instead of 2. Count the grows and the rows copied for 1000 pushes. Why do some allocators prefer it?
- Allocate with **
memops_arena_realloc_** instead of a new block and amemcpy. When does it move the rows, and when can it extend the block in place? - Shrink: when a swap-remove leaves the column less than a quarter full, move the rows to a block of half the capacity. Why a quarter and not a half? (Push and remove one row, over and over, at the boundary.) In an arena, what does shrinking actually give back?
When it fails
FAIL at operation 11 (seed 1): the column was full at 4 rows and grew to 5; it should grow by at least half
debug it: build/dojo/column/harness --seed 1 --stop 11
with a breakpoint on dojo_stop_here, which runs just before that operation
python tools/dojo.py column built that harness with your kata. Run it under your debugger with those arguments and a breakpoint on dojo_stop_here (gdb --args build/dojo/column/harness --seed 1 --stop 11, then break dojo_stop_here, run). It stops just before the operation that fails; step out and into your code.
If the harness crashes instead, on a slice index past the end say, the runner runs it again with --trace 1 and prints the operation that was running.
The playground. The kata's harness is dojo/4.column/harness.rin, and the levels start from dojo/4.column/level1.rin to level3.rin.
1// Dojo 4, the playground: a type-erased column.2//3// A column is one component's rows, kept as bytes. It doesn't know the type4// it holds, only how many bytes make a row and how they must be aligned, and5// that is enough to grow, to zero a row, and to move the last row into a hole.6// The type comes back only at the edge: a typed view turns the bytes into a7// []T, for a T of the column's size and alignment.8//9// Two things make it work:10//11// * the storage starts on a boundary of the column's alignment, so every12// row of the type lands where it may live;13// * a full column grows by a factor, not by a row, so the copies that14// growing costs add up to about one per push.15//16// This program fills a column, draws each row's bytes next to the value they17// are, grows it, swap-removes from it, and asks it for the wrong type. Break18// on `column_grow` in a debugger and watch the rows move to the new block.19//20// The switch: set grow_by_doubling to 0, so a full column grows by one row,21// and run it again. Look at the last line: the same 1000 pushes, and how many22// rows were copied to make room for them. It's a global, read each time a23// column grows, so a debugger can flip it mid-run too: stop on column_grow, set24// it to 0 in the watch window, and continue.25 26import "std/print.rin"27import "std/memops.rin"28 29grow_by_doubling: b32 = 1;30 31vec2: struct = {32 x: f32;33 y: f32;34}35 36// 8 bytes, like a vec2, but aligned to 1.37bytes8: struct = {38 b: [8]u8;39}40 41column: struct = {42 arena: *memops_arena;43 data: ^u8;44 size: u64;45 align: u64;46 count: u64;47 capacity: u64;48 // What growing has cost so far.49 grows: u64;50 copied: u64;51}52 53column_init: proc(c: *column, arena: *memops_arena, size: u64, align: u64) -> void = {54 c* = {};55 c*.arena = arena;56 c*.size = size;57 c*.align = align;58}59 60// A new block from the arena, and the rows in use copied into it. The old61// block stays in the arena, unused: an arena frees everything at once, never62// one allocation.63column_grow: proc(c: *column) -> void = {64 capacity: u64 = 4;65 if (c*.capacity > 0) {66 capacity = grow_by_doubling ? c*.capacity * 2 : c*.capacity + 1;67 }68 data: ^u8 = cast(memops_arena_push_zero(c*.arena, c*.size * capacity, c*.align), ^u8);69 if (c*.count > 0) {70 memcpy(data, c*.data, c*.size * c*.count);71 }72 c*.data = data;73 c*.grows += 1;74 c*.copied += c*.count;75 c*.capacity = capacity;76}77 78column_push_zeroed: proc(c: *column) -> u64 = {79 if (c*.count == c*.capacity) {80 column_grow(c);81 }82 row: u64 = c*.count;83 c*.count += 1;84 memset(c*.data[row * c*.size]&, 0, c*.size);85 return row;86}87 88column_swap_remove: proc(c: *column, row: u64) -> void = {89 last: u64 = c*.count - 1;90 if (row != last) {91 memcpy(c*.data[row * c*.size]&, c*.data[last * c*.size]&, c*.size);92 }93 c*.count = last;94}95 96column_row: proc(c: *column, row: u64) -> []u8 = {97 return c*.data[row * c*.size..(row + 1) * c*.size];98}99 100// Whether T can be the column's type: its size and its alignment. The kata101// stops the program for a T that can't; this program wants to keep going.102column_fits: proc<T>(c: *column) -> b32 = {103 size: u64 = sizeof(T);104 align: u64 = alignof(T);105 return size == c*.size && align == c*.align;106}107 108column_slice: proc<T>(c: *column) -> []T = {109 raw: ^T = cast(c*.data, ^T);110 return raw[0..c*.count];111}112 113draw: proc(c: *column) -> void = {114 printfmt(" capacity {}, {} rows in use, {} bytes of storage\n", c*.capacity, c*.count, c*.capacity * c*.size);115 view: []vec2 = column_slice<vec2>(c);116 for (row: u64 = 0; row < c*.count; row += 1) {117 printfmt(" row {} ", row);118 bytes: []u8 = column_row(c, row);119 for (i: u64 = 0; i < bytes.length; i += 1) {120 printf(" %02x", cast(bytes[i], u32));121 }122 printf(" as a vec2 (%g, %g)\n", cast(view[row].x, f64), cast(view[row].y, f64));123 }124}125 126push: proc(c: *column, x: f32, y: f32) -> void = {127 row: u64 = column_push_zeroed(c);128 view: []vec2 = column_slice<vec2>(c);129 view[row] = { .x = x, .y = y };130}131 132refused: proc(fits: b32) -> ^const char = {133 return fits ? "a slice" : "refused";134}135 136main: proc() -> i32 = {137 arena: memops_arena = {};138 memops_arena_initialize(arena&);139 c: column = {};140 column_init(c&, arena&, sizeof(vec2), alignof(vec2));141 printfmt("a column of vec2: {}-byte rows aligned to {}, kept in blocks from an arena\n", c.size, c.align);142 143 printfmt("push 3 zeroed rows, and write each through the typed view\n");144 for (i: u64 = 0; i < 3; i += 1) {145 push(c&, cast(2 * i + 1, f32), cast(2 * i + 2, f32));146 }147 draw(c&);148 149 printfmt("push 2 more: the column is full at 4 rows, so it grows before the fifth\n");150 push(c&, 7.0, 8.0);151 before: u64 = c.grows;152 push(c&, 9.0, 10.0);153 if (c.grows > before) {154 printfmt(" grew to {} rows: a new block from the arena, {} rows copied into it, the old one left there\n",155 c.capacity, c.count - 1);156 printfmt(" a slice taken before the grow still points into the old block: writes through it are lost\n");157 }158 draw(c&);159 160 printfmt("swap_remove row 0: the last row's 8 bytes move into row 0\n");161 column_swap_remove(c&, 0);162 draw(c&);163 164 printfmt("ask for the rows as other types\n");165 printfmt(" as u32: {}, a u32 is 4 bytes and these rows are 8\n", refused(column_fits<u32>(c&)));166 printfmt(" as bytes8: {}, the size is right, but it is aligned to 1, not 4\n", refused(column_fits<bytes8>(c&)));167 168 big: column = {};169 column_init(big&, arena&, sizeof(u32), alignof(u32));170 for (i: u32 = 0; i < 1000; i += 1) {171 row: u64 = column_push_zeroed(big&);172 values: []u32 = column_slice<u32>(big&);173 values[row] = i;174 }175 if (grow_by_doubling) {176 printfmt("1000 pushes into a u32 column, growing by doubling\n");177 } else {178 printfmt("1000 pushes into a u32 column, growing by one row at a time\n");179 }180 printfmt(" {} grows, {} rows copied to make room\n", big.grows, big.copied);181 sum: u32 = 0;182 values: []u32 = column_slice<u32>(big&);183 for (i: u64 = 0; i < values.length; i += 1) {184 sum += values[i];185 }186 printfmt(" and the rows are all there: they add up to {}\n", sum);187 return 0;188}What play_column prints. tools/check.py fails the playground if a single byte differs.
a column of vec2: 8-byte rows aligned to 4, kept in blocks from an arena
push 3 zeroed rows, and write each through the typed view
capacity 4, 3 rows in use, 32 bytes of storage
row 0 00 00 80 3f 00 00 00 40 as a vec2 (1, 2)
row 1 00 00 40 40 00 00 80 40 as a vec2 (3, 4)
row 2 00 00 a0 40 00 00 c0 40 as a vec2 (5, 6)
push 2 more: the column is full at 4 rows, so it grows before the fifth
grew to 8 rows: a new block from the arena, 4 rows copied into it, the old one left there
a slice taken before the grow still points into the old block: writes through it are lost
capacity 8, 5 rows in use, 64 bytes of storage
row 0 00 00 80 3f 00 00 00 40 as a vec2 (1, 2)
row 1 00 00 40 40 00 00 80 40 as a vec2 (3, 4)
row 2 00 00 a0 40 00 00 c0 40 as a vec2 (5, 6)
row 3 00 00 e0 40 00 00 00 41 as a vec2 (7, 8)
row 4 00 00 10 41 00 00 20 41 as a vec2 (9, 10)
swap_remove row 0: the last row's 8 bytes move into row 0
capacity 8, 4 rows in use, 64 bytes of storage
row 0 00 00 10 41 00 00 20 41 as a vec2 (9, 10)
row 1 00 00 40 40 00 00 80 40 as a vec2 (3, 4)
row 2 00 00 a0 40 00 00 c0 40 as a vec2 (5, 6)
row 3 00 00 e0 40 00 00 00 41 as a vec2 (7, 8)
ask for the rows as other types
as u32: refused, a u32 is 4 bytes and these rows are 8
as bytes8: refused, the size is right, but it is aligned to 1, not 4
1000 pushes into a u32 column, growing by doubling
9 grows, 1020 rows copied to make room
and the rows are all there: they add up to 499500
The dojo · block 5
A command buffer
dojo/5.commands/play_commands.rin
Systems walk tables, and while they walk, they decide to change which things exist: a bullet's time is up, a turret fires, a drone catches fire. Doing it on the spot breaks the walk. A despawn swap-removes, so the last row moves into the hole, behind the loop, and is never looked at. So a system records the change instead, as a command, and the frame applies every command, in order, once nothing is walking anything.
Three details make it work:
- Pending spawns. A spawn has no handle until it is applied, so recording one hands back a pending number, and later commands in the same buffer can name the spawn by it.
- Skipping the dead. Two systems may both despawn the same thing, or set a value on something an earlier command removed. Applied in order, the second finds a dead handle and is skipped. Generations make that one comparison.
- A frame arena. After the buffer is applied its vecs are cleared, and keep their room, so once a game has run a few frames, recording a command allocates nothing.
The invariant: recording changes nothing but the buffer. Applying it has exactly the effect of running every command straight away, in the order it was recorded, with each pending number meaning the handle its spawn made.
In the course: 3.3 builds the buffer and shows the bug it fixes, and 3.4's game runs on it. The Advanced lesson 6 has one too.
Watch it: the playground
python tools/dojo.py play commands
A packed world of five things, and one frame of three systems. The expire system records a despawn for each thing whose life is up, the spawner records a replacement and sets its life through its pending number, and a second system despawns something the first already did. Then the buffer is applied:
apply the buffer, in the order it was recorded
despawn 0:1: ok, 4:1 moves into 0:1's row
despawn 2:1: ok, 3:1 moves into 2:1's row
despawn 3:1: ok, it was the last row
spawn with life 1: pending 0 is 3:2
set pending 0 life 5: ok, 3:2 has life 5
despawn 2:1: skipped, 2:1 is dead
Step through it: break on commands_apply and walk the list. Then flip the switch at the top, defer, to 0, so every system changes the world on the spot, and run it again. The expire system despawns 0:1, 4:1 moves into its row behind the loop, and when the loop ends, 3:1 is still there with no life:
things left with no life: none
becomes things left with no life: 3:1, moved behind the loop and never looked at. That's 3.3's bug.
Do reps: the kata
You write commands. A command names its target either way, a handle or a spawn of this buffer:
target: struct = {
is_pending: b32;
handle: dojo_handle;
spawn: pending;
}
made by target_entity(h) or target_pending(p), so commands_set and commands_despawn take either:
commands_set: proc(c: *commands, t: target, value: u32) -> void = {
commands_apply: proc(c: *commands, w: *world) -> applied = {
plus commands_init, commands_spawn(c, value), which returns a pending, commands_despawn(c, t), and commands_len, the commands waiting. The world is given, in the harness: a generational store of numbers with world_spawn, world_despawn and world_set, where the last two return dojo_status.dead for a dead handle. commands_apply returns applied, the count of commands that changed something and of those that were skipped.
| level | you write | target |
|---|---|---|
| 1 | commands_apply, in an otherwise finished file | 3:00 |
| 2 | every proc body; the types and signatures are given | 15:00 |
| 3 | the whole file, from a blank page | 20:00 |
python tools/dojo.py start commands 2 kata.rin from level 2, and the clock starts
python tools/dojo.py commands your kata.rin against the harness
The harness first runs a short scenario: a set, a despawn, a spawn that reuses the despawned index, a set through the spawn's pending number, and two commands that find their target dead. Then a fuzz run, frame after frame: random commands recorded against live handles, dead handles and the frame's own pending spawns, then applied. A model runs the same commands straight away on a copy of the world from before the frame, and the harness checks the two worlds match, the applied and skipped counts match, and the buffer is empty afterwards. The copies live in a frame arena of the harness's own, cleared after every frame. The seeds give the same runs as rust-dod's dojo.
Level 4, variations. The same harness, different insides:
- Keep the spawned handles in the command list itself, as 3.3 does: a spawn command holds the handle
commands_applygives it, and a pending number finds its spawn through it. Nospawnedvec. - Record the whole buffer as **one
vec<u8>**: a kind byte, then the command's fields as bytes. One allocation for any mix of commands, read back in order. - Keep the buffer in an arena of its own, and clear the arena after
commands_applyinstead of clearing the vecs, as lesson 6 does. What must not live in that arena?
When it fails
FAIL at operation 28 (seed 1): commands_apply returned applied 7, skipped 4, the model says applied 9, skipped 2
debug it: build/dojo/commands/harness --seed 1 --stop 28
with a breakpoint on dojo_stop_here, which runs just before that operation
python tools/dojo.py commands built that harness with your kata. Run it under your debugger with those arguments and a breakpoint on dojo_stop_here (gdb --args build/dojo/commands/harness --seed 1 --stop 28, then break dojo_stop_here, run). It stops just before the operation that fails; step out and into your code.
If the harness crashes instead, on a pending number past the spawns applied so far say, the runner runs it again with --trace 1 and prints the operation that was running.
The playground. The kata's harness is dojo/5.commands/harness.rin, and the levels start from dojo/5.commands/level1.rin to level3.rin.
1// Dojo 5, the playground: a command buffer.2//3// Systems walk tables, and while they walk they decide to change which things4// exist: a bullet's time is up, a turret fires. Doing it on the spot breaks the5// walk. A despawn swap-removes, so the last row moves into the hole, behind the6// loop, and is never looked at. So a system records the change instead, a7// command, and the frame applies every command, in order, once no one is8// walking anything:9//10// * a spawn has no handle until it is applied, so the buffer hands back a11// pending number, and later commands can name the spawn by it;12// * a command whose target died earlier in the same buffer is skipped, which13// generations make one comparison;14// * the buffer is cleared after it is applied, and keeps its memory for the15// next frame: a frame arena.16//17// This program builds a small packed world, runs one frame of three systems,18// and prints every command, then the buffer being applied. Break on19// `commands_apply` in a debugger and step through the list.20//21// The switch: set defer to 0, so the systems change the world on the spot, and22// run it again. Look at which things are left with no life. It's a global, so23// a debugger can flip it too: the frame reads it once, at its start, so it's24// all one way or the other; stop on that line, set it to 0 in the watch window,25// and continue.26 27import "std/print.rin"28import "std/memops.rin"29import "std/vec.rin"30 31defer: b32 = 1;32 33handle: struct = {34 index: u32;35 generation: u32;36}37 38slot: struct = {39 generation: u32;40 row: u64;41}42 43// A packed world, like the course's tables: things in rows 0 to count - 1,44// each with a life in frames, and a slot per index that says which row it is45// in.46world: struct = {47 arena: *memops_arena;48 slots: vec<slot>;49 free: vec<u32>;50 owners: vec<handle>;51 lives: vec<u32>;52}53 54world_alive: proc(w: *world, h: handle) -> b32 = {55 return cast(h.index, u64) < w*.slots.length56 && vec<slot>at(w*.slots&, cast(h.index, u64))*.generation == h.generation;57}58 59world_spawn: proc(w: *world, life: u32) -> handle = {60 index: u32 = 0;61 if (!vec<u32>is_empty(w*.free&)) {62 index = vec<u32>pop(w*.free&);63 } else {64 index = cast(w*.slots.length, u32);65 _ = vec<slot>append(w*.arena, w*.slots&, { .generation = 1 });66 }67 s: *slot = vec<slot>at(w*.slots&, cast(index, u64));68 s*.row = w*.owners.length;69 h: handle = { .index = index, .generation = s*.generation };70 _ = vec<handle>append(w*.arena, w*.owners&, h);71 _ = vec<u32>append(w*.arena, w*.lives&, life);72 return h;73}74 75// What a despawn did: 0 refused, the target was dead; 1 the last row moved76// into the hole, into `moved`; 2 the target was the last row.77despawn_result: struct = {78 what: u32;79 moved: handle;80}81 82world_despawn: proc(w: *world, h: handle) -> despawn_result = {83 result: despawn_result = {};84 if (!world_alive(w, h)) {85 return result;86 }87 s: *slot = vec<slot>at(w*.slots&, cast(h.index, u64));88 row: u64 = s*.row;89 s*.generation += 1;90 _ = vec<u32>append(w*.arena, w*.free&, h.index);91 last_owner: handle = vec<handle>pop(w*.owners&);92 last_life: u32 = vec<u32>pop(w*.lives&);93 if (row == w*.owners.length) {94 result.what = 2;95 return result;96 }97 vec<handle>at(w*.owners&, row)* = last_owner;98 vec<u32>at(w*.lives&, row)* = last_life;99 vec<slot>at(w*.slots&, cast(last_owner.index, u64))*.row = row;100 result.what = 1;101 result.moved = last_owner;102 return result;103}104 105world_set_life: proc(w: *world, h: handle, life: u32) -> b32 = {106 if (!world_alive(w, h)) {107 return 0;108 }109 row: u64 = vec<slot>at(w*.slots&, cast(h.index, u64))*.row;110 vec<u32>at(w*.lives&, row)* = life;111 return 1;112}113 114draw: proc(w: *world) -> void = {115 for (row: u64 = 0; row < w*.owners.length; row += 1) {116 h: handle = vec<handle>at(w*.owners&, row)*;117 printfmt(" row {} {}:{} life {}\n", row, h.index, h.generation, vec<u32>at(w*.lives&, row)*);118 }119}120 121print_despawned: proc(r: despawn_result, h: handle) -> void = {122 if (r.what == 1) {123 printfmt("ok, {}:{} moves into {}:{}'s row\n", r.moved.index, r.moved.generation, h.index, h.generation);124 } else if (r.what == 2) {125 printfmt("ok, it was the last row\n");126 } else {127 printfmt("skipped, {}:{} is dead\n", h.index, h.generation);128 }129}130 131// ---------------------------------------------------------------------------132// The buffer133// ---------------------------------------------------------------------------134 135command_kind: enum = {136 spawn,137 set_life,138 despawn,139}140 141// A command, and who it is for: a handle, or the `spawn`-th spawn of this142// buffer if `to_pending` is set.143command: struct = {144 kind: command_kind;145 to_pending: b32;146 entity: handle;147 spawn: u64;148 life: u32;149}150 151commands: struct = {152 arena: *memops_arena;153 list: vec<command>;154 spawns: u64;155 spawned: vec<handle>;156}157 158print_target: proc(c: command) -> void = {159 if (c.to_pending) {160 printfmt("pending {}", c.spawn);161 } else {162 printfmt("{}:{}", c.entity.index, c.entity.generation);163 }164}165 166print_command: proc(c: command) -> void = {167 if (c.kind == command_kind.spawn) {168 printfmt("spawn with life {}", c.life);169 } else if (c.kind == command_kind.set_life) {170 printfmt("set ");171 print_target(c);172 printfmt(" life {}", c.life);173 } else {174 printfmt("despawn ");175 print_target(c);176 }177}178 179commands_record: proc(b: *commands, c: command) -> void = {180 printfmt(" record: ");181 print_command(c);182 printfmt("\n");183 _ = vec<command>append(b*.arena, b*.list&, c);184}185 186commands_spawn: proc(b: *commands, life: u32) -> u64 = {187 commands_record(b, { .kind = command_kind.spawn, .life = life });188 b*.spawns += 1;189 return b*.spawns - 1;190}191 192commands_resolve: proc(b: *commands, c: command) -> handle = {193 if (c.to_pending) {194 return vec<handle>at(b*.spawned&, c.spawn)*;195 }196 return c.entity;197}198 199// Every command in order, then an empty buffer that keeps its memory.200commands_apply: proc(b: *commands, w: *world) -> void = {201 for (i: u64 = 0; i < b*.list.length; i += 1) {202 c: command = vec<command>at(b*.list&, i)*;203 printfmt(" ");204 print_command(c);205 printfmt(": ");206 if (c.kind == command_kind.spawn) {207 h: handle = world_spawn(w, c.life);208 _ = vec<handle>append(b*.arena, b*.spawned&, h);209 printfmt("pending {} is {}:{}\n", b*.spawned.length - 1, h.index, h.generation);210 } else if (c.kind == command_kind.set_life) {211 h: handle = commands_resolve(b, c);212 if (world_set_life(w, h, c.life)) {213 printfmt("ok, {}:{} has life {}\n", h.index, h.generation, c.life);214 } else {215 printfmt("skipped, {}:{} is dead\n", h.index, h.generation);216 }217 } else {218 h: handle = commands_resolve(b, c);219 print_despawned(world_despawn(w, h), h);220 }221 }222 vec<command>clear(b*.list&);223 vec<handle>clear(b*.spawned&);224 b*.spawns = 0;225}226 227main: proc() -> i32 = {228 arena: memops_arena = {};229 memops_arena_initialize(arena&);230 w: world = {};231 w.arena = arena&;232 lives: [5]u32 = { 0, 3, 0, 0, 2 };233 for (i: u64 = 0; i < 5; i += 1) {234 _ = world_spawn(w&, lives[i]);235 }236 cmds: commands = {};237 cmds.arena = arena&;238 printfmt("a world of 5 things, packed in rows; a life of 0 means its time is up\n");239 draw(w&);240 241 deferring: b32 = defer;242 if (deferring) {243 printfmt("the expire system walks the rows, and records a despawn for each life of 0\n");244 } else {245 printfmt("the expire system walks the rows, and despawns each life of 0 on the spot\n");246 }247 row: u64 = 0;248 while (row < w.owners.length) {249 h: handle = vec<handle>at(w.owners&, row)*;250 if (vec<u32>at(w.lives&, row)* == 0) {251 if (deferring) {252 commands_record(cmds&, { .kind = command_kind.despawn, .entity = h });253 } else {254 printfmt(" row {}: despawn {}:{}: ", row, h.index, h.generation);255 print_despawned(world_despawn(w&, h), h);256 }257 }258 row += 1;259 }260 261 printfmt("the spawner makes a replacement, and gives it a life through its pending number\n");262 if (deferring) {263 p: u64 = commands_spawn(cmds&, 1);264 commands_record(cmds&, { .kind = command_kind.set_life, .to_pending = 1, .spawn = p, .life = 5 });265 } else {266 h: handle = world_spawn(w&, 1);267 printfmt(" spawn: {}:{}\n", h.index, h.generation);268 if (world_set_life(w&, h, 5)) {269 printfmt(" set {}:{} life 5: ok\n", h.index, h.generation);270 } else {271 printfmt(" set {}:{} life 5: dead\n", h.index, h.generation);272 }273 }274 275 doomed: handle = { .index = 2, .generation = 1 };276 printfmt("a second system despawns {}:{} too: it doesn't know the first one did\n", doomed.index, doomed.generation);277 if (deferring) {278 commands_record(cmds&, { .kind = command_kind.despawn, .entity = doomed });279 printfmt("the buffer holds {} commands, and the world has not changed:\n", cmds.list.length);280 draw(w&);281 printfmt("apply the buffer, in the order it was recorded\n");282 commands_apply(cmds&, w&);283 } else {284 printfmt(" despawn {}:{}: ", doomed.index, doomed.generation);285 print_despawned(world_despawn(w&, doomed), doomed);286 }287 288 printfmt("the world after the frame\n");289 draw(w&);290 printfmt("things left with no life:");291 left: u64 = 0;292 for (r: u64 = 0; r < w.owners.length; r += 1) {293 if (vec<u32>at(w.lives&, r)* == 0) {294 h: handle = vec<handle>at(w.owners&, r)*;295 printfmt(" {}:{}", h.index, h.generation);296 left += 1;297 }298 }299 if (left == 0) {300 printfmt(" none\n");301 } else {302 printfmt(", moved behind the loop and never looked at\n");303 }304 printfmt("commands left in the buffer: {}\n", cmds.list.length);305 return 0;306}What play_commands prints. tools/check.py fails the playground if a single byte differs.
a world of 5 things, packed in rows; a life of 0 means its time is up
row 0 0:1 life 0
row 1 1:1 life 3
row 2 2:1 life 0
row 3 3:1 life 0
row 4 4:1 life 2
the expire system walks the rows, and records a despawn for each life of 0
record: despawn 0:1
record: despawn 2:1
record: despawn 3:1
the spawner makes a replacement, and gives it a life through its pending number
record: spawn with life 1
record: set pending 0 life 5
a second system despawns 2:1 too: it doesn't know the first one did
record: despawn 2:1
the buffer holds 6 commands, and the world has not changed:
row 0 0:1 life 0
row 1 1:1 life 3
row 2 2:1 life 0
row 3 3:1 life 0
row 4 4:1 life 2
apply the buffer, in the order it was recorded
despawn 0:1: ok, 4:1 moves into 0:1's row
despawn 2:1: ok, 3:1 moves into 2:1's row
despawn 3:1: ok, it was the last row
spawn with life 1: pending 0 is 3:2
set pending 0 life 5: ok, 3:2 has life 5
despawn 2:1: skipped, 2:1 is dead
the world after the frame
row 0 4:1 life 2
row 1 1:1 life 3
row 2 3:2 life 5
things left with no life: none
commands left in the buffer: 0
The dojo · block 6
Bitsets
dojo/6.bitset/play_bitset.rin
A set of small numbers, 0 to 63, fits in one u64: bit id is set when id is in the set. An ECS keeps what each kind of entity is made of this way, a signature with a bit per component, and a query as two sets: the ids a table must have, and the ids it must not. The questions an ECS asks all day are then one or two instructions each, however many components there are:
with(set, id) set | (1 shl id)
without(set, id) set & ~(1 shl id)
has(set, id) (set & (1 shl id)) != 0
contains_all(set, all) (set & all) == all
intersects(set, none) (set & none) != 0
The same trick is a file's permissions, which of 64 workers are idle, and which chunks of a map are loaded.
The invariant: bit id is set exactly when id is in the set. Everything else follows from it: one set contains all of another when masking it with the other leaves the other whole, and two sets share an id when their AND isn't zero.
In the course: 2.1 gives each component a bit and each kind of entity a signature, 2.4 finds a table by its signature, 2.6 moves an entity by setting or clearing one bit, and 3.1 matches queries against tables with contains_all and intersects.
Watch it: the playground
python tools/dojo.py play bitset
It gives five kinds of thing a signature each, draws them as rows of bits, highest id first, and runs queries over them:
query move: all {position velocity}, none {frozen}
all 00000011
none 01000000
player 00011111 matches
drone 00001111 matches
turret 00101101 no
rock 00001001 no
ghost 01001011 no
Read it by columns: a kind matches when it has a 1 under every 1 of all, and a 0 under every 1 of none. Then the drone is frozen and the ghost thaws, one bit each, and the same query gives a different answer. Each kind keeps its ids twice, as a signature and as a b32 per id, and verify checks the fast answer against the slow one after every step.
Step through it: break on matches, and watch set & wanted in the watch window, in binary. Then flip the switch at the top, all_not_any, to 0 and run it again. contains_all now asks whether the signature has any of the ids. The turret and the rock have a position, so both match the movement query, and the turret and the ghost each have half of what the think query wants:
verify: turret matches, but it has no velocity
It's the most common bitset bug, and it compiles: the compiler has no idea which question you meant.
Do reps: the kata
You write procs over a u64. The one with the idea in it:
bitset_contains_all: proc(set: u64, wanted: u64) -> b32 = {
return (set & wanted) == wanted;
}
and a query, built from it:
bitset_matches: proc(set: u64, all: u64, none: u64) -> b32 = {
return bitset_contains_all(set, all) && !bitset_intersects(set, none);
}
plus bitset_bit (the set of id alone), bitset_with, bitset_without, bitset_has, bitset_intersects, bitset_count and bitset_ids, which writes the ids, smallest first, into a slice with room for 64 and returns how many. Ids are u32s, 0 to 63.
| level | you write | target |
|---|---|---|
| 1 | bitset_contains_all, in an otherwise finished file | 1:00 |
| 2 | every proc body; the signatures are given | 5:00 |
| 3 | the whole file, from a blank page | 6:00 |
python tools/dojo.py start bitset 2
python tools/dojo.py bitset
The harness runs the playground's kinds and queries as a scenario, then 20,000 random operations on a set: adding and taking away ids, and asking has, contains_all, intersects, matches, count and ids, against a model that keeps a b32 per id and answers one id at a time. The sets it asks about are mostly made from the set itself: part of it, part of it plus one more id, or ids it doesn't have, so the answers aren't nearly always false. Ids go up to 63, so a shift done in 32 bits shows. The seeds give the same runs as rust-dod's dojo.
rin notes. rin binds & tighter than ==, as Rust does, so set & wanted == wanted means what it says; in C it would be set & (wanted == wanted). The reference keeps the parentheses anyway, for anyone reading it with C eyes. And rin gives a bare 1 the type its result is going to, so in a proc that returns a u64, 1 shl id is already a 64-bit shift. The 32-bit bug needs a u32 on the left, one: u32 = 1; one shl id, and --sanitize names it: UBSan reports a shift exponent too large for a 32-bit type.
Level 4, variations. The same harness, different insides:
- Count without a loop over 64:
set & (set - 1)clears the lowest set bit. Loop until the set is empty, one step per id rather than per bit. - **
bitset_idsthe same way**:low = set & (0 - set)is the lowest set bit alone, its id is the number of bits below it,bitset_count(low - 1), and clearing it moves on. - More than 64 components: a struct of two
u64s, with idiat biti % 64of wordi / 64, and the same procs. What doescontains_allbecome? (Not checked by the harness; write your own test.)
When it fails
FAIL at operation 2 (seed 1): without({}, 57) gave {57}, it should give {}
debug it: build/dojo/bitset/harness --seed 1 --stop 2
with a breakpoint on dojo_stop_here, which runs just before that operation
That one is set ^ bitset_bit(id): an XOR flips the bit, so taking away an id the set doesn't have puts it in. Sets print as their ids, smallest first. Run the harness under your debugger with those arguments and a breakpoint on dojo_stop_here, which runs just before that operation.
The playground. The kata's harness is dojo/6.bitset/harness.rin, and the levels start from dojo/6.bitset/level1.rin to level3.rin.
1// Dojo 6, the playground: bitsets.2//3// A set of component ids fits in a u64: bit `id` is set when `id` is in the4// set. In an ECS that set is a signature, what a kind of entity is made of,5// and a query is two sets: the ids an entity must have, and the ids it must6// not. A query against a signature is then two ANDs and two compares,7// however many components there are:8//9// (signature & all) == all && (signature & none) == 010//11// This program gives five kinds of thing a signature each, draws them as rows12// of bits, runs queries over them, and changes two of them. Each kind keeps13// its ids twice: the fast way, a signature, and the slow way, a b32 per id.14// After every step, verify checks the fast way against the slow way and says15// what disagrees; break inside its `if`s to stop on the first.16//17// The switch: set all_not_any to 0 and run it again. contains_all then asks18// whether the signature has any of the ids, not all of them, and the turret,19// which can't move, matches the movement query. It's a global, read each time20// it's used, so a debugger can flip it mid-run too: stop on the first `query`21// in main, set it to 0 in the watch window, and continue.22 23import "std/print.rin"24 25all_not_any: b32 = 1;26 27names: [8]^const char = { "position", "velocity", "health", "sprite", "player", "turret", "frozen", "loot" };28position: u32 = 0;29velocity: u32 = 1;30health: u32 = 2;31sprite: u32 = 3;32player: u32 = 4;33turret: u32 = 5;34frozen: u32 = 6;35 36bit: proc(id: u32) -> u64 = {37 return 1'u64 shl cast(id, u64);38}39 40with: proc(set: u64, id: u32) -> u64 = {41 return set | bit(id);42}43 44without: proc(set: u64, id: u32) -> u64 = {45 return set & ~bit(id);46}47 48has: proc(set: u64, id: u32) -> b32 = {49 return (set & bit(id)) != 0;50}51 52contains_all: proc(set: u64, wanted: u64) -> b32 = {53 if (all_not_any) {54 return (set & wanted) == wanted;55 }56 return (set & wanted) != 0;57}58 59matches: proc(set: u64, all: u64, none: u64) -> b32 = {60 return contains_all(set, all) && (set & none) == 0;61}62 63// Eight bits, highest id first, the way a number is written.64print_bits: proc(set: u64) -> void = {65 for (id: u32 = 8; id > 0; id -= 1) {66 if (has(set, id - 1)) {67 printf("1");68 } else {69 printf("0");70 }71 }72}73 74print_names: proc(set: u64) -> void = {75 printf("{");76 first: b32 = 1;77 for (id: u32 = 0; id < 8; id += 1) {78 if (has(set, id)) {79 if (!first) {80 printf(" ");81 }82 printf("%s", names[id]);83 first = 0;84 }85 }86 printf("}");87}88 89// A kind of thing: its signature and, the slow way, a b32 per id.90kind: struct = {91 name: ^const char;92 signature: u64;93 parts: [8]b32;94}95 96// Up to eight kinds.97kinds: struct = {98 all: [8]kind;99 count: u32;100}101 102// What a system asks for: all of these ids, and none of those.103query: struct = {104 all: u64;105 none: u64;106}107 108add_kind: proc(k: *kinds, name: ^const char, ids: []u32) -> u32 = {109 at: u32 = k*.count;110 k*.all[at].name = name;111 for (i: u64 = 0; i < ids.length; i += 1) {112 k*.all[at].signature = with(k*.all[at].signature, ids[i]);113 k*.all[at].parts[ids[i]] = 1;114 }115 k*.count += 1;116 return at;117}118 119draw_kind: proc(it: *kind) -> void = {120 printf(" %-8s", it*.name);121 print_bits(it*.signature);122 printf(" ");123 print_names(it*.signature);124 printf("\n");125}126 127set_part: proc(k: *kinds, at: u32, id: u32, on: b32) -> void = {128 if (on) {129 k*.all[at].signature = with(k*.all[at].signature, id);130 } else {131 k*.all[at].signature = without(k*.all[at].signature, id);132 }133 k*.all[at].parts[id] = on;134 draw_kind(k*.all[at]&);135}136 137run_query: proc(k: *kinds, q: query) -> void = {138 printf(" all ");139 print_bits(q.all);140 printf("\n none ");141 print_bits(q.none);142 printf("\n");143 for (i: u32 = 0; i < k*.count; i += 1) {144 printf(" %-8s", k*.all[i].name);145 print_bits(k*.all[i].signature);146 if (matches(k*.all[i].signature, q.all, q.none)) {147 printf(" matches\n");148 } else {149 printf(" no\n");150 }151 }152}153 154// The fast way against the slow way, for every kind: the signature's bits155// against its b32s, and the query's answer, if there is one, against one id156// at a time.157verify: proc(k: *kinds, q: *query) -> void = {158 for (i: u32 = 0; i < k*.count; i += 1) {159 it: *kind = k*.all[i]&;160 for (id: u32 = 0; id < 8; id += 1) {161 if (has(it*.signature, id) != it*.parts[id]) {162 printf(" verify: %s's signature and its parts disagree about %s\n", it*.name, names[id]);163 }164 }165 if (q == null) {166 continue;167 }168 missing: i32 = -1;169 barred: i32 = -1;170 for (id: u32 = 0; id < 8; id += 1) {171 if (missing < 0 && has(q*.all, id) && !it*.parts[id]) {172 missing = cast(id, i32);173 }174 if (barred < 0 && has(q*.none, id) && it*.parts[id]) {175 barred = cast(id, i32);176 }177 }178 slow: b32 = missing < 0 && barred < 0;179 if (matches(it*.signature, q*.all, q*.none) != slow) {180 if (missing >= 0) {181 printf(" verify: %s matches, but it has no %s\n", it*.name, names[missing]);182 } else if (barred >= 0) {183 printf(" verify: %s matches, but it has %s\n", it*.name, names[barred]);184 } else {185 printf(" verify: %s should match, and doesn't\n", it*.name);186 }187 }188 }189}190 191main: proc() -> i32 = {192 k: kinds = {};193 move_query: query = { .all = bit(position) | bit(velocity), .none = bit(frozen) };194 draw_query: query = { .all = bit(position) | bit(sprite), .none = 0 };195 think_query: query = { .all = bit(velocity) | bit(health), .none = bit(player) };196 player_ids: [5]u32 = { position, velocity, health, sprite, player };197 drone_ids: [4]u32 = { position, velocity, health, sprite };198 turret_ids: [4]u32 = { position, health, sprite, turret };199 rock_ids: [2]u32 = { position, sprite };200 ghost_ids: [4]u32 = { position, velocity, sprite, frozen };201 202 printf("five kinds of thing, one bit per component, highest id first\n");203 add_kind(k&, "player", player_ids[0..5]);204 drone: u32 = add_kind(k&, "drone", drone_ids[0..4]);205 add_kind(k&, "turret", turret_ids[0..4]);206 add_kind(k&, "rock", rock_ids[0..2]);207 ghost: u32 = add_kind(k&, "ghost", ghost_ids[0..4]);208 for (i: u32 = 0; i < k.count; i += 1) {209 draw_kind(k.all[i]&);210 }211 verify(k&, null);212 213 printf("query move: all {position velocity}, none {frozen}\n");214 run_query(k&, move_query);215 verify(k&, move_query&);216 217 printf("query draw: all {position sprite}, none {}\n");218 run_query(k&, draw_query);219 verify(k&, draw_query&);220 221 printf("query think: all {velocity health}, none {player}\n");222 run_query(k&, think_query);223 verify(k&, think_query&);224 225 printf("the drone is frozen: with(drone, frozen)\n");226 set_part(k&, drone, frozen, 1);227 verify(k&, null);228 229 printf("the ghost thaws: without(ghost, frozen)\n");230 set_part(k&, ghost, frozen, 0);231 verify(k&, null);232 233 printf("query move again: the drone stops, the ghost starts\n");234 run_query(k&, move_query);235 verify(k&, move_query&);236 237 hero: u64 = k.all[0].signature;238 count: u32 = 0;239 for (id: u32 = 0; id < 64; id += 1) {240 if (has(hero, id)) {241 count += 1;242 }243 }244 printfmt("the player's signature is the number {}: {} ids,", hero, count);245 for (id: u32 = 0; id < 64; id += 1) {246 if (has(hero, id)) {247 printfmt(" {}", id);248 }249 }250 printfmt("\n");251 return 0;252}What play_bitset prints. tools/check.py fails the playground if a single byte differs.
five kinds of thing, one bit per component, highest id first
player 00011111 {position velocity health sprite player}
drone 00001111 {position velocity health sprite}
turret 00101101 {position health sprite turret}
rock 00001001 {position sprite}
ghost 01001011 {position velocity sprite frozen}
query move: all {position velocity}, none {frozen}
all 00000011
none 01000000
player 00011111 matches
drone 00001111 matches
turret 00101101 no
rock 00001001 no
ghost 01001011 no
query draw: all {position sprite}, none {}
all 00001001
none 00000000
player 00011111 matches
drone 00001111 matches
turret 00101101 matches
rock 00001001 matches
ghost 01001011 matches
query think: all {velocity health}, none {player}
all 00000110
none 00010000
player 00011111 no
drone 00001111 matches
turret 00101101 no
rock 00001001 no
ghost 01001011 no
the drone is frozen: with(drone, frozen)
drone 01001111 {position velocity health sprite frozen}
the ghost thaws: without(ghost, frozen)
ghost 00001011 {position velocity sprite}
query move again: the drone stops, the ghost starts
all 00000011
none 01000000
player 00011111 matches
drone 01001111 no
turret 00101101 no
rock 00001001 no
ghost 00001011 matches
the player's signature is the number 31: 5 ids, 0 1 2 3 4
The dojo · block 7
An arena
dojo/7.arena/play_arena.rin
Most of what a frame allocates dies with the frame: commands, scratch lists, text for the screen. An arena hands all of it out of one buffer and keeps one number, how much of the buffer is used. A push rounds that number up to the alignment asked for, hands out the address there, and moves the number past the new bytes. Nothing is freed on its own. A mark remembers the number, a rewind puts it back, and a clear sets it to 0: everything pushed since goes at once, in one store, with no list of what to free.
push(4, 4) with 1 used: round 1 up to 4, hand out base + 4, used is now 8
For an align that is a power of two, rounding up is (used + align - 1) & ~(align - 1): adding align - 1 reaches the next multiple unless used is one already, and the mask drops what's past it.
The invariant: every push not yet let go of starts at a multiple of its alignment, ends at or before used, and stops before the next push starts; used never passes the end of the buffer.
In the course: everywhere. Every lesson allocates from a memops_arena, from 2.2's columns on, and never frees anything on its own. 3.3 adds a second arena for one frame's commands, cleared after they're applied, and kayan does the same in 6. This block is that arena, small enough to write in a few minutes.
Watch it: the playground
python tools/dojo.py play arena
It pushes things of different sizes and alignments into a 64-byte buffer, rewinds, fills it to the last byte, and clears it. After each step it draws the buffer, 16 bytes to a group: a letter for each byte of a push, - for padding, . for free.
push b: a u32 count, 4 bytes, align 4
offset 4, used 8 of 64: 3 bytes of padding first
a---bbbb........|................|................|................
...
rewind to the mark: e and f are gone at once
a---bbbbcccccccc|dddddddddddddddd|................|................
Each push is filled with its letter, and verify checks after every step that each push is aligned, inside used, clear of the next, and still holds its letter. Step through it: break on bump_push and watch used. The buffer is a struct[align(16)], so an offset that is a multiple of 16 is an address that is one too.
Then flip the switch at the top, align_offsets, to 0 and run it again. Pushes are handed out at used as it is, packed with no padding, and the u32, the u64 and the 16-byte vector land at offsets 1, 5 and 13:
verify: b is at offset 1, not a multiple of 4: its address isn't aligned
The playground only ever writes bytes, with memset, so nothing breaks yet. Read a u32 through a *u32 from there, though, and it's undefined behaviour in C, so in rin: on x86 it usually works, slower when it straddles two cache lines; an aligned SIMD load faults; and the optimiser is allowed to assume it never happens. --sanitize reports it, as UBSan's "misaligned address".
Do reps: the kata
You write bump, an arena over bytes it's given. It isn't called arena, since the one every lesson uses already has that name: memops_arena.
bump: struct = {
base: ^u8;
capacity: u64;
// Bytes before `used` are handed out; bytes after it are free.
used: u64;
}
Its one proc with an idea in it:
bump_push: proc(b: *bump, size: u64, align: u64) -> ^u8 = {
the address of size bytes at an offset from base that is a multiple of align, or null, changing nothing, when they would end past the last byte. Plus bump_init(b, base, capacity), bump_used, bump_capacity, bump_mark, bump_rewind(b, mark) and bump_clear.
| level | you write | target |
|---|---|---|
| 1 | bump_push, in an otherwise finished file | 2:00 |
| 2 | every proc body; the fields and signatures are given | 6:00 |
| 3 | the whole file, from a blank page | 8:00 |
python tools/dojo.py start arena 2
python tools/dojo.py arena
python tools/dojo.py arena --sanitize the same with AddressSanitizer and UBSan
The harness runs the playground's steps as a scenario, then 20,000 random pushes, marks, rewinds and clears on a 256-byte buffer, against a model that keeps the used count and every push not yet let go of. Pushes are 1 to 24 bytes at alignments 1 to 16, with some sized to end exactly at the last byte and some one byte past it. Each push is filled with a pattern through the address it gave, and read back later, so a push that lands on another shows. The seeds give the same runs as rust-dod's dojo.
rin notes. bump_push hands out an address, as memops_arena_push does. rust-dod's kata hands out an offset instead, and bytes_mut(offset, len) to write through: a slice from push would borrow the whole arena for as long as it was kept, and the arena couldn't push again until it was gone. rin lets you keep the address, and keep it past a rewind too, which is the arena's one way to go wrong: an address from before a rewind or a clear points at bytes the next push will hand out again.
Level 4, variations. The same harness, different insides:
- Align the address, not the offset: round
cast(base, usize) + usedup, and subtractbaseagain.memops_arenadoes it this way. The answers are the same for the harness's buffers, which start 16-aligned; what changes for a buffer that doesn't? - **
bump_push_zero**: the same, with the bytes zeroed. Rewind, push again, and read the bytes before zeroing: what's there? (Not checked by the harness; write your own test.) - Temporary scopes:
bump_temp_begin(b)returns a record of the mark andbump_temp_end(t)rewinds to it, likememops_arena_temp_beginand_end. What goes wrong if two scopes end out of order? (Not checked by the harness; write your own test.)
When it fails
FAIL in the scenario: push 4 bytes, align 4: 3 bytes of padding first gave 1, it should give 4
No rounding up at all: the offset is used as it is. The fuzz run is what finds the rest:
FAIL at operation 29 (seed 1): push(12, 2) gave offset 59, which isn't a multiple of 2
debug it: build/dojo/arena/harness --seed 1 --stop 29
with a breakpoint on dojo_stop_here, which runs just before that operation
That one rounds every alignment but 2, which the scenario never asks for. Run the harness under your debugger with those arguments and a breakpoint on dojo_stop_here, which runs just before that operation, and step into bump_push.
If the harness crashes instead, on a write through an address outside the buffer say, the runner runs it again with --trace 1 and prints the operation that was running. --sanitize stops on the bad write itself.
The playground. The kata's harness is dojo/7.arena/harness.rin, and the levels start from dojo/7.arena/level1.rin to level3.rin.
1// Dojo 7, the playground: an arena.2//3// An arena is one buffer and one number: how much of the buffer is used. A4// push rounds that number up to the alignment asked for, hands out the5// address there, and moves the number past the new bytes. Nothing is freed on6// its own. A mark remembers the number, a rewind goes back to it, and a clear7// goes back to 0, and everything pushed since is gone at once, for the cost8// of one store. That is all the memory a frame of a game needs.9//10// This program pushes things of different sizes and alignments into a 64-byte11// buffer, rewinds, fills it to the last byte, and clears it. After each step it12// draws the buffer, 16 bytes to a group: a letter for each byte of a push, -13// for padding, . for free. Each push is filled with its letter, and verify14// checks the arena's invariant and that every push still holds its letter;15// break inside its `if`s to stop on the first thing wrong.16//17// The switch: set align_offsets to 0 and run it again. A push then hands out18// the used count as it is, without rounding it up, and the u32, the u64 and19// the 16-byte vector land at addresses they can't be read from as what they20// are. It's a global, read each time it's used, so a debugger can flip it21// mid-run too: stop on bump_push, set it to 0 in the watch window, and22// continue.23 24import "std/print.rin"25import "std/cstd.rin"26 27align_offsets: b32 = 1;28 29// 64 bytes, starting on a 16-byte boundary, so an offset that is a multiple30// of 16, or of anything less, is an address that is one too.31buffer: struct[align(16)] = {32 bytes: [64]u8;33}34 35bump: struct = {36 buffer: buffer;37 // Bytes before `used` are handed out; bytes after it are free.38 used: u64;39}40 41// `size` bytes at a multiple of `align`, a power of two, or null if they don't42// fit before the end.43bump_push: proc(b: *bump, size: u64, align: u64) -> ^u8 = {44 offset: u64 = b*.used;45 if (align_offsets) {46 offset = (b*.used + align - 1) & ~(align - 1);47 }48 if (offset + size > 64) {49 return null;50 }51 b*.used = offset + size;52 base: ^u8 = cast(b*.buffer.bytes[0]&, ^u8);53 return base + offset;54}55 56bump_mark: proc(b: *bump) -> u64 = {57 return b*.used;58}59 60bump_rewind: proc(b: *bump, mark: u64) -> void = {61 b*.used = mark;62}63 64bump_clear: proc(b: *bump) -> void = {65 b*.used = 0;66}67 68// What the program has pushed and not yet let go of, to draw and to check:69// up to eight pushes, each a letter, an offset, a size and an alignment.70push_record: struct = {71 letter: u8;72 offset: u64;73 size: u64;74 align: u64;75}76 77pushes: struct = {78 live: [8]push_record;79 count: u32;80}81 82push: proc(b: *bump, ps: *pushes, letter: u8, size: u64, align: u64) -> void = {83 before: u64 = b*.used;84 p: ^u8 = bump_push(b, size, align);85 if (p == null) {86 if (size == 1) {87 printfmt(" no room for 1 byte after {} of 64\n", before);88 } else {89 printfmt(" no room for {} bytes after {} of 64\n", size, before);90 }91 return;92 }93 memset(p, cast(letter, i32), size);94 offset: u64 = cast(cast(p, usize) - cast(b*.buffer.bytes[0]&, usize), u64);95 ps*.live[ps*.count] = { .letter = letter, .offset = offset, .size = size, .align = align };96 ps*.count += 1;97 padding: u64 = offset - before;98 printfmt(" offset {}, used {} of 64", offset, b*.used);99 if (padding == 0 && b*.used == 64) {100 printfmt(": the last byte");101 } else if (padding == 1) {102 printfmt(": 1 byte of padding first");103 } else if (padding > 1) {104 printfmt(": {} bytes of padding first", padding);105 }106 printfmt("\n");107}108 109// Rewinding or clearing lets go of every push from after the mark.110let_go: proc(b: *bump, ps: *pushes) -> void = {111 while (ps*.count > 0 && ps*.live[ps*.count - 1].offset >= b*.used) {112 ps*.count -= 1;113 }114}115 116draw: proc(b: *bump, ps: *pushes) -> void = {117 cells: [64]u8 = {};118 for (i: u64 = 0; i < 64; i += 1) {119 cells[i] = i < b*.used ? '-' : '.';120 }121 for (k: u32 = 0; k < ps*.count; k += 1) {122 for (i: u64 = 0; i < ps*.live[k].size; i += 1) {123 cells[ps*.live[k].offset + i] = ps*.live[k].letter;124 }125 }126 text: [72]c8 = {};127 at: u64 = 0;128 for (i: u64 = 0; i < 64; i += 1) {129 if (i > 0 && i % 16 == 0) {130 text[at] = '|';131 at += 1;132 }133 text[at] = cast(cells[i], c8);134 at += 1;135 }136 printf(" %s\n", text[0]&);137}138 139// The invariant: every push is aligned, inside the used bytes, clear of the140// one after it, and still holds its letter.141verify: proc(b: *bump, ps: *pushes) -> void = {142 if (b*.used > 64) {143 printfmt(" verify: used is {}, past the end of the buffer\n", b*.used);144 }145 for (k: u32 = 0; k < ps*.count; k += 1) {146 p: *push_record = ps*.live[k]&;147 name: c8 = cast(p*.letter, c8);148 if (cast(b*.buffer.bytes[p*.offset]&, usize) % p*.align != 0) {149 printf(" verify: %c is at offset %d, not a multiple of %d: its address isn't aligned\n", name,150 cast(p*.offset, i32), cast(p*.align, i32));151 }152 if (p*.offset + p*.size > b*.used) {153 printf(" verify: %c ends at %d, past the used count, %d\n", name, cast(p*.offset + p*.size, i32),154 cast(b*.used, i32));155 }156 if (k + 1 < ps*.count && p*.offset + p*.size > ps*.live[k + 1].offset) {157 printf(" verify: %c runs into %c\n", name, cast(ps*.live[k + 1].letter, c8));158 }159 for (i: u64 = 0; i < p*.size; i += 1) {160 if (b*.buffer.bytes[p*.offset + i] != p*.letter) {161 printf(" verify: %c's bytes have been written over\n", name);162 break;163 }164 }165 }166}167 168main: proc() -> i32 = {169 b: bump = {};170 ps: pushes = {};171 printfmt("a 64-byte buffer, 16 bytes to a group: a letter per byte pushed, - for padding, . for free\n");172 173 printfmt("push a: a u8 flag, 1 byte, align 1\n");174 push(b&, ps&, 'a', 1, 1);175 draw(b&, ps&);176 verify(b&, ps&);177 178 printfmt("push b: a u32 count, 4 bytes, align 4\n");179 push(b&, ps&, 'b', 4, 4);180 draw(b&, ps&);181 verify(b&, ps&);182 183 printfmt("push c: a u64 id, 8 bytes, align 8\n");184 push(b&, ps&, 'c', 8, 8);185 draw(b&, ps&);186 verify(b&, ps&);187 188 printfmt("push d: four f32s for SIMD, 16 bytes, align 16\n");189 push(b&, ps&, 'd', 16, 16);190 draw(b&, ps&);191 verify(b&, ps&);192 193 mark: u64 = bump_mark(b&);194 printfmt("mark: remember used, {}\n", mark);195 196 printfmt("push e: a name, 3 bytes, align 1\n");197 push(b&, ps&, 'e', 3, 1);198 draw(b&, ps&);199 verify(b&, ps&);200 201 printfmt("push f: a u64 time, 8 bytes, align 8\n");202 push(b&, ps&, 'f', 8, 8);203 draw(b&, ps&);204 verify(b&, ps&);205 206 printfmt("rewind to the mark: e and f are gone at once\n");207 bump_rewind(b&, mark);208 let_go(b&, ps&);209 draw(b&, ps&);210 verify(b&, ps&);211 212 printfmt("push g: a path, 40 bytes, align 8\n");213 push(b&, ps&, 'g', 40, 8);214 draw(b&, ps&);215 verify(b&, ps&);216 217 printfmt("push h: a 2x4 matrix of f32s, 32 bytes, align 16\n");218 push(b&, ps&, 'h', 32, 16);219 draw(b&, ps&);220 verify(b&, ps&);221 222 printfmt("push i: one more byte\n");223 push(b&, ps&, 'i', 1, 1);224 draw(b&, ps&);225 verify(b&, ps&);226 227 printfmt("clear: the frame is over, and everything goes at once\n");228 bump_clear(b&);229 let_go(b&, ps&);230 draw(b&, ps&);231 verify(b&, ps&);232 233 printfmt("push j: the next frame's first u64, over a's old bytes\n");234 push(b&, ps&, 'j', 8, 8);235 draw(b&, ps&);236 verify(b&, ps&);237 return 0;238}What play_arena prints. tools/check.py fails the playground if a single byte differs.
a 64-byte buffer, 16 bytes to a group: a letter per byte pushed, - for padding, . for free
push a: a u8 flag, 1 byte, align 1
offset 0, used 1 of 64
a...............|................|................|................
push b: a u32 count, 4 bytes, align 4
offset 4, used 8 of 64: 3 bytes of padding first
a---bbbb........|................|................|................
push c: a u64 id, 8 bytes, align 8
offset 8, used 16 of 64
a---bbbbcccccccc|................|................|................
push d: four f32s for SIMD, 16 bytes, align 16
offset 16, used 32 of 64
a---bbbbcccccccc|dddddddddddddddd|................|................
mark: remember used, 32
push e: a name, 3 bytes, align 1
offset 32, used 35 of 64
a---bbbbcccccccc|dddddddddddddddd|eee.............|................
push f: a u64 time, 8 bytes, align 8
offset 40, used 48 of 64: 5 bytes of padding first
a---bbbbcccccccc|dddddddddddddddd|eee-----ffffffff|................
rewind to the mark: e and f are gone at once
a---bbbbcccccccc|dddddddddddddddd|................|................
push g: a path, 40 bytes, align 8
no room for 40 bytes after 32 of 64
a---bbbbcccccccc|dddddddddddddddd|................|................
push h: a 2x4 matrix of f32s, 32 bytes, align 16
offset 32, used 64 of 64: the last byte
a---bbbbcccccccc|dddddddddddddddd|hhhhhhhhhhhhhhhh|hhhhhhhhhhhhhhhh
push i: one more byte
no room for 1 byte after 64 of 64
a---bbbbcccccccc|dddddddddddddddd|hhhhhhhhhhhhhhhh|hhhhhhhhhhhhhhhh
clear: the frame is over, and everything goes at once
................|................|................|................
push j: the next frame's first u64, over a's old bytes
offset 0, used 8 of 64
jjjjjjjj........|................|................|................
The dojo · block 8
Index, not pointer
dojo/8.index/play_index.rin
A linked list whose nodes live side by side in one vec, and link to each other by index: node 1's next is 0, a position in the vec, not an address. When the vec is full and grows, it moves: the nodes are copied to a bigger place, and the old one is let go. Every index is still right after the move, because an index says where a node is in the vec, wherever the vec is. A pointer says where the node is in memory, so it still points at the old place, and nothing tells you.
That's the whole idea, and it's everywhere in data-oriented code: an ECS keeps tables by index because making a table can move the others, a mesh keeps triangles as three vertex indices, a scene graph keeps a parent index. A freed node goes on a free list and is used again before the vec grows, as in dojo 1.
The invariant: walking next from the head visits every live node once and ends at the tail; each node's prev is the node before it; and a node's index names it from the moment it's made until it's removed, however many times the vec moves in between.
In the course: 2.4 keeps tables by index, because making a table can grow the tables vec and move every table. It's why a pointer into 2.2's columns goes stale when a column grows, and why 2.5 says to keep handles and borrow pointers only to use them.
Watch it: the playground
python tools/dojo.py play index
Memory in the playground is an array of 16 cells, small enough to see whole in a debugger, and the list takes 2 cells of it, then 4, then 8. It keeps node b two ways: by its index, and by a pointer to its node. Then the list fills up and moves, and the program writes to b:
push_back 30 as c: the list is full
the list moves from cells 0..2 to cells 2..6
c is node 2
...
write 21 into b through its index, 1: cell 3 now
nodes 0: 10, prev -, next 1 | 1: 21, prev 0, next 2 | 2: 30, prev 1, next -
list 10 21 30
memory 0..2 old copy: 10 20 | 2..6 the list: 10 21 30 .
After every step verify walks the links and checks that every node holds what the program last wrote to it. Put a breakpoint on broken, which it calls when something is wrong.
Then break on the line that writes 21 into b, set use_index to 0 in the watch window, and continue. The write goes through the kept pointer, into the old copy, and the list never sees it:
write 21 into b through what was kept, cell 1: where b was before the move
...
memory 0..2 old copy: 10 21 | 2..6 the list: 10 20 30 .
verify: node 1 holds 20, but the program last wrote 21 to it
Nothing crashes, and AddressSanitizer stays quiet too: the old copy is still memory the program owns, as an arena's old blocks are until the arena is cleared. The write is simply lost.
Do reps: the kata
You write list. The idea is in list_remove: the two neighbours link to each other, past the node, and the node's index goes on the free list:
list_remove: proc(l: *list, node: u32) -> option<u64> = {
n: *list_node = list_live(l, node);
if (n == null) {
return option<u64>none();
}
if (n*.prev != list_none) {
list_at(l, n*.prev)*.next = n*.next;
} else {
l*.head = n*.next;
}
if (n*.next != list_none) {
list_at(l, n*.next)*.prev = n*.prev;
} else {
l*.tail = n*.prev;
}
and the kata holds itself to the playground's lesson, in list_insert_after:
// A new node between `node` and the one after it; list_none if `node` isn't
// in the list. Every pointer into the nodes is taken after list_take, which
// can move them.
list_insert_after: proc(l: *list, node: u32, value: u64) -> u32 = {
plus list_init, list_push_back, list_push_front, list_get, list_len, and the links to walk the list by: list_head, list_tail, list_next and list_prev. A link past either end is list_none, the largest u32, which no node ever gets.
| level | you write | target |
|---|---|---|
| 1 | list_remove, in an otherwise finished file | 3:00 |
| 2 | every proc body; the fields and signatures are given | 12:00 |
| 3 | the whole file, from a blank page | 15:00 |
python tools/dojo.py start index 2
python tools/dojo.py index
The harness runs the playground's steps as a scenario, then 20,000 random pushes, inserts, removes, reads and writes against a model that keeps the live nodes in list order. After every operation it walks your list both ways, list_head to the end by list_next, and list_tail back by list_prev, so a link left wrong is caught by the operation that left it. The list grows to about a thousand nodes, moving many times, and every node is checked by the index it was given. A freed index must be handed out again before a new one is made. The seeds give the same runs as rust-dod's dojo.
rin notes. Rust's kata has Option<u32> links; here a link is a u32, and list_none stands for none, as UINT32_MAX would in C. list_get returns a pointer, null for a node not in the list, to read or to change the value; Rust's kata has get and get_mut.
Level 4, variations. The same harness, different insides:
- The free list through the nodes: a freed node's
nextis the next free node, and the list keepsfirst_free: u32. Nofreevec. - **A pointer kept across
list_take**, which fails on purpose: inlist_insert_after, takelist_live(l, node)once, beforelist_take, and write through it after. Which operation does the harness stop at, and where did the write go? Then put it back. - A list walker:
list_each(l, visit), withvisitaproc(node: u32, value: u64) -> void, called head to tail. (Not checked by the harness; write your own test.)
When it fails
FAIL at operation 17 (seed 1): walking head to tail, list_next(3) gives 2, but 3 is the last of 3 nodes
debug it: build/dojo/index/harness --seed 1 --stop 17
with a breakpoint on dojo_stop_here, which runs just before that operation
A link pointing somewhere it shouldn't is usually a node from the free list that kept its old links, or a neighbour that wasn't told about a change: an insert at the tail that didn't move the tail, a remove that fixed next but not prev, or a write through a pointer taken before the nodes moved. Run the harness under your debugger with those arguments and a breakpoint on dojo_stop_here, which runs just before that operation, and look at the nodes the message names.
If the harness crashes instead, on a node past the end of the vec say, the runner runs it again with --trace 1 and prints the operation that was running, or says the crash came in the scenario, after the last step printed.
An index alone can't tell a removed node from the new node that took its place: a is node 0, and so is e after it. That's dojo 1's problem, and its answer, a generation beside the index, works here unchanged.
The playground. The kata's harness is dojo/8.index/harness.rin, and the levels start from dojo/8.index/level1.rin to level3.rin.
1// Dojo 8, the playground: index, not pointer.2//3// A linked list whose nodes live side by side in one array and link to each4// other by index: a node's next is "node 2", not an address. When the array is5// full it moves: the nodes are copied to a place twice the size, and the old6// place is let go. An index is a position in the array, so it finds its node7// after the move. A pointer is a place in memory, so it still points where8// the node was.9//10// Memory here is an array of 16 cells, small enough to see whole in a11// debugger's watch window. The list takes 2 cells, then 4, then 8: cells 0..2,12// then 2..6, then 6..14. Each move leaves the old copy where it was, the way13// freed memory keeps its bytes until something else is put there.14//15// The program keeps node b two ways: its index, and a pointer to it. Then the16// list moves, and the program writes to b. verify() runs after every step: it17// checks the links, and that every node holds what the program last wrote to18// it. Put a breakpoint on `broken` to stop where it finds something wrong.19//20// The switch: use_index. Break on the line that writes 21 into b, set it to 021// in the watch window, and continue. The write goes through the kept pointer22// to the old copy, and verify reports it lost.23 24import "std/print.rin"25import "std/option.rin"26 27use_index: b32 = 1;28 29// No node: the link past either end.30no_node: u32 = 0xffffffff'u32;31 32node: struct = {33 value: u64;34 prev: u32;35 next: u32;36 alive: b32;37}38 39list: struct = {40 // All of memory. Node i is in cell base + i, and the list owns cells41 // base..base + capacity.42 memory: [16]node;43 base: u32;44 capacity: u32;45 // Nodes made so far are 0..made; the freed ones wait in free.46 made: u32;47 free: [8]u32;48 free_count: u32;49 head: u32;50 tail: u32;51}52 53// Where node `index` is now.54list_node: proc(l: *list, index: u32) -> *node = {55 return l*.memory[l*.base + index]&;56}57 58// A node for `value`: a freed one, or a new one, moving the list to twice the59// room first when it's full.60list_take: proc(l: *list, value: u64) -> u32 = {61 index: u32 = 0;62 if (l*.free_count > 0) {63 l*.free_count -= 1;64 index = l*.free[l*.free_count];65 } else {66 if (l*.made == l*.capacity) {67 to: u32 = l*.base + l*.capacity;68 printfmt(" the list moves from cells {}..{} to cells {}..{}\n", l*.base, to, to, to + 2 * l*.capacity);69 for (i: u32 = 0; i < l*.made; i += 1) {70 l*.memory[to + i] = l*.memory[l*.base + i];71 }72 l*.base = to;73 l*.capacity *= 2;74 }75 l*.made += 1;76 index = l*.made - 1;77 }78 list_node(l, index)* = { .value = value, .prev = no_node, .next = no_node, .alive = 1 };79 return index;80}81 82list_push_back: proc(l: *list, value: u64) -> u32 = {83 index: u32 = list_take(l, value);84 list_node(l, index)*.prev = l*.tail;85 if (l*.tail != no_node) {86 list_node(l, l*.tail)*.next = index;87 } else {88 l*.head = index;89 }90 l*.tail = index;91 return index;92}93 94list_push_front: proc(l: *list, value: u64) -> u32 = {95 index: u32 = list_take(l, value);96 list_node(l, index)*.next = l*.head;97 if (l*.head != no_node) {98 list_node(l, l*.head)*.prev = index;99 } else {100 l*.tail = index;101 }102 l*.head = index;103 return index;104}105 106list_insert_after: proc(l: *list, at: u32, value: u64) -> u32 = {107 index: u32 = list_take(l, value);108 next: u32 = list_node(l, at)*.next;109 list_node(l, index)* = { .value = value, .prev = at, .next = next, .alive = 1 };110 list_node(l, at)*.next = index;111 if (next != no_node) {112 list_node(l, next)*.prev = index;113 } else {114 l*.tail = index;115 }116 return index;117}118 119list_remove: proc(l: *list, index: u32) -> void = {120 n: *node = list_node(l, index);121 if (n*.prev != no_node) {122 list_node(l, n*.prev)*.next = n*.next;123 } else {124 l*.head = n*.next;125 }126 if (n*.next != no_node) {127 list_node(l, n*.next)*.prev = n*.prev;128 } else {129 l*.tail = n*.prev;130 }131 n*.alive = 0;132 l*.free[l*.free_count] = index;133 l*.free_count += 1;134}135 136// A link, or - for none.137print_link: proc(link: u32) -> void = {138 if (link == no_node) {139 printfmt("-");140 } else {141 printfmt("{}", link);142 }143}144 145// A value, or - for none.146print_value: proc(value: option<u64>) -> void = {147 if (value.has_value) {148 printfmt("{}", value.value);149 } else {150 printfmt("-");151 }152}153 154draw: proc(l: *list) -> void = {155 printfmt(" nodes ");156 for (i: u32 = 0; i < l*.made; i += 1) {157 if (i > 0) {158 printfmt(" | ");159 }160 n: *node = l*.memory[l*.base + i]&;161 if (n*.alive) {162 printfmt("{}: {}, prev ", i, n*.value);163 print_link(n*.prev);164 printfmt(", next ");165 print_link(n*.next);166 } else {167 printfmt("{}: free", i);168 }169 }170 printfmt("\n list ");171 count: u32 = 0;172 for (at: u32 = l*.head; at != no_node && count < l*.made; at = l*.memory[l*.base + at].next) {173 if (count > 0) {174 printfmt(" ");175 }176 printfmt("{}", l*.memory[l*.base + at].value);177 count += 1;178 }179 printfmt("\n");180}181 182// Every place the list has lived, oldest first, and what's in each cell.183draw_memory: proc(l: *list) -> void = {184 printfmt(" memory ");185 from: u32 = 0;186 size: u32 = 2;187 while (from <= l*.base) {188 if (from > 0) {189 printfmt(" | ");190 }191 if (from == l*.base) {192 printfmt("{}..{} the list:", from, from + size);193 } else {194 printfmt("{}..{} old copy:", from, from + size);195 }196 for (c: u32 = from; c < from + size; c += 1) {197 if (l*.memory[c].alive) {198 printfmt(" {}", l*.memory[c].value);199 } else {200 printfmt(" .");201 }202 }203 from += size;204 size *= 2;205 }206 printfmt("\n");207}208 209// verify calls this when it finds something wrong, then says what.210broken: proc() -> void = {211 printfmt(" verify: ");212}213 214// The list's invariant, and the program's: walking from the head reaches215// every live node once, each node's prev is the one before it, the walk ends216// at the tail, and every node holds what the program last wrote to it.217verify: proc(l: *list, wrote: []option<u64>) -> void = {218 before: u32 = no_node;219 linked: u32 = 0;220 for (at: u32 = l*.head; at != no_node; at = l*.memory[l*.base + at].next) {221 n: *node = l*.memory[l*.base + at]&;222 if (!n*.alive) {223 broken();224 printfmt("node {} is linked, but it's free\n", at);225 }226 if (n*.prev != before) {227 broken();228 printfmt("node {} comes after ", at);229 print_link(before);230 printfmt(", but its prev is ");231 print_link(n*.prev);232 printfmt("\n");233 }234 linked += 1;235 if (linked > l*.made) {236 broken();237 printfmt("the links go round in a loop\n");238 break;239 }240 before = at;241 }242 if (l*.tail != before) {243 broken();244 printfmt("the list ends at ");245 print_link(before);246 printfmt(", but the tail is ");247 print_link(l*.tail);248 printfmt("\n");249 }250 alive: u32 = 0;251 for (i: u32 = 0; i < l*.made; i += 1) {252 n: *node = l*.memory[l*.base + i]&;253 holds: option<u64> = option<u64>none();254 if (n*.alive) {255 holds = option<u64>some(n*.value);256 alive += 1;257 }258 want: option<u64> = wrote[i];259 if (holds.has_value != want.has_value || holds.value != want.value) {260 broken();261 printfmt("node {} holds ", i);262 print_value(holds);263 printfmt(", but the program last wrote ");264 print_value(want);265 printfmt(" to it\n");266 }267 }268 if (linked != alive) {269 broken();270 printfmt("{} nodes are linked, but {} are alive\n", linked, alive);271 }272}273 274main: proc() -> i32 = {275 l: list = {};276 l.capacity = 2;277 l.head = no_node;278 l.tail = no_node;279 // What the program last wrote to each node, for verify.280 wrote: [8]option<u64> = {};281 printfmt("a list with room for 2 nodes, in cells 0..2 of memory\n");282 283 printfmt("push_back 10 as a\n");284 a: u32 = list_push_back(l&, 10);285 wrote[a] = option<u64>some(10);286 printfmt(" a is node {}\n", a);287 draw(l&);288 verify(l&, wrote);289 290 printfmt("push_back 20 as b\n");291 b: u32 = list_push_back(l&, 20);292 wrote[b] = option<u64>some(20);293 printfmt(" b is node {}\n", b);294 draw(l&);295 verify(l&, wrote);296 297 kept: *node = list_node(l&, b);298 kept_cell: u32 = l.base + b;299 printfmt("keep b's index, {}, and where it is, cell {}, which is what a pointer holds: the same, until the list moves\n",300 b, kept_cell);301 302 printfmt("push_back 30 as c: the list is full\n");303 c: u32 = list_push_back(l&, 30);304 wrote[c] = option<u64>some(30);305 printfmt(" c is node {}\n", c);306 draw(l&);307 verify(l&, wrote);308 309 if (use_index) {310 printfmt("write 21 into b through its index, {}: cell {} now\n", b, l.base + b);311 list_node(l&, b)*.value = 21;312 } else {313 printfmt("write 21 into b through what was kept, cell {}: where b was before the move\n", kept_cell);314 kept*.value = 21;315 }316 wrote[b] = option<u64>some(21);317 draw(l&);318 draw_memory(l&);319 verify(l&, wrote);320 321 printfmt("push_front 5 as d\n");322 d: u32 = list_push_front(l&, 5);323 wrote[d] = option<u64>some(5);324 printfmt(" d is node {}\n", d);325 draw(l&);326 verify(l&, wrote);327 328 printfmt("remove a: node {} goes on the free list\n", a);329 list_remove(l&, a);330 wrote[a] = option<u64>none();331 draw(l&);332 verify(l&, wrote);333 334 printfmt("insert 25 after b as e\n");335 e: u32 = list_insert_after(l&, b, 25);336 wrote[e] = option<u64>some(25);337 printfmt(" e is node {}: a's node, from the free list\n", e);338 draw(l&);339 verify(l&, wrote);340 341 printfmt("push_back 40 as f: the list is full again\n");342 f: u32 = list_push_back(l&, 40);343 wrote[f] = option<u64>some(40);344 printfmt(" f is node {}\n", f);345 draw(l&);346 draw_memory(l&);347 verify(l&, wrote);348 349 printfmt("read b through its index, {}: {}\n", b, list_node(l&, b)*.value);350 return 0;351}What play_index prints. tools/check.py fails the playground if a single byte differs.
a list with room for 2 nodes, in cells 0..2 of memory
push_back 10 as a
a is node 0
nodes 0: 10, prev -, next -
list 10
push_back 20 as b
b is node 1
nodes 0: 10, prev -, next 1 | 1: 20, prev 0, next -
list 10 20
keep b's index, 1, and where it is, cell 1, which is what a pointer holds: the same, until the list moves
push_back 30 as c: the list is full
the list moves from cells 0..2 to cells 2..6
c is node 2
nodes 0: 10, prev -, next 1 | 1: 20, prev 0, next 2 | 2: 30, prev 1, next -
list 10 20 30
write 21 into b through its index, 1: cell 3 now
nodes 0: 10, prev -, next 1 | 1: 21, prev 0, next 2 | 2: 30, prev 1, next -
list 10 21 30
memory 0..2 old copy: 10 20 | 2..6 the list: 10 21 30 .
push_front 5 as d
d is node 3
nodes 0: 10, prev 3, next 1 | 1: 21, prev 0, next 2 | 2: 30, prev 1, next - | 3: 5, prev -, next 0
list 5 10 21 30
remove a: node 0 goes on the free list
nodes 0: free | 1: 21, prev 3, next 2 | 2: 30, prev 1, next - | 3: 5, prev -, next 1
list 5 21 30
insert 25 after b as e
e is node 0: a's node, from the free list
nodes 0: 25, prev 1, next 2 | 1: 21, prev 3, next 0 | 2: 30, prev 0, next - | 3: 5, prev -, next 1
list 5 21 25 30
push_back 40 as f: the list is full again
the list moves from cells 2..6 to cells 6..14
f is node 4
nodes 0: 25, prev 1, next 2 | 1: 21, prev 3, next 0 | 2: 30, prev 0, next 4 | 3: 5, prev -, next 1 | 4: 40, prev 2, next -
list 5 21 25 30 40
memory 0..2 old copy: 10 20 | 2..6 old copy: 25 21 30 5 | 6..14 the list: 25 21 30 5 40 . . .
read b through its index, 1: 21
The dojo · block 9
Interning
dojo/9.interning/play_interning.rin
An interner gives each distinct key a small number, an id: the next one, 0, 1, 2, the first time it sees the key, and the same one every time after. Find it, or make it. The ids are dense, so they index plain vecs: the key of id 3 is keys[3], and whatever else belongs to that key can live at index 3 of its own vec.
Behind it is a hash table, written here by hand: a row of slots, each empty or holding an id. A key's hash says which slot to look in first. If that one holds another key, look in the next one, and the next, wrapping round at the end, until the key or an empty slot turns up. This is linear probing. A full table would make those runs long, and an absent key's run endless, so the table doubles before it's more than 3/4 full, and every id is put back where its key's hash says in the bigger table: a different slot, since the hash now picks from twice as many.
The invariant: every id is in exactly one slot, and a walk from its key's hash reaches that slot before it reaches an empty one; the table is never more than 3/4 full; and ids are 0 to len - 1, in the order their keys first came.
In the course: 2.4's kayan_table_for is an interner. Its keys are signatures and its ids are table indices, with std's hashmap doing the lookup, and std's hashmap is this same design: linear probing, a power of two, grown before 3/4. Here you write it. 2.6 then makes the lookup rare, by remembering where each add and remove leads, and dojo 10 practises that.
Watch it: the playground
python tools/dojo.py play interning
It interns the signatures of a few kinds of thing in a game, in a table of 8 slots that grows to 16, and prints every slot it looks at:
intern rock, signature 7
slots looked at: 4 5 6; new: id 3, in slot 6
table 1:turret 4:drone 5:player 6:rock, 4 of 8 slots used
...
intern bird, signature 15
slots looked at: 1 2; not there
7 ids would fill more than 3/4 of 8 slots: the table grows to 16, and every id
goes where its key's hash says now
slots looked at: 9 10; new: id 6, in slot 10
table 4:rock 5:player 7:chest 8:tank 9:turret 10:bird 12:drone, 7 of 16 slots used
After every step verify checks the invariant; put a breakpoint on broken, which it calls when something is wrong.
Then break on the first interner_intern, set mix_hash to 0 in the watch window, and continue. The hash is now the key itself, so the first slot is the key's lowest bits. Those are the components nearly every kind has, position, sprite and health, since they were made first, so every kind starts at the same slot and the runs pile up:
intern chest, signature 135
slots looked at: 7 0 1 2 3; new: id 4, in slot 3
...
slots looked at, in all: 41
against 17 with the hash. Nothing is wrong, only slow, so verify stays quiet. Flip it later instead, with ids already in the table, and verify complains straight away: the ids are where the old hash put them, and the new hash looks somewhere else. A table's hash can't change under it.
Do reps: the kata
You write interner, its table included. The idea is in interner_intern:
interner_intern: proc(t: *interner, key: u64) -> u32 = {
at: u64 = interner_slot_of(t, key);
if (vec<u32>at(t*.slots&, at)* != 0) {
return vec<u32>at(t*.slots&, at)* - 1;
}
if ((t*.keys.length + 1) * 4 > t*.slots.length * 3) {
interner_grow(t);
at = interner_slot_of(t, key);
}
and the API around it:
interner_find: proc(t: *interner, key: u64) -> option<u32> = {
interner_key: proc(t: *interner, id: u32) -> u64 = {
plus interner_init and interner_len. A slot holds id + 1, so that 0 can mean empty and key 0 is a key like any other.
| level | you write | target |
|---|---|---|
| 1 | interner_intern, in an otherwise finished file | 3:00 |
| 2 | every proc body; the fields and signatures are given | 10:00 |
| 3 | the whole file, from a blank page | 12:00 |
python tools/dojo.py start interning 2
python tools/dojo.py interning
The harness runs the playground's steps as a scenario, then 20,000 random interns and finds against a model, std's hashmap and a vec of the keys in order. The keys are the kinds an ECS's tables meet: small numbers, 0 among them; signatures with a few bits set; keys that differ only in their high bits; and any u64 at all. Ids must be dense, in the order keys were first interned, and never change; interner_find must agree with interner_intern; and interner_key must give the key back. It checks what the interner does, not how fast: an interner with no mixing in its hash passes, and the playground shows what it costs. The seeds give the same runs as rust-dod's dojo.
Level 4, variations. The same harness, different insides:
- The hash beside the id: two u32s a slot, the id and the low bits of its key's hash, so a run compares hashes and reads
keysonly when they match, andinterner_grownever hashes a key again. - Robin Hood: on the way along a run, an id that is closer to its home slot than the new one gives its slot up and moves on instead. The longest run gets much shorter. How does
interner_findstop early on an absent key? - Grow at 1/2: what happens to the slots looked at in the playground, and to the memory?
When it fails
FAIL in the scenario: find turret gave none, it should give 2
A key that was interned and can't be found usually means interner_grow copied the slots across instead of putting each id back by its key's hash: in a table twice the size, a key's run starts somewhere else. A run that steps off the end without wrapping stops the program with rin runtime: vec index out of bounds, and the runner runs it again with --trace 1 to show the operation that was running. If the run stops and never ends, the table filled up: a lookup for a key that isn't there walks round and round, never meeting an empty slot to stop at. Break into it in your debugger and look at slots.
In the fuzz run, a failure says which operation and seed, as in the other blocks; run the harness under your debugger with the --seed and --stop it gives, and a breakpoint on dojo_stop_here.
The playground. The kata's harness is dojo/9.interning/harness.rin, and the levels start from dojo/9.interning/level1.rin to level3.rin.
1// Dojo 9, the playground: interning.2//3// An interner gives each distinct key a small id, the first time it sees the4// key, and the same id every time after: find it, or make it. The ECS does5// this with signatures, to find the table for a set of components. Behind it6// is a hash table: a row of slots, each empty or holding an id. A key's hash7// says which slot to look in first; if that one holds another key, look in the8// next, and the next, until the key or an empty slot turns up. A table more9// than 3/4 full makes those runs long, so it doubles before that, and every id10// is put back where its key's hash says in the bigger table.11//12// Here the table has 8 slots, then 16, small enough to see whole in a13// debugger's watch window. The keys are the signatures of a few kinds of14// thing in a game: a bit per component, with the common components first.15//16// verify() runs after every step: every id must be reachable from its key's17// hash without crossing an empty slot, and the table at most 3/4 full. Put a18// breakpoint on `broken` to stop where it finds something wrong.19//20// The switch: mix_hash. Break on the first intern, set it to 0 in the watch21// window, and continue. The hash is then the key itself, and the slot its22// lowest bits: the components every kind of thing has. Watch the runs get23// long, and the count of slots looked at at the end. Flip it later instead,24// with ids already in the table, and verify says why a table's hash can't25// change under it.26 27import "std/print.rin"28 29mix_hash: b32 = 1;30 31// The kinds of thing, and their signatures, which main fills in.32kind_names: [7]^const char = { "player", "drone", "turret", "rock", "chest", "tank", "bird" };33kind_keys: [7]u64 = {};34 35name: proc(key: u64) -> ^const char = {36 for (i: u32 = 0; i < 7; i += 1) {37 if (kind_keys[i] == key) {38 return kind_names[i];39 }40 }41 return "?";42}43 44// splitmix64's finisher, or, with the switch off, the key as it is.45hash: proc(key: u64) -> u64 = {46 if (!mix_hash) {47 return key;48 }49 x: u64 = key;50 x ^= x shr 30;51 x *= 0xbf58476d1ce4e5b9'u64;52 x ^= x shr 27;53 x *= 0x94d049bb133111eb'u64;54 return x ^ (x shr 31);55}56 57interner: struct = {58 // slots[0..capacity] is the table: 0 is empty, anything else an id + 1.59 slots: [16]u32;60 capacity: u32;61 keys: [12]u64;62 length: u32;63 // Slots looked at, in all.64 probes: u32;65}66 67// Where `key` is, or the empty slot where it would go, printing the slots it68// looks at on the way.69interner_slot_of: proc(t: *interner, key: u64) -> u32 = {70 mask: u32 = t*.capacity - 1;71 at: u32 = cast(hash(key), u32) & mask;72 printfmt(" slots looked at:");73 while (1) {74 printfmt(" {}", at);75 t*.probes += 1;76 s: u32 = t*.slots[at];77 if (s == 0 || t*.keys[s - 1] == key) {78 return at;79 }80 at = (at + 1) & mask;81 }82 return at;83}84 85// The key's id + 1, or 0 if it isn't there.86interner_find: proc(t: *interner, key: u64) -> u32 = {87 return t*.slots[interner_slot_of(t, key)];88}89 90interner_intern: proc(t: *interner, key: u64) -> u32 = {91 at: u32 = interner_slot_of(t, key);92 if (t*.slots[at] != 0) {93 printfmt("; found: id {}\n", t*.slots[at] - 1);94 return t*.slots[at] - 1;95 }96 if ((t*.length + 1) * 4 > t*.capacity * 3) {97 printfmt("; not there\n");98 printfmt(" {} ids would fill more than 3/4 of {} slots: the table grows to {}, and every id\n",99 t*.length + 1, t*.capacity, t*.capacity * 2);100 printfmt(" goes where its key's hash says now\n");101 t*.capacity *= 2;102 for (i: u32 = 0; i < 16; i += 1) {103 t*.slots[i] = 0;104 }105 mask: u32 = t*.capacity - 1;106 for (id: u32 = 0; id < t*.length; id += 1) {107 i: u32 = cast(hash(t*.keys[id]), u32) & mask;108 while (t*.slots[i] != 0) {109 i = (i + 1) & mask;110 }111 t*.slots[i] = id + 1;112 }113 at = interner_slot_of(t, key);114 }115 id: u32 = t*.length;116 t*.keys[t*.length] = key;117 t*.length += 1;118 t*.slots[at] = id + 1;119 printfmt("; new: id {}, in slot {}\n", id, at);120 return id;121}122 123draw: proc(t: *interner) -> void = {124 printfmt(" table ");125 first: b32 = 1;126 for (i: u32 = 0; i < t*.capacity; i += 1) {127 if (t*.slots[i] != 0) {128 if (!first) {129 printfmt(" ");130 }131 printfmt("{}:{}", i, name(t*.keys[t*.slots[i] - 1]));132 first = 0;133 }134 }135 printfmt(", {} of {} slots used\n", t*.length, t*.capacity);136}137 138// verify calls this when it finds something wrong, then says what.139broken: proc() -> void = {140 printfmt(" verify: ");141}142 143// The table's invariant: each id is in one slot, reachable from its key's144// hash without crossing an empty slot, and the table is at most 3/4 full.145verify: proc(t: *interner) -> void = {146 mask: u32 = t*.capacity - 1;147 used: u32 = 0;148 for (i: u32 = 0; i < t*.capacity; i += 1) {149 if (t*.slots[i] != 0) {150 used += 1;151 }152 }153 if (used != t*.length) {154 broken();155 printfmt("{} slots are used, but there are {} ids\n", used, t*.length);156 }157 if (t*.length * 4 > t*.capacity * 3) {158 broken();159 printfmt("{} ids in {} slots is more than 3/4 full\n", t*.length, t*.capacity);160 }161 for (id: u32 = 0; id < t*.length; id += 1) {162 key: u64 = t*.keys[id];163 start: u32 = cast(hash(key), u32) & mask;164 at: u32 = start;165 found: b32 = 0;166 for (step: u32 = 0; step < t*.capacity; step += 1) {167 if (t*.slots[at] == id + 1) {168 found = 1;169 break;170 }171 if (t*.slots[at] == 0) {172 break;173 }174 at = (at + 1) & mask;175 }176 if (!found) {177 broken();178 printfmt("id {}, {}, can't be found: its run from slot {} meets the empty slot {} first\n",179 id, name(key), start, at);180 }181 }182}183 184main: proc() -> i32 = {185 // Components, a bit each. The ones nearly everything has were made first,186 // so they have the lowest bits.187 position: u64 = 1'u64 shl 0;188 sprite: u64 = 1'u64 shl 1;189 health: u64 = 1'u64 shl 2;190 velocity: u64 = 1'u64 shl 3;191 player_tag: u64 = 1'u64 shl 4;192 ai: u64 = 1'u64 shl 5;193 damage: u64 = 1'u64 shl 6;194 loot: u64 = 1'u64 shl 7;195 player: u64 = position | sprite | health | velocity | player_tag;196 drone: u64 = position | sprite | health | velocity | ai;197 turret: u64 = position | sprite | health | ai | damage;198 rock: u64 = position | sprite | health;199 chest: u64 = position | sprite | health | loot;200 tank: u64 = position | sprite | health | velocity | ai | damage;201 bird: u64 = position | sprite | health | velocity;202 kind_keys[0] = player;203 kind_keys[1] = drone;204 kind_keys[2] = turret;205 kind_keys[3] = rock;206 kind_keys[4] = chest;207 kind_keys[5] = tank;208 kind_keys[6] = bird;209 210 t: interner = {};211 t.capacity = 8;212 printfmt("an interner with 8 slots; a slot holds an id + 1, or 0 when it's empty\n");213 214 printfmt("intern player, signature {}\n", player);215 _ = interner_intern(t&, player);216 draw(t&);217 verify(t&);218 219 printfmt("intern drone, signature {}\n", drone);220 _ = interner_intern(t&, drone);221 draw(t&);222 verify(t&);223 224 printfmt("intern turret, signature {}\n", turret);225 _ = interner_intern(t&, turret);226 draw(t&);227 verify(t&);228 229 printfmt("intern drone again\n");230 _ = interner_intern(t&, drone);231 verify(t&);232 233 printfmt("intern rock, signature {}\n", rock);234 _ = interner_intern(t&, rock);235 draw(t&);236 verify(t&);237 238 printfmt("intern chest, signature {}\n", chest);239 _ = interner_intern(t&, chest);240 draw(t&);241 verify(t&);242 243 printfmt("find tank, signature {}\n", tank);244 found: u32 = interner_find(t&, tank);245 if (found != 0) {246 printfmt("; found: id {}\n", found - 1);247 } else {248 printfmt("; not there\n");249 }250 verify(t&);251 252 printfmt("intern tank\n");253 _ = interner_intern(t&, tank);254 draw(t&);255 verify(t&);256 257 printfmt("intern bird, signature {}\n", bird);258 _ = interner_intern(t&, bird);259 draw(t&);260 verify(t&);261 262 printfmt("find turret\n");263 found = interner_find(t&, turret);264 if (found != 0) {265 printfmt("; found: id {}, and id {}'s key is {}\n", found - 1, found - 1, t.keys[found - 1]);266 } else {267 printfmt("; not there\n");268 }269 verify(t&);270 271 printfmt("slots looked at, in all: {}\n", t.probes);272 return 0;273}What play_interning prints. tools/check.py fails the playground if a single byte differs.
an interner with 8 slots; a slot holds an id + 1, or 0 when it's empty
intern player, signature 31
slots looked at: 5; new: id 0, in slot 5
table 5:player, 1 of 8 slots used
intern drone, signature 47
slots looked at: 4; new: id 1, in slot 4
table 4:drone 5:player, 2 of 8 slots used
intern turret, signature 103
slots looked at: 1; new: id 2, in slot 1
table 1:turret 4:drone 5:player, 3 of 8 slots used
intern drone again
slots looked at: 4; found: id 1
intern rock, signature 7
slots looked at: 4 5 6; new: id 3, in slot 6
table 1:turret 4:drone 5:player 6:rock, 4 of 8 slots used
intern chest, signature 135
slots looked at: 7; new: id 4, in slot 7
table 1:turret 4:drone 5:player 6:rock 7:chest, 5 of 8 slots used
find tank, signature 111
slots looked at: 7 0; not there
intern tank
slots looked at: 7 0; new: id 5, in slot 0
table 0:tank 1:turret 4:drone 5:player 6:rock 7:chest, 6 of 8 slots used
intern bird, signature 15
slots looked at: 1 2; not there
7 ids would fill more than 3/4 of 8 slots: the table grows to 16, and every id
goes where its key's hash says now
slots looked at: 9 10; new: id 6, in slot 10
table 4:rock 5:player 7:chest 8:tank 9:turret 10:bird 12:drone, 7 of 16 slots used
find turret
slots looked at: 9; found: id 2, and id 2's key is 103
slots looked at, in all: 17
The dojo · block 10
Memoized edges
dojo/10.edges/play_edges.rin
In an archetype ECS, an entity that gains a component moves to the table whose signature has one more bit. Finding that table by its signature is a search. But a game moves entities between the same few tables over and over: a drone catches fire, the fire goes out, it catches fire again. So each table remembers where each component leads, once a move has found it:
node 3 {position velocity health} --+burning--> node 4
node 4 {position velocity health burning} --burning--> node 3
The first move from 3 with burning searches, and writes the answer on both nodes: an add edge one way and a remove edge back. Every move after that, in either direction, follows an edge. Remembering the answer to a slow question is memoizing, and the nodes and their edges are the archetype graph that flecs is built around.
The invariant: every known edge leads to the node one component away, and that node has the edge back. Each move that changes a signature searches at most once; after that, both directions are edges.
In the course: 2.6 adds add_edges and remove_edges to each table, and moves entities along them.
Watch it: the playground
python tools/dojo.py play edges
A drone gains and loses components, eight nodes at most. Each move says where it went and how, and the graph is drawn twice:
catches fire: node 3 -> 4 {position velocity health burning}, by a lookup
the fire goes out: node 4 -> 3 {position velocity health}, by an edge
catches fire again: node 3 -> 4 {position velocity health burning}, by an edge
...
node 3 {position velocity health}: +burning 4, -velocity 5, -health 2
...
moves that looked up a node by signature: 7; that followed an edge: 4
Step through it: break on with and watch the two edges get written after find_or_make. verify runs after every move and prints any edge that leads to the wrong node or has no edge back. Then flip the switch at the top, memo, to 0 and run it again. The drone goes to the same nodes, but nothing is remembered: eleven moves, eleven lookups, and every node drawn with no edges known. With a few hundred tables and a game moving thousands of entities a frame, that's the difference.
Do reps: the kata
You write graph. The move that adds a component:
graph_with: proc(g: *graph, node: u32, id: u32) -> u32 = {
signature: u64 = graph_signature(g, node);
if ((signature & graph_bit(id)) != 0) {
return node;
}
edge: u32 = vec<graph_node>at(g*.nodes&, cast(node, u64))*.add[id];
if (edge != 0) {
g*.hits += 1;
return edge - 1;
}
to: u32 = graph_find_or_make(g, signature | graph_bit(id));
// graph_find_or_make can make a node, and the vec can move them all to
// grow, so both pointers are taken after it.
vec<graph_node>at(g*.nodes&, cast(node, u64))*.add[id] = to + 1;
vec<graph_node>at(g*.nodes&, cast(to, u64))*.remove[id] = node + 1;
return to;
}
and its mirror, graph_without, plus graph_init (node 0 is the empty signature), graph_signature, graph_len, graph_lookups (moves that searched) and graph_hits (moves that followed an edge). The edges are stored as one more than the node, so a zeroed node knows none, 64 each way:
graph_node: struct = {
signature: u64;
// add[id] is one more than the node with component id added, once a move
// has looked it up, so 0 means not known yet; remove[id] is the same, with
// it removed.
add: [64]u32;
remove: [64]u32;
}
| level | you write | target |
|---|---|---|
| 1 | graph_with, in an otherwise finished file | 3:00 |
| 2 | every proc body; the fields and signatures are given | 10:00 |
| 3 | the whole file, from a blank page | 12:00 |
python tools/dojo.py start edges 2
python tools/dojo.py edges
The harness runs the playground's moves as a scenario, then 20,000 random moves from random nodes, with ids up to 63, against a model that knows each node's signature and which edges a graph that writes both directions knows by now. After every move it checks the node, its signature, and both counters: a known edge is a hit, any other move that changes the signature is one lookup, and a move that changes nothing (adding what the node has) is neither. The seeds give the same runs as rust-dod's dojo.
rin notes. The comment in graph_with is the one to remember: a pointer into nodes taken before graph_find_or_make can point at memory the vec has moved out of, so the edges are written through fresh pointers after it. That's dojo 8's lesson, and 2.6's tables have the same comment.
Level 4, variations. The same harness, different insides:
- One hash map of edges,
hashmap<u64, u32>keyed bynode * 128 + id * 2 + adding, instead of two 64-wide arrays per node. Which takes less memory with 300 tables, and which is faster to follow? - A hash map for the slow path:
graph_find_or_makesearches every node. Keep ahashmap<u64, u32>from signature to node, as 2.4 does, or build one in dojo 9. - Removing a node. An ECS that deletes empty tables must forget every edge into them. Which nodes have edges into node
n? (Not checked by the harness; write your own test.)
When it fails
FAIL in the scenario: lookups and hits gave 5 1, it should give 4 2
Five lookups where the model counts four: the fire going out searched, instead of following the edge back that catching fire should have written. The edge back is the one people forget, or write on the wrong node: the move from 3 to 4 sets node 3's add edge and node 4's remove edge.
The scenario's ids stop at 3. The fuzz run's go up to 63, and a failure there says which operation and seed:
FAIL at operation 9 (seed 1): with(3, 63) gave node 6, which is {2 3 31}; from {2 3} it should be {2 3 63}
debug it: build/dojo/edges/harness --seed 1 --stop 9
with a breakpoint on dojo_stop_here, which runs just before that operation
That's a bit made in 32 bits, 1 shl id on a u32: bit 63 lands on bit 31. Run the harness under your debugger with those arguments and a breakpoint on dojo_stop_here, and step into the move. --sanitize reports the shift itself, as undefined behaviour.
The playground. The kata's harness is dojo/10.edges/harness.rin, and the levels start from dojo/10.edges/level1.rin to level3.rin.
1// Dojo 10, the playground: memoized edges.2//3// In an archetype ECS, an entity that gains a component moves to the table4// whose signature has one more bit. Finding that table by its signature is a5// search. But a game moves entities between the same few tables over and6// over, so each table remembers where each component leads: an add edge per7// component, and a remove edge back. The tables and their edges form a graph,8// the archetype graph of the course's 2.6.9//10// Here the nodes are bare signatures, eight at most, over four components. A11// drone gains and loses components, and each move prints the node it went to12// and whether it took a lookup or followed an edge. verify runs after every13// move: every known edge must lead to the node with that one component added14// (or removed) and have its edge back, and the drone's node must have the15// components the drone has. Put a breakpoint inside its failure branch to stop16// where the graph goes wrong.17//18// The switch: set memo to 0 and run it again. Nothing is remembered: the19// drone goes to the same nodes, but every move is a lookup. It's a global,20// read on every move, so a debugger can flip it mid-run too: stop on the21// first `lose` in main, set it to 0 in the watch window, and continue.22 23import "std/print.rin"24 25memo: b32 = 1;26 27names: [4]^const char = { "position", "velocity", "health", "burning" };28position: const u32 = 0;29velocity: const u32 = 1;30health: const u32 = 2;31burning: const u32 = 3;32 33bit: proc(id: u32) -> u64 = {34 return 1'u64 shl cast(id, u64);35}36 37node: struct = {38 signature: u64;39 // add[id] is one more than the node with component id added, once a move40 // has found it, so 0 means not known yet; remove[id] the same, with it41 // removed.42 add: [4]u32;43 remove: [4]u32;44}45 46graph: struct = {47 nodes: [8]node;48 count: u32;49 lookups: u32;50 edges: u32;51}52 53// The slow path: search every node for the signature, and make one if none54// has it.55find_or_make: proc(g: *graph, signature: u64) -> u32 = {56 g*.lookups += 1;57 for (i: u32 = 0; i < g*.count; i += 1) {58 if (g*.nodes[i].signature == signature) {59 return i;60 }61 }62 g*.nodes[g*.count] = { .signature = signature };63 g*.count += 1;64 return g*.count - 1;65}66 67with: proc(g: *graph, from: u32, id: u32) -> u32 = {68 if (memo && g*.nodes[from].add[id] != 0) {69 g*.edges += 1;70 return g*.nodes[from].add[id] - 1;71 }72 to: u32 = find_or_make(g, g*.nodes[from].signature | bit(id));73 if (memo) {74 g*.nodes[from].add[id] = to + 1;75 g*.nodes[to].remove[id] = from + 1;76 }77 return to;78}79 80without: proc(g: *graph, from: u32, id: u32) -> u32 = {81 if (memo && g*.nodes[from].remove[id] != 0) {82 g*.edges += 1;83 return g*.nodes[from].remove[id] - 1;84 }85 to: u32 = find_or_make(g, g*.nodes[from].signature & ~bit(id));86 if (memo) {87 g*.nodes[from].remove[id] = to + 1;88 g*.nodes[to].add[id] = from + 1;89 }90 return to;91}92 93print_names: proc(signature: u64) -> void = {94 printfmt("{");95 first: b32 = 1;96 for (id: u32 = 0; id < 4; id += 1) {97 if ((signature & bit(id)) != 0) {98 if (!first) {99 printfmt(" ");100 }101 printfmt("{}", names[id]);102 first = 0;103 }104 }105 printfmt("}");106}107 108drone: struct = {109 node: u32;110 // What the drone has, kept apart from the graph: verify checks the two111 // agree.112 has: u64;113}114 115// Every edge leads to the node one component away, and comes back; and the116// drone's node is the one with its components.117verify: proc(g: *graph, d: *drone) -> void = {118 for (n: u32 = 0; n < g*.count; n += 1) {119 here: *node = g*.nodes[n]&;120 for (id: u32 = 0; id < 4; id += 1) {121 if (here*.add[id] != 0) {122 there: *node = g*.nodes[here*.add[id] - 1]&;123 if (there*.signature != (here*.signature | bit(id)) || there*.remove[id] != n + 1) {124 printfmt("verify: node {}'s +{} edge leads to node {} ", n, names[id], here*.add[id] - 1);125 print_names(there*.signature);126 printfmt("\n");127 }128 }129 if (here*.remove[id] != 0) {130 there: *node = g*.nodes[here*.remove[id] - 1]&;131 if (there*.signature != (here*.signature & ~bit(id)) || there*.add[id] != n + 1) {132 printfmt("verify: node {}'s -{} edge leads to node {} ", n, names[id], here*.remove[id] - 1);133 print_names(there*.signature);134 printfmt("\n");135 }136 }137 }138 }139 if (g*.nodes[d*.node].signature != d*.has) {140 printfmt("verify: the drone has ");141 print_names(d*.has);142 printfmt(", but its node {} is ", d*.node);143 print_names(g*.nodes[d*.node].signature);144 printfmt("\n");145 }146}147 148report: proc(g: *graph, d: *drone, what: ^const char, from: u32, lookups: u32) -> void = {149 printfmt("{}: node {} -> {} ", what, from, d*.node);150 print_names(g*.nodes[d*.node].signature);151 if (g*.lookups > lookups) {152 printfmt(", by a lookup\n");153 } else {154 printfmt(", by an edge\n");155 }156 verify(g, d);157}158 159gain: proc(g: *graph, d: *drone, id: u32, what: ^const char) -> void = {160 from: u32 = d*.node;161 lookups: u32 = g*.lookups;162 d*.node = with(g, d*.node, id);163 d*.has = d*.has | bit(id);164 report(g, d, what, from, lookups);165}166 167lose: proc(g: *graph, d: *drone, id: u32, what: ^const char) -> void = {168 from: u32 = d*.node;169 lookups: u32 = g*.lookups;170 d*.node = without(g, d*.node, id);171 d*.has = d*.has & ~bit(id);172 report(g, d, what, from, lookups);173}174 175draw: proc(g: *graph) -> void = {176 for (n: u32 = 0; n < g*.count; n += 1) {177 here: *node = g*.nodes[n]&;178 printfmt(" node {} ", n);179 print_names(here*.signature);180 printfmt(":");181 known: u32 = 0;182 for (id: u32 = 0; id < 4; id += 1) {183 if (here*.add[id] != 0) {184 printfmt("{} +{} {}", known > 0 ? "," : "", names[id], here*.add[id] - 1);185 known += 1;186 }187 }188 for (id: u32 = 0; id < 4; id += 1) {189 if (here*.remove[id] != 0) {190 printfmt("{} -{} {}", known > 0 ? "," : "", names[id], here*.remove[id] - 1);191 known += 1;192 }193 }194 if (known == 0) {195 printfmt(" no edges known");196 }197 printfmt("\n");198 }199}200 201main: proc() -> i32 = {202 g: graph = {};203 g.count = 1;204 d: drone = {};205 printfmt("the drone starts at node 0 {}\n", "{}");206 gain(g&, d&, position, "gains position");207 gain(g&, d&, velocity, "gains velocity");208 gain(g&, d&, health, "gains health");209 gain(g&, d&, burning, "catches fire");210 lose(g&, d&, burning, "the fire goes out");211 gain(g&, d&, burning, "catches fire again");212 lose(g&, d&, burning, "the fire goes out");213 printfmt("the graph so far:\n");214 draw(g&);215 lose(g&, d&, velocity, "stops");216 gain(g&, d&, burning, "catches fire");217 gain(g&, d&, velocity, "moves again");218 lose(g&, d&, burning, "the fire goes out");219 printfmt("the graph:\n");220 draw(g&);221 printfmt("moves that looked up a node by signature: {}; that followed an edge: {}\n", g.lookups, g.edges);222 return 0;223}What play_edges prints. tools/check.py fails the playground if a single byte differs.
the drone starts at node 0 {}
gains position: node 0 -> 1 {position}, by a lookup
gains velocity: node 1 -> 2 {position velocity}, by a lookup
gains health: node 2 -> 3 {position velocity health}, by a lookup
catches fire: node 3 -> 4 {position velocity health burning}, by a lookup
the fire goes out: node 4 -> 3 {position velocity health}, by an edge
catches fire again: node 3 -> 4 {position velocity health burning}, by an edge
the fire goes out: node 4 -> 3 {position velocity health}, by an edge
the graph so far:
node 0 {}: +position 1
node 1 {position}: +velocity 2, -position 0
node 2 {position velocity}: +health 3, -velocity 1
node 3 {position velocity health}: +burning 4, -health 2
node 4 {position velocity health burning}: -burning 3
stops: node 3 -> 5 {position health}, by a lookup
catches fire: node 5 -> 6 {position health burning}, by a lookup
moves again: node 6 -> 4 {position velocity health burning}, by a lookup
the fire goes out: node 4 -> 3 {position velocity health}, by an edge
the graph:
node 0 {}: +position 1
node 1 {position}: +velocity 2, -position 0
node 2 {position velocity}: +health 3, -velocity 1
node 3 {position velocity health}: +burning 4, -velocity 5, -health 2
node 4 {position velocity health burning}: -velocity 6, -burning 3
node 5 {position health}: +velocity 3, +burning 6
node 6 {position health burning}: +velocity 4, -burning 5
moves that looked up a node by signature: 7; that followed an edge: 4
The dojo · block 11
A reflection walk
dojo/11.reflect/play_reflect.rin
A reflect record lists a struct's fields: each one's name, where its bytes start in the struct, how many there are, and its type, with the type's own record when it's a struct too. rin's compiler writes one for every struct: lamp<> is lamp's. With records, one piece of code can list, read, write and compare the fields of any type it's handed, by name: an inspector, an editor, a save file, a network diff. It never names the type. It walks the bytes by the record:
"at.y" in a lamp: lamp's field at, offset 8 -> vec2's field y, offset 4 -> bytes 12..16
Each offset counts from the start of the field's own struct, so a path adds them up as it goes down. And an offset is the record's to say, not something to work out from the sizes before it: a u8 followed by an f32 leaves 3 bytes of padding, and the f32 starts at 4.
The invariant: the walk reads and writes exactly the bytes the compiler gave each field. Read by path, every field is what the struct holds.
In the course: 2.7 prints any entity field by field from records. The Advanced lesson 5 builds an inspector, and an editor's write by name.
Watch it: the playground
python tools/dojo.py play reflect
It prints a lamp's record, walks the lamp by it, changes fields by path, and diffs the lamp with how it was:
a lamp's record: 20 bytes
on: offset 0, size 1, u8
power: offset 4, size 4, f32
at: offset 8, size 8, a struct
flicker: offset 16, size 4, f32, hidden
...
set at.x to 5.00: true
diff with the lamp before: power at.x
set flicker to 0.50: true
diff with the lamp before: power at.x
The walk reads the lamp through its bytes. Step through it: break on find and watch offset add up. verify runs after every step and checks every f32 the walk reads against the field read directly, l.power and so on. Then flip the switch at the top, use_record_offsets, to 0 and run it again. The walk adds up the sizes before each field instead, as if nothing had padding: power is read from offset 1, three padding bytes and one byte of the real power, and writing at.x lands on power. The record printed above it still says 4.
Do reps: the kata
You write the walk, over the compiler's records, reflect and reflect_field from std/reflect.rin. First, every field a tool shows, by path, walking into the ones that are structs:
walk_leaves: proc(arena: *memops_arena, type: *const reflect, prefix: ^const char, base: u64,
out: *vec<walk_leaf>) -> void = {
fields: ^const reflect_field = reflect_fields(type);
for (i: u64 = 0; i < reflect_field_count(type); i += 1) {
f: *const reflect_field = fields[i]&;
if (reflect_field_has_attr(f, "hidden")) {
continue;
}
leaf: walk_leaf = { .offset = base + f*.offset, .size = f*.size };
path: fmt = fmt_to(leaf.path.text);
fmt_cstr(path&, prefix);
fmt_cstr(path&, f*.name);
if (reflect_is_struct(f*.info)) {
fmt_char(path&, '.');
walk_leaves(arena, f*.info, path.data, leaf.offset, out);
} else {
_ = vec<walk_leaf>append(arena, out, leaf);
}
}
}
and on it and on a walk down one path, walk_find, the procs the harness calls:
walk_paths: proc(arena: *memops_arena, type: *const reflect) -> vec<walk_path> = {
walk_diff: proc(arena: *memops_arena, type: *const reflect, a: []const u8, b: []const u8) -> vec<walk_path> = {
walk_get_f32: proc(type: *const reflect, bytes: []const u8, path: ^const char) -> option<f32> = {
walk_set_f32: proc(type: *const reflect, bytes: []u8, path: ^const char, value: f32) -> b32 = {
A path is a walk_path, 48 characters at most, ending in a zero. A hidden field is left out of walk_paths and walk_diff, with everything inside it, but a path still reaches it: hiding is for tools that list, not for code that asks by name. walk_get_f32 and walk_set_f32 refuse a path that names a struct, a u8, or nothing.
| level | you write | target |
|---|---|---|
| 1 | walk_leaves, the walk, in an otherwise finished file | 4:00 |
| 2 | every proc body; the types and signatures are given | 15:00 |
| 3 | the whole file, from a blank page | 18:00 |
python tools/dojo.py start reflect 2
python tools/dojo.py reflect
The harness's test types have the traps in them: lamp has padding after on: u8, room has a lamp inside it (paths three deep, offsets added twice), and a hidden draft: vec2. It runs the playground's steps as a scenario, then 20,000 random reads, writes and diffs of a room by path, each checked against the same fields reached directly, r.light.at.x and so on. The paths include ones that name a struct, a u8, or nothing: light.atx.x, corner.x.y, light.. The seeds give the same runs as rust-dod's dojo.
rin notes. rin's compiler writes the records; Rust's kata gets them from a macro in its harness, the way 2.7 makes them, and has paths return Vec<String> where this one fills a vec<walk_path> from the arena it's given. The harness gives each fuzz operation a temporary arena mark, so what a walk allocates is gone after it. A field's bytes are cut out with a slice, bytes[at..at + 4], so a wrong offset past the end stops a Debug build instead of reading past the value.
Level 4, variations. The same harness, different insides:
- **
walk_findby recursion**: find the field the first name names, and recurse into its record with the rest of the path. - Paths without a vec of leaves: walk with a callback, a
proc(path: ^const char, offset: u64, f: *const reflect_field) -> void, and letwalk_pathsandwalk_diffeach pass their own. - A tiny text format: write a value as
path = valuelines and read it back. That's a save file and a network message. (Not checked by the harness; write your own test.)
When it fails
FAIL at operation 4 (seed 1): set_f32("light.", -50.25) returns true, it should return false
debug it: build/dojo/reflect/harness --seed 1 --stop 4
with a breakpoint on dojo_stop_here, which runs just before that operation
The name after the dot is empty, and it matched a field anyway: a name compare that stops at the end of the shorter of the two finds range for nothing, and at for atx. Run the harness under your debugger with those arguments and a breakpoint on dojo_stop_here, then step through walk_find, watching each name and offset. If it crashes instead, the runner runs it again with --trace 1 and prints the operation that was running.
The playground. The kata's harness is dojo/11.reflect/harness.rin, and the levels start from dojo/11.reflect/level1.rin to level3.rin.
1// Dojo 11, the playground: a reflection walk.2//3// A reflect record lists a struct's fields: each one's name, where its bytes4// start in the struct, how many there are, and its type, with the type's own5// record if it's a struct too. With records, one piece of code can list,6// read, write and compare the fields of any type it's handed, by name, the way7// an inspector, an editor, a save file or a network diff does. It never names8// the type: it walks the bytes by the record.9//10// rin's compiler writes a record for every struct: lamp<> is lamp's. The walk11// below reads the lamp through its bytes, and verify runs after every step:12// every f32 the walk reads by path must be what the lamp holds, read as13// l.power and so on. Put a breakpoint inside its failure branch to stop where14// they differ.15//16// The switch: set use_record_offsets to 0 and run it again. The walk adds up17// the sizes of the fields before each one instead, as if structs had no18// padding. That's right for `x` and `y`, and wrong from `power` on: it starts19// at 4, after 3 bytes of padding, not at 1. It's a global, read on every step,20// so a debugger can flip it mid-run too: stop on the first `set` in main, set21// it to 0 in the watch window, and continue.22 23import "std/print.rin"24import "std/fmt.rin"25import "std/reflect.rin"26 27use_record_offsets: b32 = 1;28 29vec2: struct = { x: f32; y: f32; }30lamp: struct = { on: u8; power: f32; at: vec2; flicker: f32 @ "hidden"; }31 32// Where field i starts in its struct: what the record says or, with the33// switch off, the sizes of the fields before it added up.34offset_in: proc(type: *const reflect, i: u64) -> u64 = {35 fields: ^const reflect_field = reflect_fields(type);36 if (use_record_offsets) {37 return fields[i].offset;38 }39 sum: u64 = 0;40 for (k: u64 = 0; k < i; k += 1) {41 sum += fields[k].size;42 }43 return sum;44}45 46read_f32: proc(row: []u8, at: u64) -> f32 = {47 field: []u8 = row[at..at + 4];48 return cast(field[0]&, *f32)*;49}50 51// Every field a tool shows, depth first, printed with its value. Struct52// fields are walked into, and hidden ones left out.53show: proc(type: *const reflect, row: []u8, prefix: ^const char, base: u64) -> void = {54 fields: ^const reflect_field = reflect_fields(type);55 for (i: u64 = 0; i < type*.count; i += 1) {56 f: *const reflect_field = fields[i]&;57 if (reflect_field_has_attr(f, "hidden")) {58 continue;59 }60 text: [48]c8 = {};61 path: fmt = fmt_to(text);62 fmt_cstr(path&, prefix);63 fmt_cstr(path&, f*.name);64 at: u64 = base + offset_in(type, i);65 if (reflect_is_struct(f*.info)) {66 fmt_char(path&, '.');67 show(f*.info, row, path.data, at);68 } else if (reflect_cstr_equal(f*.type, "f32")) {69 printf(" %s: %.2f\n", path.data, cast(read_f32(row, at), f64));70 } else {71 printfmt(" {}: {}\n", path.data, row[at]);72 }73 }74}75 76// The paths of the fields a tool shows whose bytes differ, each after a space.77diff: proc(type: *const reflect, a: []u8, b: []u8, prefix: ^const char, base: u64) -> void = {78 fields: ^const reflect_field = reflect_fields(type);79 for (i: u64 = 0; i < type*.count; i += 1) {80 f: *const reflect_field = fields[i]&;81 if (reflect_field_has_attr(f, "hidden")) {82 continue;83 }84 text: [48]c8 = {};85 path: fmt = fmt_to(text);86 fmt_cstr(path&, prefix);87 fmt_cstr(path&, f*.name);88 at: u64 = base + offset_in(type, i);89 if (reflect_is_struct(f*.info)) {90 fmt_char(path&, '.');91 diff(f*.info, a, b, path.data, at);92 } else {93 for (k: u64 = at; k < at + f*.size; k += 1) {94 if (a[k] != b[k]) {95 printfmt(" {}", path.data);96 break;97 }98 }99 }100 }101}102 103// Is `name` exactly path[start..end]?104name_is: proc(name: ^const char, path: ^const char, start: u64, end: u64) -> b32 = {105 i: u64 = 0;106 while (start + i < end) {107 if (name[i] != path[start + i]) {108 return 0;109 }110 i += 1;111 }112 return name[i] == 0;113}114 115// The field a path names, and where its bytes start; null if none has it.116find: proc(type: *const reflect, path: ^const char, offset: *u64) -> *const reflect_field = {117 found: *const reflect_field = null;118 record: *const reflect = type;119 start: u64 = 0;120 more: b32 = 1;121 offset* = 0;122 while (more) {123 end: u64 = start;124 while (path[end] != 0 && path[end] != '.') {125 end += 1;126 }127 found = null;128 for (i: u64 = 0; i < reflect_field_count(record); i += 1) {129 if (found == null && name_is(reflect_fields(record)[i].name, path, start, end)) {130 found = reflect_fields(record)[i]&;131 offset* += offset_in(record, i);132 }133 }134 if (found == null) {135 return null;136 }137 record = found*.info;138 more = path[end] != 0;139 start = end + 1;140 }141 return found;142}143 144get_f32: proc(type: *const reflect, row: []u8, path: ^const char, value: *f32) -> b32 = {145 at: u64 = 0;146 f: *const reflect_field = find(type, path, at&);147 if (f == null || !reflect_cstr_equal(f*.type, "f32")) {148 return 0;149 }150 value* = read_f32(row, at);151 return 1;152}153 154set_f32: proc(type: *const reflect, row: []u8, path: ^const char, value: f32) -> b32 = {155 at: u64 = 0;156 f: *const reflect_field = find(type, path, at&);157 if (f == null || !reflect_cstr_equal(f*.type, "f32")) {158 return 0;159 }160 field: []u8 = row[at..at + 4];161 cast(field[0]&, *f32)* = value;162 return 1;163}164 165bytes_of: proc(l: *lamp) -> []u8 = {166 return cast(l, ^u8)[0..lamp<>.size];167}168 169// Every f32 of the lamp read by path, against the same field read directly.170// The first that differs is printed, and how many do.171verify: proc(l: *lamp) -> void = {172 paths: [4]^const char = { "power", "at.x", "at.y", "flicker" };173 held: [4]f32 = { l*.power, l*.at.x, l*.at.y, l*.flicker };174 wrong: u32 = 0;175 first: u64 = 0;176 for (i: u64 = 0; i < 4; i += 1) {177 walked: f32 = 0.0;178 if (!get_f32(lamp<>&, bytes_of(l), paths[i], walked&) || walked != held[i]) {179 if (wrong == 0) {180 first = i;181 }182 wrong += 1;183 }184 }185 if (wrong > 0) {186 walked: f32 = 0.0;187 printf("verify: the walk reads %s as ", paths[first]);188 if (get_f32(lamp<>&, bytes_of(l), paths[first], walked&)) {189 printf("%.2f", cast(walked, f64));190 } else {191 printf("nothing");192 }193 printf(", the lamp holds %.2f; %u of 4 f32s are wrong\n", cast(held[first], f64), wrong);194 }195}196 197set: proc(l: *lamp, path: ^const char, value: f32) -> void = {198 done: b32 = set_f32(lamp<>&, bytes_of(l), path, value);199 printf("set %s to %.2f: %s\n", path, cast(value, f64), done ? "true" : "false");200 verify(l);201}202 203main: proc() -> i32 = {204 info: *const reflect = lamp<>&;205 printfmt("a lamp's record: {} bytes\n", info*.size);206 for (i: u64 = 0; i < info*.count; i += 1) {207 f: *const reflect_field = reflect_fields(info)[i]&;208 printfmt(" {}: offset {}, size {}, ", f*.name, f*.offset, f*.size);209 if (reflect_is_struct(f*.info)) {210 printfmt("a struct");211 } else {212 printfmt("{}", f*.type);213 }214 if (reflect_field_has_attr(f, "hidden")) {215 printfmt(", hidden");216 }217 printfmt("\n");218 }219 l: lamp = {};220 memset(l&, 0, cast(lamp<>.size, usize));221 l.on = 1;222 l.power = 60.0;223 l.at = { .x = 3.0, .y = 4.0 };224 l.flicker = 0.25;225 printfmt("the lamp, walked by its record:\n");226 show(lamp<>&, bytes_of(l&), "", 0);227 verify(l&);228 before: lamp = l;229 set(l&, "power", 75.0);230 set(l&, "at.x", 5.0);231 printfmt("diff with the lamp before:");232 diff(lamp<>&, bytes_of(before&), bytes_of(l&), "", 0);233 printfmt("\n");234 set(l&, "flicker", 0.5);235 printfmt("diff with the lamp before:");236 diff(lamp<>&, bytes_of(before&), bytes_of(l&), "", 0);237 printfmt("\n");238 set(l&, "on", 0.0);239 set(l&, "at", 1.0);240 set(l&, "at.z", 1.0);241 printfmt("the lamp, walked by its record:\n");242 show(lamp<>&, bytes_of(l&), "", 0);243 return 0;244}What play_reflect prints. tools/check.py fails the playground if a single byte differs.
a lamp's record: 20 bytes
on: offset 0, size 1, u8
power: offset 4, size 4, f32
at: offset 8, size 8, a struct
flicker: offset 16, size 4, f32, hidden
the lamp, walked by its record:
on: 1
power: 60.00
at.x: 3.00
at.y: 4.00
set power to 75.00: true
set at.x to 5.00: true
diff with the lamp before: power at.x
set flicker to 0.50: true
diff with the lamp before: power at.x
set on to 0.00: false
set at to 1.00: false
set at.z to 1.00: false
the lamp, walked by its record:
on: 1
power: 75.00
at.x: 5.00
at.y: 4.00
The dojo · block 12
A watermark
dojo/12.watermark/play_watermark.rin
A query keeps the list of tables it matches, so a system doesn't search for them every frame. Tables are only ever added, at the end, and never removed. So the query remembers how many tables it has looked at, its watermark, and an update looks only at the tables past it:
tables 0 P 1 PV 2 PVH | 3 PVHB 4 H
^ movers has checked 3: an update looks at 3 and 4
A frame with no new tables costs one comparison, and each query looks at each table once, ever. It works for anything that only grows at the end: a log read by several readers, a list of events, the lines of a file being written.
The invariant: a query lists exactly the tables below its watermark that match, each once, in order, and it has looked at each of them once.
In the course: 3.1 builds the query and its checked count, and 3.2 updates every system's query at the start of every frame, which is what makes the watermark matter.
Watch it: the playground
python tools/dojo.py play watermark
It makes tables a few at a time, and updates two queries: movers, all {Position Velocity}, and healthy, all {Health}, none {Burning}. After each step it draws the tables, and under them what each query has looked at, with a | at its watermark:
update movers: it looks at tables 3 and 4 only
table 0 1 2 3 4
has P PV PVH PVHB H
movers - yes yes yes - | checked 5, scanned 5, tables 1 2 3
healthy | checked 0, scanned 0, tables none
scanned counts every table an update has looked at. With the watermark it always equals checked. Then verify checks both queries against the tables and prints what's wrong; break inside its if to stop when it fires.
Step through it: break on query_update and watch from, checked and found. Then flip the switch at the top, use_watermark, to 0, and run it again. Every update starts from table 0, and the second update of movers, with nothing new, finds tables 1 and 2 again:
movers - yes yes | checked 3, scanned 6, tables 1 2 1 2
verify: movers lists tables 1 2 1 2; it should list 1 2
verify: movers has looked at 6 tables to check 3; each should be looked at once
A system walking that list would run twice over the same entities, and the work grows with every frame. The switch is a global, read on every update, so a debugger can flip it mid-run too: stop on query_update, set it to 0 in the watch window, and continue.
Do reps: the kata
You write query:
query: struct = {
arena: *memops_arena;
all: u64;
none: u64;
// The tables that match, by index, in the order they were found.
tables: vec<u32>;
// The watermark: tables 0 to checked - 1 have been looked at.
checked: u64;
// How many tables query_update has looked at, ever. With the watermark,
// each table once.
scanned: u64;
}
and its update, which is given the world's tables as their signatures:
query_update: proc(q: *query, signatures: []u64) -> void = {
plus query_init(q, arena, all, none), query_tables (the matching tables, in increasing order), query_checked and query_scanned. A table matches when its signature has every bit of all and no bit of none.
| level | you write | target |
|---|---|---|
| 1 | query_update, in an otherwise finished file | 2:00 |
| 2 | every proc body; the fields and signatures are given | 5:00 |
| 3 | the whole file, from a blank page | 7:00 |
python tools/dojo.py start watermark 2
python tools/dojo.py watermark
The harness runs the playground's steps as a scenario, then 20,000 random operations: tables made with random signatures, four queries with random masks updated at random, and now and then one replaced by a new query, which must start from nothing. A model keeps, for each query, the tables that match among the ones it has been updated over. After every update the query must have checked the whole list, list exactly the model's tables, and have scanned each table once. Some none masks have two bits, so "none of these" and "not all of these" give different answers. The seeds give the same runs as rust-dod's dojo.
Level 4, variations. The same harness, different insides:
- **A while loop, and
scannedadded once**:scannedgrows by the number of new tables, which is known before the loop starts. - Push, don't poll: flecs does the reverse. When the world makes a table, it tells every query, and
query_updatehas nothing left to do. What does the world have to keep, and what happens to a query made after the table? (Not checked by the harness; write your own test.) - A table that can go away: the list keeps a tombstone where a removed table was, a signature no query matches. What must a query do with a table it already lists? (Not checked by the harness.)
When it fails
The scenario catches the common mistakes first:
FAIL in the scenario: make tables 3 and 4, update movers: tables gave 1 2 0, it should give 1 2 3
That's a count of the new tables pushed as if it were the table's index: a loop that counts from 0 over the tables past the watermark has to add checked back. A failure in the fuzz run says where:
FAIL at operation 27 (seed 1): after query_update, query 1 (all {0 5}, none {0 6}): it lists 1, the model 0; the first difference is at position 0: table 1, where the model has nothing
debug it: build/dojo/watermark/harness --seed 1 --stop 27
with a breakpoint on dojo_stop_here, which runs just before that operation
That one tested none the way it tested all, (signature & none) != none, which is "not every bit of none" rather than "no bit of it". It only differs when none has more than one bit. Run the harness under your debugger with those arguments and a breakpoint on dojo_stop_here, then step into query_update and look at the table it lists.
The playground. The kata's harness is dojo/12.watermark/harness.rin, and the levels start from dojo/12.watermark/level1.rin to level3.rin.
1// Dojo 12, the playground: a watermark.2//3// A query keeps the list of tables it matches, so a system doesn't search for4// them every frame. Tables are only ever added, at the end, and never removed.5// So the query remembers how many it has looked at, its watermark, and an6// update looks only at the tables past it:7//8// tables 0 P 1 PV 2 PVH | 3 PVHB 4 H9// ^ movers has checked 3: an update looks at 3 and 410//11// A frame with no new tables costs one comparison, and each query looks at12// each table once, ever. This is the course's query (3.1), with the world's13// tables reduced to their signatures.14//15// This program makes tables a few at a time and updates two queries. After16// each step it draws the tables, and under them what each query has looked17// at, with a | at its watermark. Then verify checks each query and says what18// is wrong, if anything; break inside its `if` to stop there. To see an19// update in a debugger, break on `query_update` and watch `from`, `checked`20// and `found`.21//22// The switch: set use_watermark to 0 and run it again. Every update looks at23// every table, from 0, so a query lists its tables again each time. It's a24// global, read on every update, so a debugger can flip it mid-run too: stop25// on `query_update`, set it to 0 in the watch window, and continue.26 27import "std/print.rin"28 29use_watermark: b32 = 1;30 31// Four components, a bit each.32position: u64 = 1;33velocity: u64 = 2;34health: u64 = 4;35burning: u64 = 8;36 37// The world's tables, as far as a query cares: a signature each, added at the38// end and never removed.39tables: struct = {40 signatures: [8]u64;41 count: u64;42}43 44tables_make: proc(t: *tables, signature: u64) -> void = {45 t*.signatures[t*.count] = signature;46 t*.count += 1;47}48 49query: struct = {50 name: ^const char;51 all: u64;52 none: u64;53 // The matching tables, by index.54 found: [8]u32;55 found_count: u64;56 // The watermark: tables 0 to checked - 1 have been looked at.57 checked: u64;58 // How many times an update has looked at a table.59 scanned: u64;60}61 62matches: proc(signature: u64, all: u64, none: u64) -> b32 = {63 return (signature & all) == all && (signature & none) == 0;64}65 66query_make: proc(name: ^const char, all: u64, none: u64) -> query = {67 q: query = {};68 q.name = name;69 q.all = all;70 q.none = none;71 return q;72}73 74query_update: proc(q: *query, t: *tables) -> void = {75 from: u64 = 0;76 if (use_watermark) {77 from = q*.checked;78 }79 for (ti: u64 = from; ti < t*.count; ti += 1) {80 q*.scanned += 1;81 if (matches(t*.signatures[ti], q*.all, q*.none)) {82 q*.found[q*.found_count] = cast(ti, u32);83 q*.found_count += 1;84 }85 }86 q*.checked = t*.count;87}88 89query_found: proc(q: *query) -> []u32 = {90 return { .data = q*.found[0]&, .length = q*.found_count };91}92 93print_list: proc(items: []u32) -> void = {94 if (items.length == 0) {95 printfmt("none");96 }97 for (i: u64 = 0; i < items.length; i += 1) {98 if (i > 0) {99 printfmt(" ");100 }101 printfmt("{}", items[i]);102 }103}104 105// The invariant: a query lists the tables below its watermark that match,106// each once, in order, and it has looked at each of those tables once.107verify: proc(t: *tables, q: *query) -> void = {108 want: [8]u32 = {};109 want_count: u64 = 0;110 for (ti: u64 = 0; ti < q*.checked; ti += 1) {111 if (matches(t*.signatures[ti], q*.all, q*.none)) {112 want[want_count] = cast(ti, u32);113 want_count += 1;114 }115 }116 same: b32 = want_count == q*.found_count;117 for (i: u64 = 0; same && i < want_count; i += 1) {118 same = want[i] == q*.found[i];119 }120 if (!same) {121 printfmt(" verify: {} lists tables ", q*.name);122 print_list(query_found(q));123 printfmt("; it should list ");124 print_list({ .data = want[0]&, .length = want_count });125 printfmt("\n");126 }127 if (q*.scanned != q*.checked) {128 printfmt(" verify: {} has looked at {} tables to check {}; each should be looked at once\n",129 q*.name, q*.scanned, q*.checked);130 }131}132 133// The components in a signature, a letter each, padded to `width` columns.134print_letters: proc(signature: u64, width: i32) -> void = {135 names: ^const char = "PVHB";136 letters: [5]c8 = {};137 n: u64 = 0;138 for (i: u64 = 0; i < 4; i += 1) {139 if (((signature shr i) & 1) == 1) {140 letters[n] = names[i];141 n += 1;142 }143 }144 printf("%-*s", width, letters[0]&);145}146 147has_found: proc(q: *query, table: u64) -> b32 = {148 for (i: u64 = 0; i < q*.found_count; i += 1) {149 if (cast(q*.found[i], u64) == table) {150 return 1;151 }152 }153 return 0;154}155 156// The tables, then a row per query: yes or - for each table it has looked at,157// and a | at its watermark.158show: proc(t: *tables, a: *query, b: *query) -> void = {159 printf(" table");160 for (ti: u64 = 0; ti < t*.count; ti += 1) {161 printf(ti == 0 ? " %u" : " %u", cast(ti, u32));162 }163 printf("\n has ");164 for (ti: u64 = 0; ti < t*.count; ti += 1) {165 print_letters(t*.signatures[ti], ti + 1 < t*.count ? 5 : 0);166 }167 printf("\n");168 queries: [2]*query = {};169 queries[0] = a;170 queries[1] = b;171 for (k: u64 = 0; k < 2; k += 1) {172 q: *query = queries[k];173 printf(" %-10s", q*.name);174 for (ti: u64 = 0; ti < q*.checked; ti += 1) {175 printf(has_found(q, ti) ? "yes " : "- ");176 }177 printf("| ");178 for (ti: u64 = q*.checked; ti < t*.count; ti += 1) {179 printf(" ");180 }181 printfmt("checked {}, scanned {}, tables ", q*.checked, q*.scanned);182 print_list(query_found(q));183 printfmt("\n");184 }185 verify(t, a);186 verify(t, b);187}188 189main: proc() -> i32 = {190 t: tables = {};191 movers: query = query_make("movers", position | velocity, 0);192 healthy: query = query_make("healthy", health, burning);193 printf("components: P position, V velocity, H health, B burning\n");194 printf("movers: all {P V}, none {}; healthy: all {H}, none {B}\n");195 196 printfmt("make tables 0 to 2\n");197 tables_make(t&, position);198 tables_make(t&, position | velocity);199 tables_make(t&, position | velocity | health);200 show(t&, movers&, healthy&);201 202 printfmt("update movers: it looks at tables 0 to 2\n");203 query_update(movers&, t&);204 show(t&, movers&, healthy&);205 206 printfmt("update movers again: no new tables, so it looks at nothing\n");207 query_update(movers&, t&);208 show(t&, movers&, healthy&);209 210 printfmt("make tables 3 and 4\n");211 tables_make(t&, position | velocity | health | burning);212 tables_make(t&, health);213 show(t&, movers&, healthy&);214 215 printfmt("update movers: it looks at tables 3 and 4 only\n");216 query_update(movers&, t&);217 show(t&, movers&, healthy&);218 219 printfmt("update healthy, for the first time: it looks at all five\n");220 query_update(healthy&, t&);221 show(t&, movers&, healthy&);222 223 printfmt("make table 5\n");224 tables_make(t&, velocity | health);225 show(t&, movers&, healthy&);226 227 printfmt("update healthy: it looks at table 5 only\n");228 query_update(healthy&, t&);229 show(t&, movers&, healthy&);230 return 0;231}What play_watermark prints. tools/check.py fails the playground if a single byte differs.
components: P position, V velocity, H health, B burning
movers: all {P V}, none {}; healthy: all {H}, none {B}
make tables 0 to 2
table 0 1 2
has P PV PVH
movers | checked 0, scanned 0, tables none
healthy | checked 0, scanned 0, tables none
update movers: it looks at tables 0 to 2
table 0 1 2
has P PV PVH
movers - yes yes | checked 3, scanned 3, tables 1 2
healthy | checked 0, scanned 0, tables none
update movers again: no new tables, so it looks at nothing
table 0 1 2
has P PV PVH
movers - yes yes | checked 3, scanned 3, tables 1 2
healthy | checked 0, scanned 0, tables none
make tables 3 and 4
table 0 1 2 3 4
has P PV PVH PVHB H
movers - yes yes | checked 3, scanned 3, tables 1 2
healthy | checked 0, scanned 0, tables none
update movers: it looks at tables 3 and 4 only
table 0 1 2 3 4
has P PV PVH PVHB H
movers - yes yes yes - | checked 5, scanned 5, tables 1 2 3
healthy | checked 0, scanned 0, tables none
update healthy, for the first time: it looks at all five
table 0 1 2 3 4
has P PV PVH PVHB H
movers - yes yes yes - | checked 5, scanned 5, tables 1 2 3
healthy - - yes - yes | checked 5, scanned 5, tables 2 4
make table 5
table 0 1 2 3 4 5
has P PV PVH PVHB H VH
movers - yes yes yes - | checked 5, scanned 5, tables 1 2 3
healthy - - yes - yes | checked 5, scanned 5, tables 2 4
update healthy: it looks at table 5 only
table 0 1 2 3 4 5
has P PV PVH PVHB H VH
movers - yes yes yes - | checked 5, scanned 5, tables 1 2 3
healthy - - yes - yes yes | checked 6, scanned 6, tables 2 4 5
The dojo · block 13
A dispatch table
dojo/13.dispatch/play_dispatch.rin
A frame is a list of systems run in order, and the list is data: a name and a pointer to a proc for each, in an array the program changes as it runs. Adding, removing, reordering or switching off a system is an edit to the array, not to the code that calls them:
schedule 0 move 1 shoot 2 damage 3 die
a frame for each enabled system, in order: entry.run(world)
The order is a statement about data: a system that reads what another writes runs after it. damage writes health and die reads it, so damage goes before die, and a drone dies in the frame its health runs out. The same table drives an interpreter's opcodes, a menu's actions, a network protocol's message handlers: a number or a name picks a proc.
The invariant: a run calls every enabled system once, in the list's order, and nothing else. Edits keep the order of everything they don't touch.
In the course: 3.2 makes a system a name, a query and a proc pointer, and runs a schedule of them each frame, and 3.4's game has nine, each placed after what it reads.
Watch it: the playground
python tools/dojo.py play dispatch
A drone flies at a turret. Four systems, move, shoot, damage and die, run frame by frame, and between frames the program edits the schedule: shoot is switched off while the turret reloads, back on, and removed at the end. After each step it prints the schedule or the world:
frame 2
ran move shoot damage die
drone 4 from the turret, health 60, alive
switch shoot off: the turret is reloading
schedule move (shoot off) damage die
frame 3
ran move damage die
Then verify checks that the drone is dead exactly when its health has run out; break inside its if to stop when it fires. Step through it: break on schedule_run and step into each call, through the pointer, into the system.
Then flip the switch at the top, in_order, to 0, and run it again. damage is added at the end of the schedule instead of before die, so die reads the health from before this frame's damage:
frame 5
ran move shoot die damage
drone 0 from the turret, health -20, alive
verify: the drone has health -20 and is still alive
and it dies in frame 6, a frame late. A frame late is a drone that fires once more after it should be dead, or a death sound one frame after the explosion. The switch is a global, read when damage is added, so a debugger can flip it too: stop on the first line of main, set it to 0 in the watch window, and continue.
Do reps: the kata
You write schedule. A system is a plain pointer to a proc, which takes the harness's ctx:
system_proc: alias = *proc(c: *ctx) -> void;
schedule_system: struct = {
name: ^const char;
run: system_proc;
enabled: b32;
// How many times schedule_run has called it.
calls: u32;
}
and the schedule is a vec of them, in the order they run:
schedule_insert_before: proc(s: *schedule, before: ^const char, name: ^const char, run: system_proc) -> b32 = {
schedule_run: proc(s: *schedule, c: *ctx) -> void = {
plus schedule_init, schedule_add(s, name, run) (at the end), schedule_remove(s, name), schedule_set_enabled(s, name, on), schedule_len and schedule_name(s, i) (every name, enabled or not, in order), and schedule_calls(s, name). schedule_insert_before, schedule_remove and schedule_set_enabled return 0, and change nothing, when there's no system by that name. Names are C strings, compared with strcmp.
| level | you write | target |
|---|---|---|
| 1 | schedule_run, in an otherwise finished file | 2:00 |
| 2 | every proc body; the fields and signatures are given | 10:00 |
| 3 | the whole file, from a blank page | 15:00 |
python tools/dojo.py start dispatch 2
python tools/dojo.py dispatch
The harness has twelve systems, input to save, and each one writes its name in the ctx's log when it's called, so after a run the log says what ran, in what order, and through which proc. It runs a short scenario, then 20,000 random adds, inserts, removes, enables, disables and runs against a model that keeps the schedule as a plain list. After every operation the order and every count must match the model, and every run must log exactly the enabled systems, in order. The seeds give the same runs as rust-dod's dojo.
rin notes. Rust's kata has order(), which returns the names as a Vec; here schedule_len and schedule_name give them one at a time, so nothing is allocated to read them.
Level 4, variations. The same harness, different insides:
- A fixed array:
[16]schedule_systemand a count instead of a vec, as the playground has. What happens on the seventeenth add? - Struct of arrays: the names, the procs, the flags and the counts in four vecs, one place per system.
schedule_insert_beforeandschedule_removemust move all four the same way. - Reads and writes: each system declares bitmasks of what it reads and writes, and
schedule_insert_beforerefuses a place where a system would read something a later one writes. Then: which neighbours could run at the same time, on two threads? (Not checked by the harness; write your own test.)
A system with data of its own is one more pointer beside the proc pointer, handed back on every call: 3.4's kayan_system has a ctx: *void for that, which is what a closure is, written out.
When it fails
The scenario catches the common mistakes first:
FAIL in the scenario: run: what ran gave move, it should give move damage die
That run stopped at the first disabled system, collide: a break where a skip was meant. A failure in the fuzz run says where:
FAIL at operation 110 (seed 1): after schedule_remove(die), schedule_calls(damage) is 0; the model counts 2 since it was added
debug it: build/dojo/dispatch/harness --seed 1 --stop 110
with a breakpoint on dojo_stop_here, which runs just before that operation
That kata kept the counts in a vec of their own, the struct of arrays above, and schedule_remove popped the last count rather than the removed system's, so the counts after it shifted one place. Run the harness under your debugger with those arguments and a breakpoint on dojo_stop_here, then step into schedule_remove and watch both vecs.
If the harness crashes instead, on a proc pointer that was never set, say, the runner runs it again with --trace 1 and prints the operation that was running, or says the crash came in the scenario, after the last step printed.
The playground. The kata's harness is dojo/13.dispatch/harness.rin, and the levels start from dojo/13.dispatch/level1.rin to level3.rin.
1// Dojo 13, the playground: a dispatch table.2//3// A frame is a list of systems run in order, and the list is data: a name and4// a pointer to a proc for each, in an array the program changes as it runs.5// Adding, removing, reordering or switching off a system is an edit to the6// array, not to the code that calls them:7//8// schedule 0 move 1 shoot 2 damage 3 die9// a frame for each enabled system, in order: entry.run(world)10//11// The order is a statement about data: a system that reads what another12// writes runs after it. damage writes health and die reads it, so damage13// goes before die, and a drone dies in the frame its health runs out. This is14// the course's schedule (3.2).15//16// This program builds a schedule of four systems for a drone flying at a17// turret, runs it frame by frame, and edits it between frames: shoot is18// switched off while the turret reloads, and removed at the end. After each19// step it prints the schedule or the world, and verify checks that the drone20// is dead exactly when its health has run out; break inside its `if` to stop21// there. To see the dispatch in a debugger, break on `schedule_run` and step22// into each call.23//24// The switch: set in_order to 0 and run it again. damage is added at the end25// of the schedule, after die, instead of before it, and the drone dies a26// frame late. It's a global, read when damage is added, so a debugger can27// flip it too: stop on the first line of main, set it to 0 in the watch28// window, and continue.29 30import "std/print.rin"31import "std/cstd.rin"32 33in_order: b32 = 1;34 35// A drone flying at a turret.36world: struct = {37 distance: i32;38 health: i32;39 alive: b32;40 // Hits this frame, waiting for damage.41 hits: i32;42}43 44// The systems. Each reads and writes a little of the world.45move_system: proc(w: *world) -> void = {46 if (w*.alive) {47 w*.distance -= 2;48 if (w*.distance < 0) {49 w*.distance = 0;50 }51 }52}53 54shoot_system: proc(w: *world) -> void = {55 if (w*.alive && w*.distance <= 4) {56 w*.hits += 1;57 }58}59 60damage_system: proc(w: *world) -> void = {61 if (w*.alive) {62 w*.health -= 40 * w*.hits;63 }64 w*.hits = 0;65}66 67die_system: proc(w: *world) -> void = {68 if (w*.alive && w*.health <= 0) {69 w*.alive = 0;70 }71}72 73// A system is a plain pointer to a proc.74system_proc: alias = *proc(w: *world) -> void;75 76schedule_entry: struct = {77 name: ^const char;78 run: system_proc;79 enabled: b32;80}81 82schedule: struct = {83 entries: [8]schedule_entry;84 count: u64;85}86 87// Where the system called `name` is, or count if it isn't there.88schedule_find: proc(s: *schedule, name: ^const char) -> u64 = {89 for (i: u64 = 0; i < s*.count; i += 1) {90 if (strcmp(s*.entries[i].name, name) == 0) {91 return i;92 }93 }94 return s*.count;95}96 97schedule_add: proc(s: *schedule, name: ^const char, run: system_proc) -> void = {98 s*.entries[s*.count] = { .name = name, .run = run, .enabled = 1 };99 s*.count += 1;100}101 102schedule_insert_before: proc(s: *schedule, before: ^const char, name: ^const char, run: system_proc) -> b32 = {103 at: u64 = schedule_find(s, before);104 if (at == s*.count) {105 return 0;106 }107 for (i: u64 = s*.count; i > at; i -= 1) {108 s*.entries[i] = s*.entries[i - 1];109 }110 s*.entries[at] = { .name = name, .run = run, .enabled = 1 };111 s*.count += 1;112 return 1;113}114 115schedule_remove: proc(s: *schedule, name: ^const char) -> b32 = {116 at: u64 = schedule_find(s, name);117 if (at == s*.count) {118 return 0;119 }120 for (i: u64 = at; i + 1 < s*.count; i += 1) {121 s*.entries[i] = s*.entries[i + 1];122 }123 s*.count -= 1;124 s*.entries[s*.count] = {};125 return 1;126}127 128schedule_set_enabled: proc(s: *schedule, name: ^const char, on: b32) -> b32 = {129 at: u64 = schedule_find(s, name);130 if (at == s*.count) {131 return 0;132 }133 s*.entries[at].enabled = on;134 return 1;135}136 137// One frame: every enabled system, once, in order.138schedule_run: proc(s: *schedule, w: *world) -> void = {139 printfmt(" ran ");140 for (i: u64 = 0; i < s*.count; i += 1) {141 if (s*.entries[i].enabled) {142 s*.entries[i].run(w);143 printfmt(" {}", s*.entries[i].name);144 }145 }146 printfmt("\n");147}148 149schedule_show: proc(s: *schedule) -> void = {150 printfmt(" schedule ");151 for (i: u64 = 0; i < s*.count; i += 1) {152 if (s*.entries[i].enabled) {153 printfmt(" {}", s*.entries[i].name);154 } else {155 printfmt(" ({} off)", s*.entries[i].name);156 }157 }158 printfmt("\n");159}160 161show: proc(w: *world) -> void = {162 printfmt(" drone {} from the turret, health {}, ", w*.distance, w*.health);163 if (w*.alive) {164 printfmt("alive\n");165 } else {166 printfmt("dead\n");167 }168}169 170// The invariant, after every step: the drone is dead exactly when its health171// has run out.172verify: proc(w: *world) -> void = {173 if ((w*.alive != 0) == (w*.health <= 0)) {174 if (w*.alive) {175 printfmt(" verify: the drone has health {} and is still alive\n", w*.health);176 } else {177 printfmt(" verify: the drone is dead with health {}\n", w*.health);178 }179 }180}181 182frame: proc(s: *schedule, w: *world, n: u32) -> void = {183 printfmt("frame {}\n", n);184 schedule_run(s, w);185 show(w);186 verify(w);187}188 189main: proc() -> i32 = {190 s: schedule = {};191 w: world = { .distance = 8, .health = 100, .alive = 1 };192 printfmt("a drone 8 from a turret, with health 100: it flies 2 closer a frame,\n");193 printfmt("and the turret hits it for 40 a frame once it's within 4\n");194 show(w&);195 196 printfmt("add move, shoot and die\n");197 schedule_add(s&, "move", move_system);198 schedule_add(s&, "shoot", shoot_system);199 schedule_add(s&, "die", die_system);200 schedule_show(s&);201 verify(w&);202 203 if (in_order) {204 printfmt("insert damage before die: die reads the health that damage writes\n");205 _ = schedule_insert_before(s&, "die", "damage", damage_system);206 } else {207 printfmt("add damage at the end, after die\n");208 schedule_add(s&, "damage", damage_system);209 }210 schedule_show(s&);211 verify(w&);212 213 frame(s&, w&, 1);214 frame(s&, w&, 2);215 216 printfmt("switch shoot off: the turret is reloading\n");217 _ = schedule_set_enabled(s&, "shoot", 0);218 schedule_show(s&);219 verify(w&);220 221 frame(s&, w&, 3);222 223 printfmt("switch shoot back on\n");224 _ = schedule_set_enabled(s&, "shoot", 1);225 schedule_show(s&);226 verify(w&);227 228 frame(s&, w&, 4);229 frame(s&, w&, 5);230 frame(s&, w&, 6);231 232 printfmt("remove shoot: the turret is gone\n");233 _ = schedule_remove(s&, "shoot");234 schedule_show(s&);235 verify(w&);236 return 0;237}What play_dispatch prints. tools/check.py fails the playground if a single byte differs.
a drone 8 from a turret, with health 100: it flies 2 closer a frame,
and the turret hits it for 40 a frame once it's within 4
drone 8 from the turret, health 100, alive
add move, shoot and die
schedule move shoot die
insert damage before die: die reads the health that damage writes
schedule move shoot damage die
frame 1
ran move shoot damage die
drone 6 from the turret, health 100, alive
frame 2
ran move shoot damage die
drone 4 from the turret, health 60, alive
switch shoot off: the turret is reloading
schedule move (shoot off) damage die
frame 3
ran move damage die
drone 2 from the turret, health 60, alive
switch shoot back on
schedule move shoot damage die
frame 4
ran move shoot damage die
drone 0 from the turret, health 20, alive
frame 5
ran move shoot damage die
drone 0 from the turret, health -20, dead
frame 6
ran move shoot damage die
drone 0 from the turret, health -20, dead
remove shoot: the turret is gone
schedule move damage die
Advanced · seven lessons
Layout and SIMD
docs/README.md
particle is 68 bytes. Movement reads the first 24.In the array of structs, every cache line movement pulls in is about two thirds tint, age, life, size and name. In the struct of arrays, movement fetches only the position and velocity columns: 6,291,456 bytes a step instead of 17,825,792.
Seven lessons for after Build an ECS. The course builds the ECS. These go underneath it, to the memory and the machine code:
- Lesson 1, aos vs soa. The same particles in three layouts: what a loop fetches per cache line, what
noaliaspromises the compiler, which loops it vectorizes, and how to check that in the assembly. - Lessons 2 to 6 build an archetype ECS again, compressed into five steps, and end with a small game. They came first and move fast. After the course, read them as a recap, or as a second take on the same design. Lesson 4 builds the world in
includes/kayan.rin, and lessons 5 and 6 import it unchanged. - Lesson 7, layout follows access. Four layouts under streams and under random lookups, timed, and how an ECS decides where each component lives.
- Lesson 0, from kayan, reviews the original C kayan: what these lessons kept from it, what they changed, and why.
| lesson | about | |
|---|---|---|
| 0 | from kayan | what was kept from the C kayan, what changed, and why |
| 1 | aos vs soa | the same particles in three layouts, what a loop fetches, and what SIMD needs |
| 2 | archetype table | one table: columns, rows, spawn, swap-remove, systems over slices |
| 3 | many archetypes | one table type for every archetype: ids, signatures, queries |
| 4 | entity handles | generational handles, slots, moving between archetypes |
| 5 | inspector | printing and editing any entity from reflection records |
| 6 | kayan | a small game, and deferring structural changes to a command buffer |
| 7 | access patterns | four layouts under streams and lookups, and how an ECS decides |
How to read them
Read each lesson's .rin file from the top. It opens with the problem it solves, and the code follows in the order it runs. Then run it and read the output against the code: every lesson prints what it does, step by step. The guides here give the background that doesn't fit in a file header, and end with exercises.
Timings only mean something in a Release build. Debug checks every slice index. Lesson 1's guide shows how to build one, and what to look for in the disassembly.
Advanced · lesson 0 · review
From kayan
docs/0_from_kayan.md
kayan is a sketch of an archetype entity component system in C, about 200 lines in src/kayan.h, with haikal generating its containers. These lessons start from its idea and rebuild the rest. This page says what was kept, what was changed, and why, so the lessons can be read as a review of it as much as a rewrite.
What kayan does
Components are bits in an enum, positions = 1 << 0, velocities = 1 << 1 and so on. Each archetype is a C struct written by hand (A, B, C, D) that starts with an ArchetypeHeader and then holds one haikal Array_T per component. The header has a u64 signature and a Map_u64 from column name to the column's byte offset in the struct.
A query walks an array of void * pointing at the archetypes, reads each header, and tests (signature & want) == want. For a match it looks up "positions" and "velocities" in the offset map, adds the offsets to the struct's address, and runs the loop over the two columns.
Kept
- An archetype is a struct of arrays. Each component is a contiguous column and every entity in the table has every column. That is the whole point, and kayan had it right from the start.
- A signature is a u64 of component bits, and a query is the test
(signature & all) == all. Lesson 3 adds anonemask beside it. - Arenas for all of it. Nothing in these lessons is freed one at a time.
Changed
- One table type for every archetype (lesson 3). kayan needs a new C struct, a new signature line and new offset-map lines for every archetype. Here an archetype is data: a signature, and the world makes its table the first time something needs it (lesson 4).
- Columns by component id, not by name (lesson 3). kayan hashes the strings
"positions"and"velocities"on every query. Here a table's columns are an array indexed by component id, so finding one is an index. Names are kept for tools, through reflection (lesson 5). - The signature comes from the columns (lesson 3). In kayan they are written separately and can disagree. In
Dthey do: it haspositionsandvelocitiescolumns, but its signature ishealth | damage | alive, so the movement query never visits it. - A row is added and removed in one place (lesson 2). kayan appends to each column at the call site, and keeps
entity_countapart from the columns. ForBandDit incrementsa's count instead of their own (kayan.hlines 142, 145, 168, 174), soAreports seven entities and the others report none. Nothing breaks only because the query loop happens to use the column's own length. - Entities have handles (lesson 4). kayan's entity id is the value of an archetype's
entity_countbefore it is bumped: meant per table, in practiceA's, as above. Nothing records where an entity lives, so there is no way to refer to one entity from elsewhere, to remove it, or to move it to another archetype. The lessons use an{index, generation}handle and a slot list that tracks each entity's table and row. - Entities move between archetypes (lesson 4). Adding or removing a component moves the entity to the matching table, copying the components both tables share.
- Structural changes wait for the end of the frame (lesson 6), through a command buffer in a frame arena. kayan never changes structure while a query runs, but only because it never changes structure at all.
- Reflection replaces hand-written tables (lessons 3 to 5). A component's size and alignment come from
T<>, and so do the names a tool prints. The offset map goes away entirely.
Not carried over
- **
vec4sfor positions.** kayan pads 3D vectors to 16 bytes, which suits SIMD loads. The lessons usevec2andvec3and plain loops, and leave vectorizing to the C compiler.noaliason the kernels' parameters, in lessons 1, 2, 3 and 6, lets it do that without run-time overlap checks. Lesson 1 adds a third layout, onef32array per coordinate, which is the one SIMD runs cleanly over, and its exercises come back tovec4. Lesson 7 weighs that layout against lookups, which it is bad at. - cglm and the haikal containers. rin's generics and std (
vec<T>,hashmap<K, V>, slices) cover what they did.
Small things in kayan.h
- The query's inner loop declares
i32 i, shadowing the outeri(line 201). - Each lookup on lines 199 and 200 hashes and compares a string, once per column per archetype per query. That is cheap next to a large loop, but it is work that an array index does for nothing.
Advanced · lesson 1
AoS vs SoA
src/1.aos_vs_soa/aos_vs_soa.rin
src/1.aos_vs_soa/aos_vs_soa.rin
The same 262,144 particles, stored three times: as an array of 68-byte particle structs (AoS), as seven arrays, one per field (SoA), and with position and velocity split one step further into six arrays of f32, one per coordinate (split). Sixty steps of movement run over each, and the program checks that all three end with the same positions.
What the loop fetches
Memory reaches the CPU in 64-byte cache lines. A loop that reads 24 bytes of each 68-byte particle still pulls every byte of every line those particles sit in, so about two thirds of what movement fetches in the AoS loop is tint, age, lifetime, size and name. In the SoA loop the two arrays it reads are the only thing it fetches, and the hardware prefetcher sees two simple streams. The split loop fetches the same 24 bytes per particle as SoA, as six streams.
The program prints that arithmetic:
bytes fetched per step: 17825792 (AoS: 68 bytes x 262144)
bytes fetched per step: 6291456 (SoA: 24 bytes x 262144)
bytes fetched per step: 6291456 (split: 6 x 4 bytes x 262144)
What the CPU does with each layout
SSE registers hold four floats, AVX registers eight. One instruction adds four (or eight) pairs of floats at once, but only if they arrive side by side. Here is what clang-cl /O2 makes of each loop for Windows x64, read from the object file:
| loop | particles per pass | instructions per pass | per particle | shuffles |
|---|---|---|---|---|
move_aos | 1 | 16 | 16 | 1 |
move_soa | 4 | 77 | about 19 | 30 |
move_split | 4 | 20 | 5 | 0 |
move_split, /arch:AVX2 | 8 | 14 | under 2 | 0 |
- AoS goes one particle at a time: each particle's fields are 68 bytes from the next particle's, so there is nothing to load four of.
- SoA is vectorized, four particles per pass, but each
vec3keeps x, y and z side by side. To get four x's into one register the compiler loads them one float at a time and stitches them together, then takes them apart again to store. About a tenth of the loop is arithmetic, and per particle it costs about what AoS does. - Split is what SIMD wants. Each line of
move_splitbecomes one load of four consecutive floats, one multiply, one add, one store:
movups (%r11,%rdi,4), %xmm4 four vx
movups (%rcx,%rdi,4), %xmm5 four px
mulps %xmm1, %xmm4 vx * dt
addps %xmm5, %xmm4 px + vx * dt
movups %xmm4, (%rcx,%rdi,4) four px back
A layout is right for SIMD when its vector loop is mostly full-width loads, math and stores. When it is mostly movss, unpcklps and shufps, the data is fighting the registers.
Split is right for this loop, which runs over every particle in order. It is wrong for a loop that reaches particles by index, where x, y and z in three arrays are three trips to memory instead of one. Lesson 7 measures both kinds of loop over four layouts.
Timings
--time prints how long each layout took. Release builds, best of five runs, on the 2.8 GHz Xeon these lessons were checked on, ms:
| build | AoS | SoA | split |
|---|---|---|---|
clang -O2 | 49.6 | 26.9 | 16.5 |
clang -O3 | 47.5 | 26.7 | 15.9 |
clang -O2 -mavx2 -mfma | 47.9 | 18.6 | 13.0 |
gcc -O3 | 34.2 | 13.7 | 14.2 |
gcc -O2 | 46.1 | 21.3 | 44.3 |
The same loops with 8,192 particles, so everything stays in L2, and 1,920 steps so the work is the same:
| build | AoS | SoA | split |
|---|---|---|---|
clang -O2 | 24.1 | 25.6 | 11.9 |
clang -O2 -mavx2 -mfma | 28.6 | 17.8 | 6.5 |
gcc -O3 | 12.7 | 12.0 | 8.9 |
- In cache, split is the only layout SIMD helps much. SoA and AoS come out about even in L2 on clang, as the instruction counts say they should.
- From the 6 MB case, SoA's smaller footprint shows. That machine has 33 MB of L3, so the data stays in cache between steps, but SoA and split still move a third of the bytes AoS does.
- **gcc
-O2does not vectorize across particles at all,** so split, with six streams and nothing gained for them, is its slowest layout. gcc users want-O3. clang-cl, rin's default, is the clang rows.
The machine is shared; expect about 20 percent noise between runs. Debug numbers are not worth comparing: every positions[i] is bounds checked there.
For a Release build, set build_type to "Release" in build.rin and run rin build again. Release defines NDEBUG, which compiles the bounds checks out. Or compile the C that tools/check.py leaves in build/check/aos_vs_soa/ with -O2 -DNDEBUG.
The parameter list is the data
move_aos(ps: noalias []particle, ...) says it needs particles. move_soa(positions: noalias []vec3, velocities: noalias []vec3, ...) and move_split(px, py, pz, vx, vy, vz, ...) say exactly which data they read and write. In lessons 2 and 3 every system is written that way. In lesson 6 each system runs its own query, then works only on the columns it named.
Naming the columns is what lets a scheduler see that two systems touch different data, and it is what makes a system easy to test: hand it two small arrays.
noalias
As far as the C compiler can tell, two []vec3 slices could overlap. So before it works on several particles at once, it checks at run time that the two are apart, and keeps a one-at-a-time copy of the loop for when they are not. noalias on a parameter promises that, for the call, nothing else reaches the memory behind it. The check and the second copy go away.
The SoA loop without and with it, Release, ms:
| build | without noalias | with noalias |
|---|---|---|
gcc -O3 | 31.7 | 13.4 |
gcc -O2 | 20.5 | 21.3 |
clang -O2 | 26.3 | 26.5 |
- **gcc
-O3** is where it shows. Without the promise, its checked vector loop was slower than not vectorizing at all (21 ms with-fno-tree-loop-vectorize), and the SoA loop lost to the AoS one. - clang vectorized the loop either way. Its check runs once per call, so removing it saves nothing that shows here.
On Windows, one slice needs it too. move_aos has a single slice, which cannot overlap another, and it still takes noalias. Windows passes a slice to a proc by reference, as a pointer to a copy, and clang-cl does not use C's type rules to decide what can overlap (it matches MSVC). So, without the promise, a float written through ps might be the slice's own pointer or length, and the loop reads both again after every particle:
mov r8,qword ptr [rcx] ps.data, every particle
movss xmm2,dword ptr [gravity] gravity, every particle
...
cmp rdx,qword ptr [rcx+8] ps.length, every particle
With noalias the compiler gets the pointer and the length once, in registers, and the loop is 16 instructions instead of 19.
**And gravity is static.** A global that other files could reach is, to gcc, something a write through a slice might change, even through a noalias one, so gcc -O3 put a run-time check in front of move_split's vector loop. gravity: static f32 keeps it to this file, nothing takes its address, and the check goes away. Passing it as a parameter would do the same.
The promise is checked in Debug. Pass soa.positions as both arguments to move_soa and the Debug build stops before the loop runs:
aos_vs_soa.rin:138: noalias parameters 'positions' and 'velocities' overlap
A Release build does not check. It runs on, and the lesson prints same result: no.
Fused multiply-add
With -mfma, -march=native or clang-cl's /arch:AVX2, clang fuses p + v * dt into one instruction that rounds once instead of twice. All three layouts are fused the same way and still agree, but the checksum's last digits move, from 38572460.505 to 38572460.519, and the byte-for-byte check against aos_vs_soa.expected fails. tools/check.py builds without those flags.
When AoS is right
When the code that touches a record touches most of its fields, AoS fetches nothing wasted, and keeping a record together is simpler. Data-oriented design does not mean SoA everywhere. It means choosing the layout from the loops that run over it. An entity component system is the case where many small loops each touch a few fields.
Try
- Run with
--timein Release. Then removetint,age,lifetimeandnamefromparticleand run again. The AoS loop should close most of the gap, because the struct shrinks toward what the loop reads. - Add
/arch:AVX2tobuild_c_flagsand compare--time. Then readmove_splitin the disassembly: eight floats per register, andvfmadd. - kayan stored positions as 16-byte
vec4s. Changevec3to four floats and see what it does to the bytes-per-step line, tomove_soa's disassembly and to the timing. - Store the split columns in blocks of eight,
{ x: [8]f32; y: [8]f32; ... }, and write the loop over blocks. Each block is a whole AVX register per coordinate, and a particle's coordinates stay near each other. - Remove
noaliasfrommove_soaand build Release with gcc-O3. Compare--time, then compile with-fopt-info-vecand find "loop versioned for vectorization because of possible aliasing". - Remove
staticfromgravity, build with gcc-O3 -fopt-info-vec, and find the same message formove_split. - Pass
soa.positionsas both arguments tomove_soa. Run it in Debug, then in Release.
1// 1. AoS vs SoA2//3// The first decision in data-oriented code is how a collection sits in memory,4// and it is made before any system is written.5//6// An array of structs (AoS) keeps each particle's fields together: position,7// velocity, tint, age, lifetime, size and a name, one particle after another.8// A struct of arrays (SoA) keeps each field together instead: every position,9// then every velocity, and so on, each in an array of its own.10//11// Both hold exactly the same data. What differs is what a loop drags through12// the cache. Movement reads position and velocity, 24 bytes of each particle.13// The CPU fetches memory 64 bytes at a time, so in AoS most of every line it14// fetches for movement is tint, age and name, which the loop never looks at. In15// SoA the two arrays movement reads are all it fetches.16//17// SoA still stores each position as a vec3, x y z side by side. A third layout18// splits those too: one array of f32 per coordinate. Each array is then a19// straight run of floats, which is what a SIMD register loads: four floats at20// once with SSE, eight with AVX, nothing to rearrange first.21//22// The movement procs below also differ in what they ask for. move_aos takes the23// whole particle. move_soa and move_split take the arrays they use and nothing24// else, so their parameter lists are lists of their data dependencies. That25// habit is what the rest of these lessons build on.26//27// The program runs the same movement over all three layouts and checks that28// they agree. With `--time` it also times them. Build Release before believing29// the numbers: Debug checks every slice index.30 31cinclude "time.h"32import "std/print.rin"33import "std/memops.rin"34 35clock: proc[external]() -> long = {}36CLOCKS_PER_SEC: const long;37 38particle_count: u64 = 262144;39steps: u64 = 60;40dt: f32 = 1.0 / 60.0;41// static: only this file can reach gravity, and nothing takes its address, so42// the compiler knows no write through a slice can change it. Without it gcc43// puts a run-time check in front of move_split's vector loop.44gravity: static f32 = -9.8;45 46vec3: struct = {47 x: f32;48 y: f32;49 z: f32;50}51 52color: struct = {53 r: f32;54 g: f32;55 b: f32;56 a: f32;57}58 59particle_name: struct = {60 text: [16]c8;61}62 63// One particle, the way a particle system tends to grow.64particle: struct = {65 position: vec3;66 velocity: vec3;67 tint: color;68 age: f32;69 lifetime: f32;70 size: f32;71 name: particle_name;72}73 74// The same particles, one array per field. Every array has the same length, and75// index i in each of them is particle i.76particles: struct = {77 positions: []vec3;78 velocities: []vec3;79 tints: []color;80 ages: []f32;81 lifetimes: []f32;82 sizes: []f32;83 names: []particle_name;84}85 86// Position and velocity split one step further, one array per coordinate.87split_columns: struct = {88 px: []f32;89 py: []f32;90 pz: []f32;91 vx: []f32;92 vy: []f32;93 vz: []f32;94}95 96// count elements of T from the arena, zeroed, as a slice that knows its length.97push_slice: proc<T>(arena: *memops_arena, count: u64) -> []T = {98 raw: ^T = cast(memops_arena_push_zero(arena, sizeof(T) * count, alignof(T)), ^T);99 return raw[0..count];100}101 102// A small LCG, so both layouts start from the same particles on every compiler.103random_state: u64 = 12345;104 105random01: proc() -> f32 = {106 random_state = random_state * 6364136223846793005'u64 + 1442695040888963407'u64;107 return cast(random_state shr 40, f32) / 16777216.0;108}109 110make_particle: proc(i: u64) -> particle = {111 p: particle = {};112 p.position = { .x = random01() * 100.0, .y = random01() * 100.0, .z = random01() * 100.0 };113 p.velocity = { .x = random01() - 0.5, .y = random01() * 4.0, .z = random01() - 0.5 };114 p.tint = { .r = random01(), .g = random01(), .b = random01(), .a = 1.0 };115 p.lifetime = 2.0 + random01() * 3.0;116 p.size = 0.1 + random01();117 p.name.text[0] = 'p';118 p.name.text[1] = cast('0' + i % 10, c8);119 return p;120}121 122// noalias even on a single slice: on Windows a slice is passed by reference,123// and clang-cl does not use C's type rules, so without it a float written124// through ps might be the slice's own pointer or length. The loop would read125// both again after every particle.126move_aos: proc(ps: noalias []particle, dt: f32) -> void = {127 for (i: u64 = 0; i < ps.length; i += 1) {128 p: *particle = ps[i]&;129 p*.velocity.y += gravity * dt;130 p*.position.x += p*.velocity.x * dt;131 p*.position.y += p*.velocity.y * dt;132 p*.position.z += p*.velocity.z * dt;133 }134}135 136// noalias: positions and velocities never overlap, so the compiler can work on137// several particles at once without first checking that they don't.138move_soa: proc(positions: noalias []vec3, velocities: noalias []vec3, dt: f32) -> void = {139 for (i: u64 = 0; i < positions.length; i += 1) {140 velocities[i].y += gravity * dt;141 positions[i].x += velocities[i].x * dt;142 positions[i].y += velocities[i].y * dt;143 positions[i].z += velocities[i].z * dt;144 }145}146 147// Every column it writes is noalias; vx and vz are only read. Each line of the148// loop is one SIMD load, multiply, add and store over consecutive floats.149move_split: proc(px: noalias []f32, py: noalias []f32, pz: noalias []f32,150 vx: []const f32, vy: noalias []f32, vz: []const f32, dt: f32) -> void = {151 for (i: u64 = 0; i < px.length; i += 1) {152 vy[i] += gravity * dt;153 px[i] += vx[i] * dt;154 py[i] += vy[i] * dt;155 pz[i] += vz[i] * dt;156 }157}158 159checksum_aos: proc(ps: []particle) -> f64 = {160 sum: f64 = 0.0;161 for (i: u64 = 0; i < ps.length; i += 1) {162 sum += cast(ps[i].position.x + ps[i].position.y + ps[i].position.z, f64);163 }164 return sum;165}166 167checksum_soa: proc(positions: []vec3) -> f64 = {168 sum: f64 = 0.0;169 for (i: u64 = 0; i < positions.length; i += 1) {170 sum += cast(positions[i].x + positions[i].y + positions[i].z, f64);171 }172 return sum;173}174 175checksum_split: proc(xs: []const f32, ys: []const f32, zs: []const f32) -> f64 = {176 sum: f64 = 0.0;177 for (i: u64 = 0; i < xs.length; i += 1) {178 sum += cast(xs[i] + ys[i] + zs[i], f64);179 }180 return sum;181}182 183seconds: proc() -> f64 = {184 return cast(clock(), f64) / cast(CLOCKS_PER_SEC, f64);185}186 187main: proc(argc: i32, argv: ^^c8) -> i32 = {188 timing: b32 = argc > 1 && strcmp(argv[1], "--time") == 0;189 190 arena: memops_arena = {};191 memops_arena_initialize(arena&);192 193 aos: []particle = push_slice<particle>(arena&, particle_count);194 soa: particles = {};195 soa.positions = push_slice<vec3>(arena&, particle_count);196 soa.velocities = push_slice<vec3>(arena&, particle_count);197 soa.tints = push_slice<color>(arena&, particle_count);198 soa.ages = push_slice<f32>(arena&, particle_count);199 soa.lifetimes = push_slice<f32>(arena&, particle_count);200 soa.sizes = push_slice<f32>(arena&, particle_count);201 soa.names = push_slice<particle_name>(arena&, particle_count);202 split: split_columns = {};203 split.px = push_slice<f32>(arena&, particle_count);204 split.py = push_slice<f32>(arena&, particle_count);205 split.pz = push_slice<f32>(arena&, particle_count);206 split.vx = push_slice<f32>(arena&, particle_count);207 split.vy = push_slice<f32>(arena&, particle_count);208 split.vz = push_slice<f32>(arena&, particle_count);209 210 // The same particle goes into every layout: whole into aos, field by field211 // into soa, coordinate by coordinate into split.212 for (i: u64 = 0; i < particle_count; i += 1) {213 p: particle = make_particle(i);214 aos[i] = p;215 soa.positions[i] = p.position;216 soa.velocities[i] = p.velocity;217 soa.tints[i] = p.tint;218 soa.ages[i] = p.age;219 soa.lifetimes[i] = p.lifetime;220 soa.sizes[i] = p.size;221 soa.names[i] = p.name;222 split.px[i] = p.position.x;223 split.py[i] = p.position.y;224 split.pz[i] = p.position.z;225 split.vx[i] = p.velocity.x;226 split.vy[i] = p.velocity.y;227 split.vz[i] = p.velocity.z;228 }229 230 t0: f64 = seconds();231 for (s: u64 = 0; s < steps; s += 1) {232 move_aos(aos, dt);233 }234 t1: f64 = seconds();235 for (s: u64 = 0; s < steps; s += 1) {236 move_soa(soa.positions, soa.velocities, dt);237 }238 t2: f64 = seconds();239 for (s: u64 = 0; s < steps; s += 1) {240 move_split(split.px, split.py, split.pz, split.vx, split.vy, split.vz, dt);241 }242 t3: f64 = seconds();243 244 used: u64 = sizeof(vec3) * 2;245 printfmt("{} particles, {} steps\n", particle_count, steps);246 printfmt("array of structs: a particle is {} bytes, movement uses {} of them\n", sizeof(particle), used);247 printfmt(" bytes fetched per step: {}\n", sizeof(particle) * particle_count);248 printfmt("struct of arrays: movement reads two arrays of {}-byte vec3\n", sizeof(vec3));249 printfmt(" bytes fetched per step: {}\n", used * particle_count);250 printfmt("split columns: movement reads six arrays of {}-byte f32\n", sizeof(f32));251 printfmt(" bytes fetched per step: {}\n", used * particle_count);252 253 a: f64 = checksum_aos(aos);254 b: f64 = checksum_soa(soa.positions);255 c: f64 = checksum_split(split.px, split.py, split.pz);256 printf("checksum aos %.3f\n", a);257 printf("checksum soa %.3f\n", b);258 printf("checksum split %.3f\n", c);259 same: b32 = a == b && b == c;260 printfmt("same result: {}\n", same ? "yes" : "no");261 262 if (timing) {263 printf("aos %.1f ms, soa %.1f ms, split %.1f ms\n",264 (t1 - t0) * 1000.0, (t2 - t1) * 1000.0, (t3 - t2) * 1000.0);265 }266 return same ? 0 : 1;267}What aos_vs_soa prints. tools/check.py fails the lesson if a single byte differs.
262144 particles, 60 steps
array of structs: a particle is 68 bytes, movement uses 24 of them
bytes fetched per step: 17825792
struct of arrays: movement reads two arrays of 12-byte vec3
bytes fetched per step: 6291456
split columns: movement reads six arrays of 4-byte f32
bytes fetched per step: 6291456
checksum aos 38572460.505
checksum soa 38572460.505
checksum split 38572460.505
same result: yes
Advanced · lesson 2
One archetype table
src/2.archetype_table/archetype_table.rin
src/2.archetype_table/archetype_table.rin
In the dojo: 2. swap-remove and the back-reference, 4. a type-erased column, each on its own, to watch and to practise.
Eight sparks in one table with three component columns, positions, velocities and lives, plus a numbers column that exists only so the output can say which spark is which. Each frame they move, they age, and the ones that ran out are removed.
Two rules
Columns never disagree. spark_spawn and spark_despawn are the only places rows come and go, and each touches every column. A new component means a new column, one line in each of those two procs, and one more argument to spark_spawn. A call site cannot add a row to some columns and forget the others.
Rows are packed. Systems loop over 0..count and never test whether a row is alive. Removing a row moves the last row into the hole:
spark 0 in row 0 expired; spark 7 moved from row 7 into it
That one copy per column is the whole cost of a despawn, whatever the table's size. The price is that a row number is not a name for a spark. After that line, "row 0" means spark 7. Lesson 4 adds names that do not move.
Removing while iterating
expire_system walks the rows backwards. When it removes row i, the row that moves in comes from the end, and the end has already been tested. A forward loop would have to stay on i after a removal, and it is easy to get wrong. Lesson 6 removes the question entirely by not removing anything during a loop.
Systems take slices
move_system(positions: noalias []vec2, velocities: []const vec2, dt) gets the rows in use, sparks.positions[0..sparks.count], and knows nothing about tables. []const says it only reads velocities. Each positions[i] is bounds checked in Debug, against the count rather than the capacity, so a system that runs past the live rows stops with a message instead of reading stale data.
noalias says that for the call, nothing else reaches the memory positions covers. Only the column the system writes needs it: a promise on the written column already rules out a write landing in the column being read. Without it, clang -O2 and gcc -O3 both vectorized this loop behind a run-time check that the two columns are apart. With it, neither checks. A Debug build checks the promise instead, once per call.
age_system(lives: noalias []f32, dt) has one slice, and still takes it. Windows passes a slice by reference, as a pointer to a copy, and clang-cl does not use C's type rules to decide what can overlap. So without the promise, a float written through lives might be the slice's own pointer or length. clang-cl then reads both again after every element and does one float at a time. With it, the loop is eight floats per pass:
movups (%rcx,%r8,4), %xmm2 four lives
movups 0x10(%rcx,%r8,4), %xmm3 four more
subps %xmm0, %xmm2 minus dt
subps %xmm0, %xmm3
movups %xmm2, (%rcx,%r8,4)
movups %xmm3, 0x10(%rcx,%r8,4)
The rule that holds on every compiler: noalias on every slice a loop writes, even when it is the only one.
lives is the one column here that is a plain run of floats. positions and velocities are vec2, x and y side by side, so move_system's vector loop spends some of its instructions pulling the x's and the y's apart. Lesson 1 measures what that costs.
Status codes
spark_spawn returns spawn_status, an enum[must_use]: calling it and ignoring the result is a compile error. That is rin's convention for a proc that can fail: a status, with the result through an out-parameter. The ninth spark shows the full case.
Try
- Rewrite
expire_systemto walk forwards with a plainfor. The output does not change, because no frame here expires a spark together with the one in the last row. Change two lives so that one does, and find the spark that outlives its life by a frame. Then fix the loop. - Delete the line in
spark_despawnthat copieslives. What does the output show, and how long did it take to notice? - Call
spark_spawnwithout using its result and read the compiler's error. Then discard it on purpose with_ = spark_spawn(...). - Make the table grow instead of reporting
full: allocate double the capacity from the arena and copy every column across. Lesson 4'skayan_table_growis one answer.
1// 2. One archetype table2//3// An archetype is one combination of components. Its table holds every entity4// that has exactly that combination: one column per component, one row per5// entity. It is lesson 1's struct of arrays with two rules added.6//7// Every column has the same length, always. Spawn appends to every column in8// one place and despawn removes from every column in one place. kayan appended9// to each column at the call site, and three appends that must happen together10// are three chances to forget one.11//12// Rows are packed. Despawn moves the last row into the hole it leaves (a13// swap-remove), so a system loops over 0..count with no gaps and no "alive"14// flag to test. The price is that a row is not a stable name for an entity: the15// last entity changes rows every time something before it dies. Lesson 4 deals16// with that.17//18// The sparks here have a position, a velocity and a life in seconds. Each frame19// they move, they age, and the ones whose life ran out are removed. Removal20// walks the rows backwards, so the row swapped into a hole has already been21// looked at this frame.22 23import "std/print.rin"24import "std/memops.rin"25 26vec2: struct = {27 x: f32;28 y: f32;29}30 31// The table for the archetype {position, velocity, life}. Each column is sized32// to the capacity once; count says how many rows are in use. `numbers` is not a33// component: it only lets the output say which spark sits in which row.34spark_table: struct = {35 count: u64;36 capacity: u64;37 numbers: []u32;38 positions: []vec2;39 velocities: []vec2;40 lives: []f32;41}42 43spawn_status: enum[must_use] = {44 ok,45 full,46}47 48push_slice: proc<T>(arena: *memops_arena, count: u64) -> []T = {49 raw: ^T = cast(memops_arena_push_zero(arena, sizeof(T) * count, alignof(T)), ^T);50 return raw[0..count];51}52 53spark_table_create: proc(arena: *memops_arena, capacity: u64) -> spark_table = {54 t: spark_table = {};55 t.capacity = capacity;56 t.numbers = push_slice<u32>(arena, capacity);57 t.positions = push_slice<vec2>(arena, capacity);58 t.velocities = push_slice<vec2>(arena, capacity);59 t.lives = push_slice<f32>(arena, capacity);60 return t;61}62 63// The only place a row is added: every column gets its value here.64spark_spawn: proc(t: *spark_table, number: u32, position: vec2, velocity: vec2, life: f32,65 row: *u64) -> spawn_status = {66 if (t*.count == t*.capacity) {67 return spawn_status.full;68 }69 r: u64 = t*.count;70 t*.numbers[r] = number;71 t*.positions[r] = position;72 t*.velocities[r] = velocity;73 t*.lives[r] = life;74 t*.count += 1;75 row* = r;76 return spawn_status.ok;77}78 79// The only place a row is removed. The last row moves into `row`, in every80// column, and the table shrinks by one. Returns the row that moved, which is81// `row` itself when the removed row was the last one.82spark_despawn: proc(t: *spark_table, row: u64) -> u64 = {83 last: u64 = t*.count - 1;84 t*.numbers[row] = t*.numbers[last];85 t*.positions[row] = t*.positions[last];86 t*.velocities[row] = t*.velocities[last];87 t*.lives[row] = t*.lives[last];88 t*.count = last;89 return last;90}91 92// Systems. Each takes the columns it reads and writes, cut to the rows in use.93// noalias on the column a system writes promises nothing else reaches it, so94// the loop can be vectorized without runtime overlap checks.95move_system: proc(positions: noalias []vec2, velocities: []const vec2, dt: f32) -> void = {96 for (i: u64 = 0; i < positions.length; i += 1) {97 positions[i].x += velocities[i].x * dt;98 positions[i].y += velocities[i].y * dt;99 }100}101 102// One slice, and still noalias: on Windows the slice arrives by reference, and103// without the promise a float written through lives might be its own pointer104// or length, so clang-cl reads both again after every element.105age_system: proc(lives: noalias []f32, dt: f32) -> void = {106 for (i: u64 = 0; i < lives.length; i += 1) {107 lives[i] -= dt;108 }109}110 111// Structural change: rows leave the table. Backwards, so the row a swap-remove112// pulls in has already been tested.113expire_system: proc(t: *spark_table) -> void = {114 i: u64 = t*.count;115 while (i > 0) {116 i -= 1;117 if (t*.lives[i] <= 0.0) {118 number: u32 = t*.numbers[i];119 moved: u64 = spark_despawn(t, i);120 if (moved != i) {121 printfmt(" spark {} in row {} expired; spark {} moved from row {} into it\n",122 number, i, t*.numbers[i], moved);123 } else {124 printfmt(" spark {} in row {} expired; it was the last row\n", number, i);125 }126 }127 }128}129 130print_table: proc(t: *spark_table) -> void = {131 for (i: u64 = 0; i < t*.count; i += 1) {132 printf(" row %d spark %d position (%6.2f, %6.2f) life %.2f\n",133 cast(i, i32), cast(t*.numbers[i], i32),134 cast(t*.positions[i].x, f64), cast(t*.positions[i].y, f64), cast(t*.lives[i], f64));135 }136}137 138main: proc() -> i32 = {139 arena: memops_arena = {};140 memops_arena_initialize(arena&);141 142 sparks: spark_table = spark_table_create(arena&, 8);143 printfmt("a table with room for {} sparks, {} bytes of components a row\n", sparks.capacity,144 sizeof(vec2) * 2 + sizeof(f32));145 146 // Eight sparks with lives of 0.5s to 4s, plus one more than fits.147 for (i: u64 = 0; i < 9; i += 1) {148 f: f32 = cast(i, f32);149 row: u64 = 0;150 life: f32 = 0.5 + cast((i * 5) % 8, f32) * 0.5;151 status: spawn_status = spark_spawn(sparks&, cast(i, u32), { .x = f, .y = 0.0 }, { .x = 1.0, .y = f },152 life, row&);153 if (status == spawn_status.full) {154 printfmt("spark {} did not fit: the table is full\n", i);155 }156 }157 print_table(sparks&);158 159 dt: f32 = 0.5;160 for (frame: i32 = 1; frame <= 4; frame += 1) {161 printfmt("frame {}\n", frame);162 move_system(sparks.positions[0..sparks.count], sparks.velocities[0..sparks.count], dt);163 age_system(sparks.lives[0..sparks.count], dt);164 expire_system(sparks&);165 print_table(sparks&);166 }167 printfmt("{} sparks left\n", sparks.count);168 return 0;169}What archetype_table prints. tools/check.py fails the lesson if a single byte differs.
a table with room for 8 sparks, 20 bytes of components a row
spark 8 did not fit: the table is full
row 0 spark 0 position ( 0.00, 0.00) life 0.50
row 1 spark 1 position ( 1.00, 0.00) life 3.00
row 2 spark 2 position ( 2.00, 0.00) life 1.50
row 3 spark 3 position ( 3.00, 0.00) life 4.00
row 4 spark 4 position ( 4.00, 0.00) life 2.50
row 5 spark 5 position ( 5.00, 0.00) life 1.00
row 6 spark 6 position ( 6.00, 0.00) life 3.50
row 7 spark 7 position ( 7.00, 0.00) life 2.00
frame 1
spark 0 in row 0 expired; spark 7 moved from row 7 into it
row 0 spark 7 position ( 7.50, 3.50) life 1.50
row 1 spark 1 position ( 1.50, 0.50) life 2.50
row 2 spark 2 position ( 2.50, 1.00) life 1.00
row 3 spark 3 position ( 3.50, 1.50) life 3.50
row 4 spark 4 position ( 4.50, 2.00) life 2.00
row 5 spark 5 position ( 5.50, 2.50) life 0.50
row 6 spark 6 position ( 6.50, 3.00) life 3.00
frame 2
spark 5 in row 5 expired; spark 6 moved from row 6 into it
row 0 spark 7 position ( 8.00, 7.00) life 1.00
row 1 spark 1 position ( 2.00, 1.00) life 2.00
row 2 spark 2 position ( 3.00, 2.00) life 0.50
row 3 spark 3 position ( 4.00, 3.00) life 3.00
row 4 spark 4 position ( 5.00, 4.00) life 1.50
row 5 spark 6 position ( 7.00, 6.00) life 2.50
frame 3
spark 2 in row 2 expired; spark 6 moved from row 5 into it
row 0 spark 7 position ( 8.50, 10.50) life 0.50
row 1 spark 1 position ( 2.50, 1.50) life 1.50
row 2 spark 6 position ( 7.50, 9.00) life 2.00
row 3 spark 3 position ( 4.50, 4.50) life 2.50
row 4 spark 4 position ( 5.50, 6.00) life 1.00
frame 4
spark 7 in row 0 expired; spark 4 moved from row 4 into it
row 0 spark 4 position ( 6.00, 8.00) life 0.50
row 1 spark 1 position ( 3.00, 2.00) life 1.00
row 2 spark 6 position ( 8.00, 12.00) life 1.50
row 3 spark 3 position ( 5.00, 6.00) life 2.00
4 sparks left
Advanced · lesson 3
Many archetypes
src/3.many_archetypes/many_archetypes.rin
src/3.many_archetypes/many_archetypes.rin
In the dojo: 4. a type-erased column, 6. bitsets, each on its own, to watch and to practise.
Four archetypes (rock, ship, bullet, turret) and four components (position, velocity, health, damage). One table type holds all four archetypes, and two queries decide which tables each system visits.
A table without a type
Lesson 2's table had one typed field per column. That cannot work for every archetype at once, so the columns become bytes:
table: struct = {
signature: u64;
count: u64;
capacity: u64;
columns: [component<>.count]^u8;
}
columns[id] is null when the table has no such component. Its element size is component_info[id]*.size, where component_info holds each component's reflect record, position<>& and so on. table_add_row and lesson 4's moves work only from sizes, so they never need to know a component's type.
Types come back at the edges. column<T>(t) asks component_of<T>() for T's id, casts that column to ^T, and returns it as a []T of the rows in use. component_of is one specialisation per component, one line each. Systems take []position, []velocity and so on, exactly as in lesson 2.
Sized by reflection
component has no count member at the end. component<>.count is the number of members, from the enum's reflect record, and it sizes both columns and component_info. With a hand-added count member the reflected count is one more than the number of components. That is how the bounds check found a bug while this lesson was being written.
components_check compares each enum member's name with the name in component_info. Swap two entries in component_info and the program stops before anything runs:
component 2 is health in the enum but damage in component_info
Different types, fewer checks
move_system takes noalias []position, as lesson 2's takes noalias []vec2, and for the same reason. regen_system takes noalias []health, its only slice, for the reason lesson 2 gives for age_system.
One difference showed before either had noalias: clang on Linux vectorized this move_system without a run-time overlap check, and lesson 2's only with one. Here the two columns are different struct types, position and velocity, and clang reads C's type rules as saying they cannot share memory; built with -fno-strict-aliasing, it checks here too. In lesson 2 both are vec2. gcc checked in both. clang-cl, rin's default compiler, does not use those rules at all, to match MSVC, so on Windows both lessons were checked. noalias makes the answer the same everywhere, whatever the column types are.
Queries
moving: query = { .all = bit(component.position) | bit(component.velocity) };
healing: query = { .all = bit(component.health), .none = bit(component.damage) };
A table matches when it has every bit in all and none in none. moving visits ships and bullets. healing visits ships but not turrets, which have health but also deal damage. The output prints each visit.
The signature cannot lie
table_create builds the signature from the component list it allocates columns for. kayan wrote them separately, and archetype D's signature left out two of its columns (see 0_from_kayan.md).
Try
- Add an
armorcomponent, give it to turrets, and make a query that visits only tables with bothhealthandarmor. - Ask a rock for
column<velocity>. Read the message, then find where it comes from. - Change
columnsto[4]^u8. Then add a fifth component and see what happens, and where. - Write
table_remove_rowwith the swap-remove from lesson 2, working only fromcomponent_infosizes andmemcpy.
1// 3. Many archetypes2//3// A game has more than one combination of components. Rocks only have a4// position. Ships move and can be hurt. Bullets move and hurt. Turrets stand5// still, can be hurt and hurt back. Each combination is a table of its own, and6// a system has to visit every table that has the columns it needs.7//8// kayan wrote a C struct per archetype, gave each a hand-set signature, and at9// query time looked each column up by name in a string map. Here one table type10// serves every archetype:11//12// * A component is a member of the `component` enum, so its id is a small13// integer, and a set of components fits in a u64 mask, one bit each.14// * A table's columns are `[component<>.count]^u8`, indexed by component id15// and null where the table has no such column. Finding a column is an array16// index, not a string lookup.17// * A column's element size comes from reflection: `component_info[id]` is the18// component type's reflect record, `position<>&` and so on.19// * A table's signature is computed from the components it is created with,20// so the signature cannot disagree with the columns. In kayan it could, and21// it did: archetype D held positions and velocities, its signature left them22// out, and the movement query skipped it.23//24// A query is two masks: a table matches when it has every component in `all`25// and none in `none`. Typed access comes back through generics: `column<T>`26// asks `component_of<T>()` which id T has, one line per component type, and27// returns that column as a []T cut to the rows in use.28 29import "std/print.rin"30import "std/memops.rin"31 32vec2: struct = {33 x: f32;34 y: f32;35}36 37// The components. Each enum member is named after its type, so the program can38// check at startup that this list and `component_info` agree. There is no39// `count` member: `component<>.count` is the number of members, from reflection.40component: enum = {41 position,42 velocity,43 health,44 damage,45}46 47position: struct = { at: vec2; }48velocity: struct = { per_second: vec2; }49health: struct = { points: f32; max: f32; }50damage: struct = { per_hit: f32; }51 52component_info: [component<>.count]*const reflect = {53 position<>&,54 velocity<>&,55 health<>&,56 damage<>&,57};58 59component_of: proc<position>() -> component = { return component.position; }60component_of: proc<velocity>() -> component = { return component.velocity; }61component_of: proc<health>() -> component = { return component.health; }62component_of: proc<damage>() -> component = { return component.damage; }63 64bit: proc(c: component) -> u64 = {65 return 1'u64 shl cast(c, u64);66}67 68// One archetype. Every column has `capacity` elements of its component's size.69table: struct = {70 name: ^const c8;71 signature: u64;72 count: u64;73 capacity: u64;74 columns: [component<>.count]^u8;75}76 77query: struct = {78 all: u64;79 none: u64;80}81 82// Stops the program if the enum and component_info were written in different83// orders: a column would silently get another component's size.84components_check: proc() -> void = {85 for (i: u64 = 0; i < component<>.count; i += 1) {86 if (strcmp(component<>.variant.values[i].name, component_info[i]*.name) != 0) {87 printfmt("component {} is {} in the enum but {} in component_info\n", i,88 component<>.variant.values[i].name, component_info[i]*.name);89 exit(1);90 }91 }92}93 94table_create: proc(arena: *memops_arena, name: ^const c8, components: []const component,95 capacity: u64) -> table = {96 t: table = {};97 t.name = name;98 t.capacity = capacity;99 for (i: u64 = 0; i < components.length; i += 1) {100 c: component = components[i];101 info: *const reflect = component_info[c];102 t.signature |= bit(c);103 t.columns[c] = cast(memops_arena_push_zero(arena, info*.size * capacity, info*.align), ^u8);104 }105 return t;106}107 108table_has: proc(t: *table, c: component) -> b32 = {109 return (t*.signature & bit(c)) != 0;110}111 112table_matches: proc(t: *table, q: query) -> b32 = {113 return (t*.signature & q.all) == q.all && (t*.signature & q.none) == 0;114}115 116// A new row with every component zeroed. The caller fills it in.117table_add_row: proc(t: *table) -> u64 = {118 if (t*.count == t*.capacity) {119 printfmt("table {} is full\n", t*.name);120 exit(1);121 }122 row: u64 = t*.count;123 for (c: u64 = 0; c < component<>.count; c += 1) {124 if (t*.columns[c] != null) {125 size: u64 = component_info[c]*.size;126 memset(t*.columns[c][row * size]&, 0, size);127 }128 }129 t*.count += 1;130 return row;131}132 133// The rows in use of T's column. Asking a table for a column it does not have134// is a bug in the caller, so it stops the program rather than return nothing.135column: proc<T>(t: *table) -> []T = {136 c: component = component_of<T>();137 if (t*.columns[c] == null) {138 printfmt("table {} has no {} column\n", t*.name, component_info[c]*.name);139 exit(1);140 }141 raw: ^T = cast(t*.columns[c], ^T);142 return raw[0..t*.count];143}144 145table_set: proc<T>(t: *table, row: u64, value: T) -> void = {146 values: []T = column<T>(t);147 values[row] = value;148}149 150// Systems take columns, as in lesson 2. The loops that feed them are queries.151move_system: proc(positions: noalias []position, velocities: []const velocity, dt: f32) -> void = {152 for (i: u64 = 0; i < positions.length; i += 1) {153 positions[i].at.x += velocities[i].per_second.x * dt;154 positions[i].at.y += velocities[i].per_second.y * dt;155 }156}157 158regen_system: proc(healths: noalias []health, per_second: f32, dt: f32) -> void = {159 for (i: u64 = 0; i < healths.length; i += 1) {160 healths[i].points += per_second * dt;161 if (healths[i].points > healths[i].max) {162 healths[i].points = healths[i].max;163 }164 }165}166 167print_tables: proc(tables: []table) -> void = {168 for (ti: u64 = 0; ti < tables.length; ti += 1) {169 t: *table = tables[ti]&;170 for (row: u64 = 0; row < t*.count; row += 1) {171 printf(" %-7s row %d", t*.name, cast(row, i32));172 if (table_has(t, component.position)) {173 ps: []position = column<position>(t);174 printf(" at (%5.1f, %5.1f)", cast(ps[row].at.x, f64), cast(ps[row].at.y, f64));175 }176 if (table_has(t, component.health)) {177 hs: []health = column<health>(t);178 printf(" health %5.1f/%.0f", cast(hs[row].points, f64), cast(hs[row].max, f64));179 }180 if (table_has(t, component.damage)) {181 ds: []damage = column<damage>(t);182 printf(" damage %.0f", cast(ds[row].per_hit, f64));183 }184 printf("\n");185 }186 }187}188 189spawn_ship: proc(t: *table, x: f32, y: f32) -> void = {190 row: u64 = table_add_row(t);191 table_set<position>(t, row, { .at = { .x = x, .y = y } });192 table_set<velocity>(t, row, { .per_second = { .x = 2.0, .y = 0.0 } });193 table_set<health>(t, row, { .points = 50.0, .max = 100.0 });194}195 196spawn_bullet: proc(t: *table, x: f32, y: f32, dy: f32) -> void = {197 row: u64 = table_add_row(t);198 table_set<position>(t, row, { .at = { .x = x, .y = y } });199 table_set<velocity>(t, row, { .per_second = { .x = 0.0, .y = dy } });200 table_set<damage>(t, row, { .per_hit = 10.0 });201}202 203main: proc() -> i32 = {204 components_check();205 206 arena: memops_arena = {};207 memops_arena_initialize(arena&);208 209 rock_components: [1]component = { component.position };210 ship_components: [3]component = { component.position, component.velocity, component.health };211 bullet_components: [3]component = { component.position, component.velocity, component.damage };212 turret_components: [3]component = { component.position, component.health, component.damage };213 214 tables: [4]table = {};215 tables[0] = table_create(arena&, "rock", rock_components, 16);216 tables[1] = table_create(arena&, "ship", ship_components, 16);217 tables[2] = table_create(arena&, "bullet", bullet_components, 16);218 tables[3] = table_create(arena&, "turret", turret_components, 16);219 220 for (ti: u64 = 0; ti < 4; ti += 1) {221 printfmt("{} signature {}:", tables[ti].name, tables[ti].signature);222 for (c: u64 = 0; c < component<>.count; c += 1) {223 if ((tables[ti].signature & (1'u64 shl c)) != 0) {224 printfmt(" {}", component_info[c]*.name);225 }226 }227 printfmt("\n");228 }229 230 rock: u64 = table_add_row(tables[0]&);231 table_set<position>(tables[0]&, rock, { .at = { .x = 5.0, .y = 5.0 } });232 spawn_ship(tables[1]&, 0.0, 0.0);233 spawn_ship(tables[1]&, 0.0, 10.0);234 spawn_bullet(tables[2]&, 1.0, 0.0, 8.0);235 spawn_bullet(tables[2]&, 2.0, 0.0, 8.0);236 spawn_bullet(tables[2]&, 3.0, 0.0, -8.0);237 turret: u64 = table_add_row(tables[3]&);238 table_set<position>(tables[3]&, turret, { .at = { .x = 20.0, .y = 0.0 } });239 table_set<health>(tables[3]&, turret, { .points = 40.0, .max = 200.0 });240 table_set<damage>(tables[3]&, turret, { .per_hit = 25.0 });241 print_tables(tables);242 243 moving: query = { .all = bit(component.position) | bit(component.velocity) };244 // Turrets have health too, but they repair by other means: no regen for245 // anything that deals damage.246 healing: query = { .all = bit(component.health), .none = bit(component.damage) };247 248 dt: f32 = 0.5;249 for (frame: i32 = 1; frame <= 2; frame += 1) {250 printfmt("frame {}\n", frame);251 for (ti: u64 = 0; ti < 4; ti += 1) {252 t: *table = tables[ti]&;253 if (table_matches(t, moving)) {254 printfmt(" move visits {}\n", t*.name);255 // move_system takes []const velocity. Widening a call's result256 // reads it twice, so rin wants it in a local first.257 velocities: []velocity = column<velocity>(t);258 move_system(column<position>(t), velocities, dt);259 }260 if (table_matches(t, healing)) {261 printfmt(" regen visits {}\n", t*.name);262 regen_system(column<health>(t), 10.0, dt);263 }264 }265 print_tables(tables);266 }267 return 0;268}What many_archetypes prints. tools/check.py fails the lesson if a single byte differs.
rock signature 1: position
ship signature 7: position velocity health
bullet signature 11: position velocity damage
turret signature 13: position health damage
rock row 0 at ( 5.0, 5.0)
ship row 0 at ( 0.0, 0.0) health 50.0/100
ship row 1 at ( 0.0, 10.0) health 50.0/100
bullet row 0 at ( 1.0, 0.0) damage 10
bullet row 1 at ( 2.0, 0.0) damage 10
bullet row 2 at ( 3.0, 0.0) damage 10
turret row 0 at ( 20.0, 0.0) health 40.0/200 damage 25
frame 1
move visits ship
regen visits ship
move visits bullet
rock row 0 at ( 5.0, 5.0)
ship row 0 at ( 1.0, 0.0) health 55.0/100
ship row 1 at ( 1.0, 10.0) health 55.0/100
bullet row 0 at ( 1.0, 4.0) damage 10
bullet row 1 at ( 2.0, 4.0) damage 10
bullet row 2 at ( 3.0, -4.0) damage 10
turret row 0 at ( 20.0, 0.0) health 40.0/200 damage 25
frame 2
move visits ship
regen visits ship
move visits bullet
rock row 0 at ( 5.0, 5.0)
ship row 0 at ( 2.0, 0.0) health 60.0/100
ship row 1 at ( 2.0, 10.0) health 60.0/100
bullet row 0 at ( 1.0, 8.0) damage 10
bullet row 1 at ( 2.0, 8.0) damage 10
bullet row 2 at ( 3.0, -8.0) damage 10
turret row 0 at ( 20.0, 0.0) health 40.0/200 damage 25
Advanced · lesson 4
Entity handles
src/4.entity_handles/entity_handles.rin
src/4.entity_handles/entity_handles.rin, includes/kayan.rin
In the dojo: 1. free list and generations, 3. the slot map, 9. interning, each on its own, to watch and to practise.
The world: tables made on demand, entity handles that survive row moves, despawning with index reuse, and moving an entity between archetypes when a component is added or removed. The code is includes/kayan.rin. The lesson program calls each part and prints what happened.
Handles, slots, rows
kayan_entity: struct = { index: u32; generation: u32; }
kayan_slot: struct = { generation: u32; table: u32; row: u32; }
A handle is what the rest of a program keeps. Its index picks a slot, and the slot says where the entity lives now. Rows move all the time: every despawn and every archetype change swap-removes. When one does, the table's entities column says which entity owned the moved row, and that entity's slot gets the new row number. Handles never change.
generation is what makes a handle safe to keep after its entity dies. Despawning bumps the slot's generation and frees the index, and the next spawn reuses the index with the new generation:
spawn d: it reuses a's index with the next generation
d = 0:2 in table 0 row 2 {position velocity} at (4.0, 0.0)
a = 0:1 is dead
kayan_alive is one comparison: the handle's generation against the slot's. A free slot already holds its next generation, which no handle carries yet, so no separate "alive" flag is needed. a cannot reach d's data, kayan_get returns null for it, and despawning it again returns kayan_status.dead and changes nothing.
Generations start at 1. A zeroed handle, {}, is 0:0, which never matches a slot, so it can stand for "no entity" without a separate flag.
Statuses
kayan_despawn, kayan_add, kayan_remove and kayan_move return kayan_status, an enum[must_use]: ok, or dead when the handle no longer names a live entity. Ignoring one is a compile error. The lesson discards them on purpose with _ = kayan_add<health>(...) where the handle is known to be alive.
Tables on demand
kayan_table_for(w, signature) looks the signature up in a hashmap<u64, u32> and makes the table if it is new. Nobody declares archetypes; they appear when an entity first needs that combination. Tables live in a vec<kayan_table> and are referred to by index, because adding one can move the whole list.
Tables start with room for 16 rows and double when full. The old columns stay in the arena unused, which is a waste these lessons accept. The last part of the program spawns 40 entities so the first table grows twice, despawns every other one, and checks the result two ways.
Checking the world
Reading back each survivor's position is a weak check. Swap-remove copies rows, so a stale row often still holds the right numbers. kayan_verify checks the invariant instead: every row of every table belongs to a live entity whose slot points back at that table and row, and the rows add up to the live count. It prints each problem it finds and returns how many there were. The lesson prints kayan_verify: 0 problems and fails if that is not so.
kayan_get also checks the slot's row against the rows in use, so in Debug a slot that points past the end of its table stops the program at the first lookup.
Moving between archetypes
give b health: b moves to a new table and keeps its position
kayan_add<health> computes signature | bit(health), finds or makes that table, appends a row, copies every component the two tables share, then swap-removes the old row. kayan_remove is the same move with the bit cleared. A move costs two row copies, one into the new table and one to fill the hole in the old, so components that come and go every frame are better as a field than as a component. A burning flag in health, say, instead of a burning component.
Try
- Despawn
band keep using its handle. Which calls notice, and which would not ifkayan_alivedid not check the generation? - Remove the line in
kayan_table_remove_rowthat updates the moved entity's slot. In Debug the program stops at the firstkayan_getthrough a stale row. Build it with-DNDEBUGand run it again: the position check still says all 20 are fine, andkayan_verifylists every broken slot. - At the end of the lesson, table 0 has 22 rows and room for 64. Hold a
[]positionfromkayan_column, spawn at least 43 more so the table grows, then write through the old slice and read the position back throughkayan_get. The write is lost, and nothing reports it. That stale slice is why lesson 6 defers spawns. - Add
kayan_has<T>(w, e) -> b32without moving anything.
Imports the world from includes/kayan.rin.
1// 4. Entity handles2//3// Lesson 2 ended on a problem: swap-remove keeps a table packed, but it moves4// rows, so "spark 7 is row 7" stops being true the moment something before it5// dies. Anything that remembers an entity by its row (a target, a parent, a UI6// selection) ends up pointing at whoever moved in.7//8// The fix is one level of indirection that does not move:9//10// * A handle is {index, generation}. The index picks a slot in the world's11// slot list; the slot says which table and row the entity is in now.12// * Every table keeps an `entities` column: the handle that owns each row.13// When a swap-remove moves the last row, that column says whose slot to14// update. That is the line lesson 2 could not write.15// * Despawning bumps the slot's generation and puts the index on a free list.16// The next spawn reuses the index with the new generation, so every old17// handle to the dead entity fails the generation check instead of quietly18// finding the newcomer. Generations start at 1, so a zeroed handle is never19// alive and can mean "no entity".20//21// The same slot is what lets an entity change archetype. Adding a component22// moves the entity to the table with one more bit: a row is appended there,23// the components both tables share are copied, and the old row is24// swap-removed. Removing a component is the same move the other way. kayan had25// no way to do either: its entity id was a running count, and nothing recorded26// where an entity lived.27//28// The world itself is in includes/kayan.rin, which lessons 5 and 6 use as it29// is. This program walks through what it does and prints each step.30 31import "../../includes/kayan.rin"32 33vec2: struct = {34 x: f32;35 y: f32;36}37 38component: enum = {39 position,40 velocity,41 health,42}43 44position: struct = { at: vec2; }45velocity: struct = { per_second: vec2; }46health: struct = { points: f32; }47 48component_of: proc<position>() -> component = { return component.position; }49component_of: proc<velocity>() -> component = { return component.velocity; }50component_of: proc<health>() -> component = { return component.health; }51 52register_components: proc(w: *kayan_world) -> void = {53 kayan_register(w, kayan_id_of<position>(), position<>&);54 kayan_register(w, kayan_id_of<velocity>(), velocity<>&);55 kayan_register(w, kayan_id_of<health>(), health<>&);56}57 58print_entity: proc(w: *kayan_world, name: ^const c8, e: kayan_entity) -> void = {59 printf(" %s = %u:%u", name, e.index, e.generation);60 if (!kayan_alive(w, e)) {61 printf(" is dead\n");62 return;63 }64 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));65 printf(" in table %u row %u ", slot*.table, slot*.row);66 kayan_print_signature(w, kayan_signature(w, e));67 p: *position = kayan_get<position>(w, e);68 if (p != null) {69 printf(" at (%.1f, %.1f)", cast(p*.at.x, f64), cast(p*.at.y, f64));70 }71 h: *health = kayan_get<health>(w, e);72 if (h != null) {73 printf(" health %.0f", cast(h*.points, f64));74 }75 printf("\n");76}77 78spawn_mover: proc(w: *kayan_world, x: f32) -> kayan_entity = {79 e: kayan_entity = kayan_spawn(w, kayan_mask_of<position>() | kayan_mask_of<velocity>());80 kayan_get<position>(w, e)* = { .at = { .x = x, .y = 0.0 } };81 kayan_get<velocity>(w, e)* = { .per_second = { .x = 0.0, .y = 1.0 } };82 return e;83}84 85main: proc() -> i32 = {86 arena: memops_arena = {};87 memops_arena_initialize(arena&);88 world: kayan_world = {};89 kayan_world_init(world&, arena&);90 register_components(world&);91 92 printfmt("spawn a, b, c\n");93 a: kayan_entity = spawn_mover(world&, 1.0);94 b: kayan_entity = spawn_mover(world&, 2.0);95 c: kayan_entity = spawn_mover(world&, 3.0);96 print_entity(world&, "a", a);97 print_entity(world&, "b", b);98 print_entity(world&, "c", c);99 100 printfmt("despawn a: c is swapped into a's row, and its slot follows\n");101 despawned: kayan_status = kayan_despawn(world&, a);102 printfmt(" despawn returned {}\n", despawned);103 print_entity(world&, "a", a);104 print_entity(world&, "c", c);105 106 printfmt("spawn d: it reuses a's index with the next generation\n");107 d: kayan_entity = spawn_mover(world&, 4.0);108 print_entity(world&, "d", d);109 print_entity(world&, "a", a);110 printfmt(" a's position through the old handle is {}\n",111 kayan_get<position>(world&, a) == null ? "null" : "found (a bug)");112 printfmt(" despawning a again returns {}\n", kayan_despawn(world&, a));113 nobody: kayan_entity = {};114 printfmt(" a zeroed handle {}:{} is {}\n", nobody.index, nobody.generation,115 kayan_alive(world&, nobody) ? "alive (a bug)" : "dead");116 117 printfmt("give b health: b moves to a new table and keeps its position\n");118 _ = kayan_add<health>(world&, b, { .points = 75.0 });119 print_entity(world&, "b", b);120 print_entity(world&, "d", d);121 122 printfmt("take b's velocity: another move\n");123 _ = kayan_remove<velocity>(world&, b);124 print_entity(world&, "b", b);125 126 printfmt("the tables, with the handles in each row\n");127 kayan_print_tables(world&);128 129 // Enough spawns to make a table grow past its first 16 rows. Then two130 // checks: every survivor still reads its own position, and kayan_verify131 // finds every row's owner and every owner's slot in agreement.132 printfmt("spawn 40 more\n");133 many: [40]kayan_entity = {};134 for (i: u64 = 0; i < 40; i += 1) {135 many[i] = spawn_mover(world&, cast(i, f32));136 }137 for (i: u64 = 0; i < 40; i += 2) {138 _ = kayan_despawn(world&, many[i]);139 }140 wrong: u32 = 0;141 for (i: u64 = 1; i < 40; i += 2) {142 p: *position = kayan_get<position>(world&, many[i]);143 if (p == null || p*.at.x != cast(i, f32)) {144 wrong += 1;145 }146 }147 problems: u64 = kayan_verify(world&);148 printfmt(" despawned every other one; {} of the 20 left read their own position\n", 20 - wrong);149 printfmt(" kayan_verify: {} problems\n", problems);150 printfmt(" {} entities alive, {} tables\n", world.live, world.tables.length);151 return wrong == 0 && problems == 0 ? 0 : 1;152}What entity_handles prints. tools/check.py fails the lesson if a single byte differs.
spawn a, b, c
a = 0:1 in table 0 row 0 {position velocity} at (1.0, 0.0)
b = 1:1 in table 0 row 1 {position velocity} at (2.0, 0.0)
c = 2:1 in table 0 row 2 {position velocity} at (3.0, 0.0)
despawn a: c is swapped into a's row, and its slot follows
despawn returned ok
a = 0:1 is dead
c = 2:1 in table 0 row 0 {position velocity} at (3.0, 0.0)
spawn d: it reuses a's index with the next generation
d = 0:2 in table 0 row 2 {position velocity} at (4.0, 0.0)
a = 0:1 is dead
a's position through the old handle is null
despawning a again returns dead
a zeroed handle 0:0 is dead
give b health: b moves to a new table and keeps its position
b = 1:1 in table 1 row 0 {position velocity health} at (2.0, 0.0) health 75
d = 0:2 in table 0 row 1 {position velocity} at (4.0, 0.0)
take b's velocity: another move
b = 1:1 in table 2 row 0 {position health} at (2.0, 0.0) health 75
the tables, with the handles in each row
table 0 {position velocity}: 2:1 0:2
table 1 {position velocity health}:
table 2 {position health}: 1:1
spawn 40 more
despawned every other one; 20 of the 20 left read their own position
kayan_verify: 0 problems
23 entities alive, 3 tables
Advanced · lesson 5
The inspector
src/5.inspector/inspector.rin
src/5.inspector/inspector.rin
In the dojo: 11. a reflection walk, on its own, to watch and to practise.
A soldier and a crate, printed field by field by code that names no component type, then edited by component and field name, then a table layout report. Everything comes from the reflect records the world already holds for sizes.
What a reflect record has
T<> is a reflect record: name, size, align, kind, count, and for a struct, count fields. Each field has its name, offset, size, type ("f32" for a builtin, "array_16_c8" for a [16]c8), elem_type and array_count for arrays, and info, the record of the field's own type when it has one. A nested struct or an enum field has info; a builtin like f32 does not.
print_fields walks that. It takes the component's bytes as a []const u8, from kayan_get_bytes, and cuts each field's bytes out of it by offset and size, so a record that disagreed with the data would stop the program in Debug. Struct fields recurse through info, enum fields print their member name through reflect_name_from_value_or, a [16]c8 prints as text, and builtins print by type name. Because components are structs, even a single float like health.points has a name to print.
Tags
health: struct = { points: f32; max: f32; regen_timer: f32 @ "hidden"; }
@ "hidden" is a string attached to the field, read with reflect_field_has_attr. The inspector skips it. The component decides what a tool shows, and the tool does not need to know the component.
Names belong in tools
set_f32_by_name(w, e, "health", "points", 55.0) finds the component by name, the field by name, checks the field is an f32, and writes it. It refuses weapon.ammo, a u32, and the crate's health, which it does not have. That is the flexibility an editor or a console needs. It is also the string comparison kayan did in every query, once per archetype. Here it runs once per edit, and systems keep using ids.
Layout
print_layout lists each table's columns with size and alignment, and each column's bytes in use out of the bytes allocated for the table's capacity. It is the kind of report that shows when a component has grown padding, when a table has far more room than rows, or when a table carries a column most of its systems never read.
Try
- Add a
bool-likeb32field toweaponand teachprint_builtinto print it asyesorno. - Give
positiona@ "readonly"tag and makeset_f32_by_namerefuse fields that carry it. - Write
set_by_nameforu32andi32as well, picking the write fromfield.type. - Print a whole table as rows and columns: one line per entity, one column per field.
Imports the world from includes/kayan.rin.
1// 5. The inspector2//3// Type-erased columns are fast and opaque: to a debugger a column is `^u8`.4// But every component already has a reflect record, registered in lesson 4 so5// the world knew its size. The same record lists the component's fields with6// their names, offsets, sizes and types, and links each struct or enum field to7// a record of its own. That is enough to print any entity, field by field,8// without writing a print proc for any component.9//10// This is where looking things up by name belongs. kayan looked up columns by11// name in every query, once per archetype, to find what to move. Here systems still12// find columns by id, as in lesson 3, and names are only used by tools: this13// inspector, and the editor-style write at the end that sets a field from its14// component and field names.15//16// Fields can carry tags with `@`. The inspector skips anything tagged "hidden",17// which is how a component keeps bookkeeping out of a tool without the tool18// knowing that component exists.19 20import "../../includes/kayan.rin"21 22vec2: struct = {23 x: f32;24 y: f32;25}26 27side: enum = {28 red,29 blue,30}31 32component: enum = {33 position,34 health,35 team,36 weapon,37 label,38}39 40position: struct = { at: vec2; heading: f32; }41health: struct = { points: f32; max: f32; regen_timer: f32 @ "hidden"; }42team: struct = { side: side; }43weapon: struct = { damage: f32; cooldown: f32; ammo: u32; }44label: struct = { text: [16]c8; }45 46component_of: proc<position>() -> component = { return component.position; }47component_of: proc<health>() -> component = { return component.health; }48component_of: proc<team>() -> component = { return component.team; }49component_of: proc<weapon>() -> component = { return component.weapon; }50component_of: proc<label>() -> component = { return component.label; }51 52register_components: proc(w: *kayan_world) -> void = {53 kayan_register(w, kayan_id_of<position>(), position<>&);54 kayan_register(w, kayan_id_of<health>(), health<>&);55 kayan_register(w, kayan_id_of<team>(), team<>&);56 kayan_register(w, kayan_id_of<weapon>(), weapon<>&);57 kayan_register(w, kayan_id_of<label>(), label<>&);58}59 60indent: proc(depth: u32) -> void = {61 for (i: u32 = 0; i < depth; i += 1) {62 printf(" ");63 }64}65 66is_type: proc(field: *const reflect_field, name: ^const char) -> b32 = {67 return strcmp(field*.type, name) == 0;68}69 70// One value of a builtin type. The C types behind the names are fixed by rin,71// so reading through a pointer of the right width is exact.72print_builtin: proc(type_name: ^const char, at: []const u8) -> void = {73 if (strcmp(type_name, "f32") == 0) {74 printf("%.2f", cast(cast(at[0]&, *const f32)*, f64));75 } else if (strcmp(type_name, "u32") == 0) {76 printf("%u", cast(at[0]&, *const u32)*);77 } else if (strcmp(type_name, "i32") == 0) {78 printf("%d", cast(at[0]&, *const i32)*);79 } else {80 printf("<%s>", type_name);81 }82}83 84// Every field of a struct whose bytes are `base`, recursing into struct85// fields. Enum fields print their member name; a [N]c8 prints as text. Each86// field's bytes are cut from `base` by its offset and size, so a record that87// disagreed with the data would stop the program in Debug.88print_fields: proc(type: *const reflect, base: []const u8, depth: u32) -> void = {89 fields: ^const reflect_field = reflect_fields(type);90 for (i: u64 = 0; i < type*.count; i += 1) {91 f: *const reflect_field = fields[i]&;92 if (reflect_field_has_attr(f, "hidden")) {93 continue;94 }95 at: []const u8 = base[f*.offset..f*.offset + f*.size];96 indent(depth);97 printf("%s:", f*.name);98 info: *const reflect = f*.info;99 if (reflect_field_is_array(f) && strcmp(f*.elem_type, "c8") == 0) {100 printf(" \"%.*s\"\n", cast(f*.array_count, i32), cast(at.data, ^const c8));101 } else if (info != null && info*.kind == reflect_kind_struct) {102 printf("\n");103 print_fields(info, at, depth + 1);104 } else if (info != null && info*.kind == reflect_kind_enum) {105 value: i32 = cast(at[0]&, *const i32)*;106 printf(" %s\n", reflect_name_from_value_or(info, value, "(not a member)"));107 } else {108 printf(" ");109 print_builtin(f*.type, at);110 printf("\n");111 }112 }113}114 115inspect: proc(w: *kayan_world, name: ^const char, e: kayan_entity) -> void = {116 printf("%s (%u:%u)\n", name, e.index, e.generation);117 signature: u64 = kayan_signature(w, e);118 for (id: u32 = 0; id < w*.component_count; id += 1) {119 if ((signature & kayan_bit(id)) != 0) {120 info: *const reflect = w*.infos[id];121 printf(" %s\n", info*.name);122 // A local first: widening a call's []u8 to []const u8 reads it twice.123 bytes: []u8 = kayan_get_bytes(w, e, id);124 print_fields(info, bytes, 2);125 }126 }127}128 129// What each table holds: per component, its size, and its column's bytes in use130// out of the bytes allocated for the table's capacity.131print_layout: proc(w: *kayan_world) -> void = {132 for (ti: u64 = 0; ti < w*.tables.length; ti += 1) {133 t: *kayan_table = vec<kayan_table>at(w*.tables&, ti);134 printf("table %d ", cast(ti, i32));135 kayan_print_signature(w, t*.signature);136 printf(", rows in use: %d of %d\n", cast(t*.count, i32), cast(t*.capacity, i32));137 for (id: u32 = 0; id < w*.component_count; id += 1) {138 if (t*.columns[id] != null) {139 info: *const reflect = w*.infos[id];140 printf(" %-8s %2d bytes, align %d, column %d of %d bytes\n", info*.name,141 cast(info*.size, i32), cast(info*.align, i32),142 cast(info*.size * t*.count, i32), cast(info*.size * t*.capacity, i32));143 }144 }145 }146}147 148// The editor's write: set an f32 field by component and field name. Slow and149// flexible, which is right for a tool and wrong for a system.150set_f32_by_name: proc(w: *kayan_world, e: kayan_entity, component_name: ^const char,151 field_name: ^const char, value: f32) -> b32 = {152 for (id: u32 = 0; id < w*.component_count; id += 1) {153 info: *const reflect = w*.infos[id];154 if (strcmp(info*.name, component_name) != 0) {155 continue;156 }157 bytes: []u8 = kayan_get_bytes(w, e, id);158 f: *const reflect_field = reflect_find_field(info, field_name);159 if (bytes.length == 0 || f == null || !is_type(f, "f32")) {160 return 0;161 }162 target: *f32 = cast(bytes[f*.offset]&, *f32);163 target* = value;164 return 1;165 }166 return 0;167}168 169// Up to 15 characters, the rest of the array zeroed, so a shorter label does170// not keep the tail of a longer one.171set_label: proc(w: *kayan_world, e: kayan_entity, text: ^const c8) -> void = {172 l: *label = kayan_get<label>(w, e);173 if (l == null) {174 return;175 }176 l* = {};177 for (i: u64 = 0; i < 15 && text[i] != 0; i += 1) {178 l*.text[i] = text[i];179 }180}181 182main: proc() -> i32 = {183 arena: memops_arena = {};184 memops_arena_initialize(arena&);185 world: kayan_world = {};186 kayan_world_init(world&, arena&);187 register_components(world&);188 189 soldier: kayan_entity = kayan_spawn(world&, kayan_mask_of<position>() | kayan_mask_of<health>()190 | kayan_mask_of<team>() | kayan_mask_of<weapon>() | kayan_mask_of<label>());191 kayan_get<position>(world&, soldier)* = { .at = { .x = 3.0, .y = 4.0 }, .heading = 90.0 };192 kayan_get<health>(world&, soldier)* = { .points = 80.0, .max = 100.0, .regen_timer = 2.5 };193 kayan_get<team>(world&, soldier)* = { .side = side.blue };194 kayan_get<weapon>(world&, soldier)* = { .damage = 12.5, .cooldown = 0.25, .ammo = 30 };195 set_label(world&, soldier, "soldier");196 197 crate: kayan_entity = kayan_spawn(world&, kayan_mask_of<position>() | kayan_mask_of<label>());198 kayan_get<position>(world&, crate)* = { .at = { .x = -1.0, .y = 0.5 } };199 set_label(world&, crate, "crate");200 201 inspect(world&, "soldier", soldier);202 inspect(world&, "crate", crate);203 204 printfmt("\nthe editor writes three fields by name\n");205 printfmt(" soldier health.points = 55: {}\n",206 set_f32_by_name(world&, soldier, "health", "points", 55.0) ? "written" : "refused");207 printfmt(" soldier weapon.ammo = 1: {}, ammo is a u32, not an f32\n",208 set_f32_by_name(world&, soldier, "weapon", "ammo", 1.0) ? "written" : "refused");209 printfmt(" crate health.points = 1: {}, the crate has no health\n",210 set_f32_by_name(world&, crate, "health", "points", 1.0) ? "written" : "refused");211 inspect(world&, "soldier", soldier);212 213 printfmt("\n");214 print_layout(world&);215 return 0;216}What inspector prints. tools/check.py fails the lesson if a single byte differs.
soldier (0:1)
position
at:
x: 3.00
y: 4.00
heading: 90.00
health
points: 80.00
max: 100.00
team
side: blue
weapon
damage: 12.50
cooldown: 0.25
ammo: 30
label
text: "soldier"
crate (1:1)
position
at:
x: -1.00
y: 0.50
heading: 0.00
label
text: "crate"
the editor writes three fields by name
soldier health.points = 55: written
soldier weapon.ammo = 1: refused, ammo is a u32, not an f32
crate health.points = 1: refused, the crate has no health
soldier (0:1)
position
at:
x: 3.00
y: 4.00
heading: 90.00
health
points: 55.00
max: 100.00
team
side: blue
weapon
damage: 12.50
cooldown: 0.25
ammo: 30
label
text: "soldier"
table 0 {position health team weapon label}, rows in use: 1 of 16
position 12 bytes, align 4, column 12 of 192 bytes
health 12 bytes, align 4, column 12 of 192 bytes
team 4 bytes, align 4, column 4 of 64 bytes
weapon 12 bytes, align 4, column 12 of 192 bytes
label 16 bytes, align 1, column 16 of 256 bytes
table 1 {position label}, rows in use: 1 of 16
position 12 bytes, align 4, column 12 of 192 bytes
label 16 bytes, align 1, column 16 of 256 bytes
Advanced · lesson 6
kayan
src/6.kayan/kayan.rin
src/6.kayan/kayan.rin
In the dojo: 5. a command buffer, 7. an arena, each on its own, to watch and to practise.
A turret at the centre, five wings of three drones, 140 frames. Seven systems run each frame, and every structural change they want goes through a command buffer applied at the end of the frame.
The frame
spawner wings of drones enter records spawns
turret shoot at the nearest drone records spawns
movement position += velocity * dt writes
lifetime bullets expire records despawns
collision live bullets against drones writes health, records despawn and add
burning burning drones lose health writes
outcome destroy dead drones, catch leaks records despawns
blast a destroyed drone hurts its neighbours writes health, records adds
apply every recorded change, in order
Writes to components happen in place, during queries. Changes to which table an entity is in are recorded, and applied once nothing is iterating.
Lifetime runs before collision, and collision skips bullets whose lifetime is used up. Those bullets are only still in their table because nothing structural happens until the end of the frame. Without the skip, a bullet that hit on its last frame would be despawned twice and counted as both a hit and a miss.
Systems and kernels
movement_system and burning_system are each a query and a kernel:
integrate: proc(ps: noalias []position, vs: []const velocity, dt: f32) -> void
burn: proc(hs: noalias []health, bs: []const burning, dt: f32) -> void
The system walks the tables and calls the kernel once for each table that matches. The kernel sees two columns, so the compiler sees a straight loop over two arrays. noalias qualifies parameters, so the loop needs a proc of its own to carry it; that is the only reason these two are split out. The output does not change.
Before the split, gcc -O3 vectorized both loops behind a run-time check that the columns were apart. After it, without the check. clang on Linux had no check either way, for the reason lesson 3 gives: the columns are different types. clang-cl, which does not use C's type rules, needs noalias here too.
burn is the cleanest loop in the game. health and burning are each one f32, so their columns are plain runs of floats, and the kernel is eight floats per pass with nothing to rearrange: loads, mulps, subps, stores. integrate works on vec2s and spends some of each pass separating x from y; lesson 1 shows what the fully split layout looks like.
integrate and burn are streams: every row of every matching table, in order. collision_system and blast_system are closer to lookups: for each bullet or each blast, they visit drones that could be anywhere. Lesson 7 measures what each layout costs for both kinds of loop, and why the components here stay small structs.
lifetime_system stays one proc. Its loop records a despawn for each expired bullet, and a loop that calls something per row is not vectorized whatever its parameters say. The other systems branch, search or record in their loops too.
Why not change tables during a query
Each of these would be a real bug with the world from lesson 4:
- Despawn. The swap-remove moves the last row into the hole, behind the loop, so that entity is not processed this frame. Meanwhile the slice the system cut before the despawn still reaches the old last row, a stale copy, and anything written to it is lost.
- Spawn. A table that is full grows, and its columns move to new memory. A system holding a
[]positionfrom before the spawn keeps writing to the old copy. Nothing crashes; the writes are just lost. - Add or remove. The entity moves to another table, which the same query may reach later. That entity is processed twice this frame.
The buffer
command: struct = { kind; entity; pending; signature; id; payload; }
Spawned entities have no handle until the buffer runs, so commands_spawn returns a pending number, and commands_set_spawned<T> writes a component into "the entity that pending spawn will create". T has to be in the spawn's signature, and applying the buffer stops the program with a message if it is not. Payloads are copies of the component value, kept as a []u8 in the buffer's frame arena next to the commands. commands_apply runs everything in order, then clears the arena. One memops_arena_clear frees the whole frame's commands.
Skipped commands
Frame 113 is the interesting one. A bullet destroys one drone, and its blast takes two burning neighbours below zero in the same loop. Each of them is destroyed in turn, and each blast reaches back to drones already recorded as destroyed:
bullet 8:9 hits drone 6:1, health -13.2
drone 6:1 is destroyed
drone 7:1 is caught in the blast, health -14.4
drone 4:2 is caught in the blast, health -4.8
drone 7:1 is destroyed
drone 6:1 is caught in the blast, health -54.0
...
The blast cannot know 6:1 is doomed: its despawn is only recorded, and it is still in its table. So it records an add<burning> against it. When the buffer is applied the despawn comes first, and kayan_move reports kayan_status.dead for the add, so the command is skipped. The summary counts three.
Try
- Call
kayan_despawndirectly inoutcome_systeminstead of recording it. The first difference is at frame 56: the program reports drone 3:1 destroyed, but the drone it removed was 1:1. Explain why, from the swap-remove and the slices the loop is holding. - Make
blast_systemskip neighbours whose health is already at or below zero. The skip count drops to zero. Is the buffer's check still needed? - Give the turret a
healthcomponent and let leaking drones damage it. When it reaches zero, despawn it through the buffer and end the game. - Print the buffer before applying it, as a log of every structural change the frame made.
- Inline
integrateback intomovement_system, build with gcc-O3 -fopt-info-vec, and find the loop it now versions for aliasing. - Run with
--sanitizeafter each change. A stale slice into a grown table is exactly what AddressSanitizer does not catch, because the old memory is still valid arena memory. Think about what would.
Imports the world from includes/kayan.rin.
1// 6. kayan2//3// Everything so far, in one small game. A turret at the centre shoots at wings4// of drones flying in from the edges. Bullets hurt drones, a badly hurt drone5// catches fire, and a drone at zero health is destroyed in a blast that hurts6// and ignites the drones around it. A drone that reaches the turret is a leak.7//8// The new idea is when structural changes happen. Spawning, despawning, adding9// and removing components all change tables, and systems are in the middle of10// walking tables when they decide to make those changes:11//12// * a despawn swap-removes: the last row moves into the hole, behind the13// loop, so that entity is not processed this frame, while a slice cut14// before the despawn still reaches the old last row, a stale copy whose15// writes are lost;16// * a spawn can grow a table, which moves every column of it to new memory,17// while the system still holds a slice of the old one;18// * an add or a remove moves an entity into another table, which the same19// query may walk later, so the entity is visited twice.20//21// So systems do not make structural changes. They record them in a command22// buffer, and the frame applies the buffer once every system has run. Plain23// writes to components are fine during a query: a hit lowers a drone's health24// in place. Only changes to which table an entity is in wait for the end of25// the frame.26//27// The buffer lives in a frame arena. Commands and their payloads are pushed28// into it during the frame and the whole arena is cleared after it is applied,29// so nothing in it is freed one at a time. A command against an entity that30// died earlier in the same buffer is skipped, which the handle's generation31// check makes cheap to detect.32 33cinclude "math.h"34import "../../includes/kayan.rin"35 36sqrtf: proc[external](x: f32) -> f32 = {}37 38vec2: struct = {39 x: f32;40 y: f32;41}42 43component: enum = {44 position,45 velocity,46 health,47 damage,48 lifetime,49 turret,50 burning,51}52 53position: struct = { at: vec2; }54velocity: struct = { per_second: vec2; }55health: struct = { points: f32; }56damage: struct = { amount: f32; }57lifetime: struct = { seconds: f32; }58turret: struct = { cooldown: f32; timer: f32; }59burning: struct = { per_second: f32; }60 61component_of: proc<position>() -> component = { return component.position; }62component_of: proc<velocity>() -> component = { return component.velocity; }63component_of: proc<health>() -> component = { return component.health; }64component_of: proc<damage>() -> component = { return component.damage; }65component_of: proc<lifetime>() -> component = { return component.lifetime; }66component_of: proc<turret>() -> component = { return component.turret; }67component_of: proc<burning>() -> component = { return component.burning; }68 69register_components: proc(w: *kayan_world) -> void = {70 kayan_register(w, kayan_id_of<position>(), position<>&);71 kayan_register(w, kayan_id_of<velocity>(), velocity<>&);72 kayan_register(w, kayan_id_of<health>(), health<>&);73 kayan_register(w, kayan_id_of<damage>(), damage<>&);74 kayan_register(w, kayan_id_of<lifetime>(), lifetime<>&);75 kayan_register(w, kayan_id_of<turret>(), turret<>&);76 kayan_register(w, kayan_id_of<burning>(), burning<>&);77}78 79// ---------------------------------------------------------------------------80// The command buffer81// ---------------------------------------------------------------------------82 83command_kind: enum = {84 spawn,85 set_spawned,86 add,87 remove,88 despawn,89}90 91// One recorded change. `pending` names an entity spawned by this same buffer,92// which has no handle until the buffer is applied.93command: struct = {94 kind: command_kind;95 entity: kayan_entity;96 pending: u32;97 signature: u64;98 id: u32;99 payload: []u8;100}101 102commands: struct = {103 frame: memops_arena;104 list: vec<command>;105 spawned: u32;106}107 108apply_result: struct = {109 applied: u32;110 skipped: u32;111}112 113commands_init: proc(c: *commands) -> void = {114 c* = {};115 memops_arena_initialize(c*.frame&);116}117 118commands_record: proc(c: *commands, cmd: command) -> void = {119 vec<command>append(c*.frame&, c*.list&, cmd);120}121 122// A copy of `value` in the frame arena, kept until the buffer is applied, as123// the bytes the component's column will receive.124commands_payload: proc<T>(c: *commands, value: T) -> []u8 = {125 copy: *T = cast(memops_arena_push(c*.frame&, sizeof(T), alignof(T)), *T);126 copy* = value;127 bytes: ^u8 = cast(copy, ^u8);128 return bytes[0..sizeof(T)];129}130 131commands_spawn: proc(c: *commands, signature: u64) -> u32 = {132 cmd: command = { .kind = command_kind.spawn, .signature = signature, .pending = c*.spawned };133 c*.spawned += 1;134 commands_record(c, cmd);135 return cmd.pending;136}137 138// T must be in the pending spawn's signature: the spawn decides the table.139commands_set_spawned: proc<T>(c: *commands, pending: u32, value: T) -> void = {140 commands_record(c, { .kind = command_kind.set_spawned, .pending = pending,141 .id = kayan_id_of<T>(), .payload = commands_payload<T>(c, value) });142}143 144commands_add: proc<T>(c: *commands, e: kayan_entity, value: T) -> void = {145 commands_record(c, { .kind = command_kind.add, .entity = e,146 .id = kayan_id_of<T>(), .payload = commands_payload<T>(c, value) });147}148 149commands_remove: proc<T>(c: *commands, e: kayan_entity) -> void = {150 commands_record(c, { .kind = command_kind.remove, .entity = e, .id = kayan_id_of<T>() });151}152 153commands_despawn: proc(c: *commands, e: kayan_entity) -> void = {154 commands_record(c, { .kind = command_kind.despawn, .entity = e });155}156 157// Copy a payload into e's component. Returns 0 if e has no such component.158write_component: proc(w: *kayan_world, e: kayan_entity, id: u32, payload: []const u8) -> b32 = {159 target: []u8 = kayan_get_bytes(w, e, id);160 if (target.length == 0) {161 return 0;162 }163 memcpy(target[0]&, payload[0]&, target.length);164 return 1;165}166 167// Every command, in the order it was recorded. Then the frame arena is cleared,168// which frees every command and payload at once.169commands_apply: proc(c: *commands, w: *kayan_world) -> apply_result = {170 result: apply_result = {};171 spawned_raw: ^kayan_entity = cast(memops_arena_push_zero(c*.frame&,172 sizeof(kayan_entity) * cast(c*.spawned, u64), alignof(kayan_entity)), ^kayan_entity);173 spawned: []kayan_entity = spawned_raw[0..cast(c*.spawned, u64)];174 list: []command = vec<command>items(c*.list&);175 for (i: u64 = 0; i < list.length; i += 1) {176 cmd: command = list[i];177 switch (cmd.kind) {178 case command_kind.spawn: {179 spawned[cmd.pending] = kayan_spawn(w, cmd.signature);180 result.applied += 1;181 }182 case command_kind.set_spawned: {183 if (!write_component(w, spawned[cmd.pending], cmd.id, cmd.payload)) {184 printfmt("set_spawned: spawn {} has no {} in its signature\n", cmd.pending,185 kayan_component_name(w, cmd.id));186 exit(1);187 }188 result.applied += 1;189 }190 case command_kind.add: {191 signature: u64 = kayan_signature(w, cmd.entity) | kayan_bit(cmd.id);192 if (kayan_move(w, cmd.entity, signature) == kayan_status.ok) {193 _ = write_component(w, cmd.entity, cmd.id, cmd.payload);194 result.applied += 1;195 } else {196 result.skipped += 1;197 }198 }199 case command_kind.remove: {200 signature: u64 = kayan_signature(w, cmd.entity) & ~kayan_bit(cmd.id);201 if (kayan_move(w, cmd.entity, signature) == kayan_status.ok) {202 result.applied += 1;203 } else {204 result.skipped += 1;205 }206 }207 case command_kind.despawn: {208 if (kayan_despawn(w, cmd.entity) == kayan_status.ok) {209 result.applied += 1;210 } else {211 result.skipped += 1;212 }213 }214 }215 }216 memops_arena_clear(c*.frame&);217 c*.list = {};218 c*.spawned = 0;219 return result;220}221 222// ---------------------------------------------------------------------------223// The game224// ---------------------------------------------------------------------------225 226stats: struct = {227 waves: u32;228 drones: u32;229 destroyed: u32;230 leaked: u32;231 shots: u32;232 hits: u32;233 misses: u32;234}235 236game: struct = {237 world: kayan_world;238 commands: commands;239 stats: stats;240 frame: i32;241 printed_frame: i32;242}243 244drone_mask: proc() -> u64 = {245 return kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<health>();246}247 248bullet_mask: proc() -> u64 = {249 return kayan_mask_of<position>() | kayan_mask_of<velocity>() | kayan_mask_of<damage>()250 | kayan_mask_of<lifetime>();251}252 253// Each frame's events are printed under one "frame N" line.254event: proc(g: *game) -> void = {255 if (g*.printed_frame != g*.frame) {256 printfmt("frame {}\n", g*.frame);257 g*.printed_frame = g*.frame;258 }259 printf(" ");260}261 262distance: proc(a: vec2, b: vec2) -> f32 = {263 dx: f32 = a.x - b.x;264 dy: f32 = a.y - b.y;265 return sqrtf(dx * dx + dy * dy);266}267 268toward: proc(from: vec2, to: vec2, speed: f32) -> vec2 = {269 d: f32 = distance(from, to);270 return { .x = (to.x - from.x) / d * speed, .y = (to.y - from.y) / d * speed };271}272 273// Drones come in wings of three, side by side, from each side in turn.274entry_points: [4]vec2 = {275 { .x = 20.0, .y = 0.0 }, { .x = 0.0, .y = -20.0 }, { .x = -20.0, .y = 0.0 }, { .x = 0.0, .y = 20.0 },276};277wave_every: i32 = 20;278waves: u32 = 5;279drone_speed: f32 = 2.0;280 281spawner_system: proc(g: *game) -> void = {282 if ((g*.frame - 1) % wave_every != 0 || g*.stats.waves == waves) {283 return;284 }285 centre: vec2 = entry_points[g*.stats.waves % 4];286 heading: vec2 = toward(centre, { .x = 0.0, .y = 0.0 }, 1.0);287 c: *commands = g*.commands&;288 for (k: i32 = -1; k <= 1; k += 1) {289 // Side by side: 2 units apart, across the direction of flight.290 side: f32 = 2.0 * cast(k, f32);291 at: vec2 = { .x = centre.x - heading.y * side, .y = centre.y + heading.x * side };292 drone: u32 = commands_spawn(c, drone_mask());293 commands_set_spawned<position>(c, drone, { .at = at });294 commands_set_spawned<velocity>(c, drone,295 { .per_second = toward(at, { .x = 0.0, .y = 0.0 }, drone_speed) });296 commands_set_spawned<health>(c, drone, { .points = 100.0 });297 g*.stats.drones += 1;298 }299 g*.stats.waves += 1;300 event(g);301 printf("a wing of three drones enters at (%.1f, %.1f)\n", cast(centre.x, f64), cast(centre.y, f64));302}303 304// The nearest drone to `from`, by walking every table a drone can be in.305nearest_drone: proc(w: *kayan_world, from: vec2, found: *vec2) -> b32 = {306 q: kayan_query = { .all = drone_mask() };307 best: f32 = 1000000.0;308 any: b32 = 0;309 for (ti: u64 = 0; ti < w*.tables.length; ti += 1) {310 t: *kayan_table = vec<kayan_table>at(w*.tables&, ti);311 if (!kayan_matches(t, q)) {312 continue;313 }314 ps: []position = kayan_column<position>(t);315 for (i: u64 = 0; i < ps.length; i += 1) {316 d: f32 = distance(from, ps[i].at);317 if (d < best) {318 best = d;319 found* = ps[i].at;320 any = 1;321 }322 }323 }324 return any;325}326 327turret_system: proc(g: *game, dt: f32) -> void = {328 w: *kayan_world = g*.world&;329 q: kayan_query = { .all = kayan_mask_of<position>() | kayan_mask_of<turret>() };330 for (ti: u64 = 0; ti < w*.tables.length; ti += 1) {331 t: *kayan_table = vec<kayan_table>at(w*.tables&, ti);332 if (!kayan_matches(t, q)) {333 continue;334 }335 ps: []position = kayan_column<position>(t);336 ts: []turret = kayan_column<turret>(t);337 for (i: u64 = 0; i < ts.length; i += 1) {338 // The timer stops at zero, so an idle turret is ready to fire but339 // does not save up shots.340 ts[i].timer -= dt;341 if (ts[i].timer < 0.0) {342 ts[i].timer = 0.0;343 }344 target: vec2 = {};345 if (ts[i].timer > 0.0 || !nearest_drone(w, ps[i].at, target&)) {346 continue;347 }348 ts[i].timer += ts[i].cooldown;349 c: *commands = g*.commands&;350 bullet: u32 = commands_spawn(c, bullet_mask());351 commands_set_spawned<position>(c, bullet, ps[i]);352 commands_set_spawned<velocity>(c, bullet, { .per_second = toward(ps[i].at, target, 12.0) });353 commands_set_spawned<damage>(c, bullet, { .amount = 30.0 });354 commands_set_spawned<lifetime>(c, bullet, { .seconds = 2.0 });355 g*.stats.shots += 1;356 }357 }358}359 360// A system is a query and a kernel. The query finds the tables; the kernel runs361// over one table's columns. noalias is a parameter qualifier, so the loop has362// to be in a proc of its own to get it.363integrate: proc(ps: noalias []position, vs: []const velocity, dt: f32) -> void = {364 for (i: u64 = 0; i < ps.length; i += 1) {365 ps[i].at.x += vs[i].per_second.x * dt;366 ps[i].at.y += vs[i].per_second.y * dt;367 }368}369 370movement_system: proc(w: *kayan_world, dt: f32) -> void = {371 q: kayan_query = { .all = kayan_mask_of<position>() | kayan_mask_of<velocity>() };372 for (ti: u64 = 0; ti < w*.tables.length; ti += 1) {373 t: *kayan_table = vec<kayan_table>at(w*.tables&, ti);374 if (kayan_matches(t, q)) {375 integrate(kayan_column<position>(t), kayan_column<velocity>(t), dt);376 }377 }378}379 380// Every bullet against every drone. A hit lowers health in place, which is not381// a structural change; removing the bullet and setting the drone on fire are.382// A bullet that expired this frame is skipped: lifetime_system has already383// recorded its despawn, and it is only still here because nothing structural384// happens until the end of the frame.385collision_system: proc(g: *game) -> void = {386 w: *kayan_world = g*.world&;387 bullets: kayan_query = { .all = bullet_mask() };388 drones: kayan_query = { .all = drone_mask() };389 for (bi: u64 = 0; bi < w*.tables.length; bi += 1) {390 bt: *kayan_table = vec<kayan_table>at(w*.tables&, bi);391 if (!kayan_matches(bt, bullets)) {392 continue;393 }394 bullet_at: []position = kayan_column<position>(bt);395 bullet_damage: []damage = kayan_column<damage>(bt);396 bullet_life: []lifetime = kayan_column<lifetime>(bt);397 bullet_ids: []kayan_entity = kayan_table_entities(bt);398 for (b: u64 = 0; b < bullet_at.length; b += 1) {399 if (bullet_life[b].seconds <= 0.0) {400 continue;401 }402 hit: b32 = 0;403 for (di: u64 = 0; di < w*.tables.length && !hit; di += 1) {404 drone_table: *kayan_table = vec<kayan_table>at(w*.tables&, di);405 if (!kayan_matches(drone_table, drones)) {406 continue;407 }408 drone_at: []position = kayan_column<position>(drone_table);409 drone_health: []health = kayan_column<health>(drone_table);410 drone_ids: []kayan_entity = kayan_table_entities(drone_table);411 for (d: u64 = 0; d < drone_at.length && !hit; d += 1) {412 if (distance(bullet_at[b].at, drone_at[d].at) > 1.0 || drone_health[d].points <= 0.0) {413 continue;414 }415 hit = 1;416 before: f32 = drone_health[d].points;417 drone_health[d].points -= bullet_damage[b].amount;418 g*.stats.hits += 1;419 commands_despawn(g*.commands&, bullet_ids[b]);420 event(g);421 printf("bullet %u:%u hits drone %u:%u, health %.1f\n", bullet_ids[b].index,422 bullet_ids[b].generation, drone_ids[d].index, drone_ids[d].generation,423 cast(drone_health[d].points, f64));424 if (before >= 50.0 && drone_health[d].points < 50.0425 && (drone_table*.signature & kayan_mask_of<burning>()) == 0) {426 commands_add<burning>(g*.commands&, drone_ids[d], { .per_second = 8.0 });427 event(g);428 printf("drone %u:%u catches fire\n", drone_ids[d].index, drone_ids[d].generation);429 }430 }431 }432 }433 }434}435 436burn: proc(hs: noalias []health, bs: []const burning, dt: f32) -> void = {437 for (i: u64 = 0; i < hs.length; i += 1) {438 hs[i].points -= bs[i].per_second * dt;439 }440}441 442burning_system: proc(w: *kayan_world, dt: f32) -> void = {443 q: kayan_query = { .all = kayan_mask_of<health>() | kayan_mask_of<burning>() };444 for (ti: u64 = 0; ti < w*.tables.length; ti += 1) {445 t: *kayan_table = vec<kayan_table>at(w*.tables&, ti);446 if (kayan_matches(t, q)) {447 burn(kayan_column<health>(t), kayan_column<burning>(t), dt);448 }449 }450}451 452// A destroyed drone's blast hurts every drone near it and sets it on fire.453// Lowering health is a plain write, so a blast can finish off a neighbour later454// in the same loop, and that neighbour's blast reaches back to the drone that455// started it. That drone's despawn is only recorded, so it is still in its456// table and gets an add<burning> too. By the time the buffer is applied it is457// gone, and commands_apply skips the add.458blast_system: proc(g: *game, from: kayan_entity, at: vec2) -> void = {459 w: *kayan_world = g*.world&;460 q: kayan_query = { .all = drone_mask() };461 for (ti: u64 = 0; ti < w*.tables.length; ti += 1) {462 t: *kayan_table = vec<kayan_table>at(w*.tables&, ti);463 if (!kayan_matches(t, q)) {464 continue;465 }466 ps: []position = kayan_column<position>(t);467 hs: []health = kayan_column<health>(t);468 ids: []kayan_entity = kayan_table_entities(t);469 for (i: u64 = 0; i < ps.length; i += 1) {470 if (ids[i].index != from.index && distance(ps[i].at, at) < 4.0) {471 hs[i].points -= 40.0;472 commands_add<burning>(g*.commands&, ids[i], { .per_second = 8.0 });473 event(g);474 printf("drone %u:%u is caught in the blast, health %.1f\n", ids[i].index,475 ids[i].generation, cast(hs[i].points, f64));476 }477 }478 }479}480 481// Drones at zero health are destroyed; drones that reach the turret leak.482outcome_system: proc(g: *game) -> void = {483 w: *kayan_world = g*.world&;484 q: kayan_query = { .all = drone_mask() };485 for (ti: u64 = 0; ti < w*.tables.length; ti += 1) {486 t: *kayan_table = vec<kayan_table>at(w*.tables&, ti);487 if (!kayan_matches(t, q)) {488 continue;489 }490 ps: []position = kayan_column<position>(t);491 hs: []health = kayan_column<health>(t);492 ids: []kayan_entity = kayan_table_entities(t);493 for (i: u64 = 0; i < hs.length; i += 1) {494 if (hs[i].points <= 0.0) {495 commands_despawn(g*.commands&, ids[i]);496 g*.stats.destroyed += 1;497 event(g);498 printf("drone %u:%u is destroyed\n", ids[i].index, ids[i].generation);499 blast_system(g, ids[i], ps[i].at);500 } else if (distance(ps[i].at, { .x = 0.0, .y = 0.0 }) < 1.5) {501 commands_despawn(g*.commands&, ids[i]);502 g*.stats.leaked += 1;503 event(g);504 printf("drone %u:%u reaches the turret with %.1f health\n", ids[i].index,505 ids[i].generation, cast(hs[i].points, f64));506 }507 }508 }509}510 511lifetime_system: proc(g: *game, dt: f32) -> void = {512 w: *kayan_world = g*.world&;513 q: kayan_query = { .all = kayan_mask_of<lifetime>() };514 for (ti: u64 = 0; ti < w*.tables.length; ti += 1) {515 t: *kayan_table = vec<kayan_table>at(w*.tables&, ti);516 if (!kayan_matches(t, q)) {517 continue;518 }519 ls: []lifetime = kayan_column<lifetime>(t);520 ids: []kayan_entity = kayan_table_entities(t);521 for (i: u64 = 0; i < ls.length; i += 1) {522 ls[i].seconds -= dt;523 if (ls[i].seconds <= 0.0) {524 commands_despawn(g*.commands&, ids[i]);525 g*.stats.misses += 1;526 }527 }528 }529}530 531main: proc() -> i32 = {532 arena: memops_arena = {};533 memops_arena_initialize(arena&);534 g: game = {};535 kayan_world_init(g.world&, arena&);536 register_components(g.world&);537 commands_init(g.commands&);538 539 tower: kayan_entity = kayan_spawn(g.world&, kayan_mask_of<position>() | kayan_mask_of<turret>());540 kayan_get<turret>(g.world&, tower)* = { .cooldown = 0.6, .timer = 0.6 };541 542 dt: f32 = 0.1;543 skipped: u32 = 0;544 frames: i32 = 140;545 for (frame: i32 = 1; frame <= frames; frame += 1) {546 g.frame = frame;547 spawner_system(g&);548 turret_system(g&, dt);549 movement_system(g.world&, dt);550 lifetime_system(g&, dt);551 collision_system(g&);552 burning_system(g.world&, dt);553 outcome_system(g&);554 result: apply_result = commands_apply(g.commands&, g.world&);555 skipped += result.skipped;556 }557 558 printfmt("after {} frames\n", frames);559 printfmt(" drones: {} entered, {} destroyed, {} leaked\n",560 g.stats.drones, g.stats.destroyed, g.stats.leaked);561 printfmt(" bullets: {} fired, {} hit, {} expired\n", g.stats.shots, g.stats.hits, g.stats.misses);562 printfmt(" commands skipped because their entity was already gone: {}\n", skipped);563 printfmt(" {} entities alive in {} tables\n", g.world.live, g.world.tables.length);564 for (ti: u64 = 0; ti < g.world.tables.length; ti += 1) {565 t: *kayan_table = vec<kayan_table>at(g.world.tables&, ti);566 printf(" table %d ", cast(ti, i32));567 kayan_print_signature(g.world&, t*.signature);568 printf(", rows in use: %d\n", cast(t*.count, i32));569 }570 return 0;571}What kayan prints. tools/check.py fails the lesson if a single byte differs.
frame 1
a wing of three drones enters at (20.0, 0.0)
frame 20
bullet 4:1 hits drone 2:1, health 70.0
frame 21
a wing of three drones enters at (0.0, -20.0)
frame 26
bullet 5:1 hits drone 2:1, health 40.0
drone 2:1 catches fire
frame 32
bullet 8:1 hits drone 2:1, health 6.0
frame 38
bullet 5:2 hits drone 2:1, health -28.8
drone 2:1 is destroyed
drone 1:1 is caught in the blast, health 60.0
drone 3:1 is caught in the blast, health 60.0
frame 41
a wing of three drones enters at (-20.0, 0.0)
frame 50
bullet 10:1 hits drone 1:1, health 21.2
frame 56
bullet 11:1 hits drone 1:1, health -13.6
drone 1:1 is destroyed
drone 3:1 is caught in the blast, health 5.6
frame 61
a wing of three drones enters at (0.0, 20.0)
frame 64
drone 3:1 is destroyed
frame 77
bullet 3:2 hits drone 6:1, health 70.0
frame 81
a wing of three drones enters at (20.0, 0.0)
frame 83
bullet 8:4 hits drone 6:1, health 40.0
drone 6:1 catches fire
frame 89
bullet 8:5 hits drone 7:1, health 70.0
frame 95
bullet 8:6 hits drone 7:1, health 40.0
drone 7:1 catches fire
frame 101
bullet 8:7 hits drone 4:2, health 70.0
frame 107
bullet 8:8 hits drone 4:2, health 40.0
drone 4:2 catches fire
frame 113
bullet 8:9 hits drone 6:1, health -13.2
drone 6:1 is destroyed
drone 7:1 is caught in the blast, health -14.4
drone 4:2 is caught in the blast, health -4.8
drone 7:1 is destroyed
drone 6:1 is caught in the blast, health -54.0
drone 4:2 is caught in the blast, health -44.8
drone 4:2 is destroyed
drone 6:1 is caught in the blast, health -94.0
drone 7:1 is caught in the blast, health -54.4
frame 122
bullet 4:3 hits drone 2:2, health 70.0
frame 128
bullet 4:4 hits drone 2:2, health 40.0
drone 2:2 catches fire
frame 134
bullet 4:5 hits drone 5:3, health 70.0
drone 5:3 reaches the turret with 70.0 health
drone 9:1 reaches the turret with 100.0 health
drone 2:2 reaches the turret with 35.2 health
after 140 frames
drones: 15 entered, 6 destroyed, 3 leaked
bullets: 20 fired, 16 hit, 3 expired
commands skipped because their entity was already gone: 3
8 entities alive in 4 tables
table 0 {position turret}, rows in use: 1
table 1 {position velocity health}, rows in use: 6
table 2 {position velocity damage lifetime}, rows in use: 1
table 3 {position velocity health burning}, rows in use: 0
Advanced · lesson 7
Layout follows access
src/7.access_patterns/access_patterns.rin
src/7.access_patterns/access_patterns.rin
Lesson 1 found the layout SIMD wants: one array per float. This lesson asks what that layout costs everywhere else. 1,048,576 bodies are stored four ways, and two kinds of loop run over each, and the program checks that all four end with the same positions.
Two kinds of loop
Stream: every body, in order, the same math. Integration is the example:
for (i: u64 = 0; i < px.length; i += 1) {
px[i] += vx[i] * dt;
...
}
The next body is always right after the last one, so the hardware prefetcher has it ready, and SIMD can take four or eight at once if they sit side by side.
Lookup: bodies picked by index, two at a time, in no particular order. Contacts are the example, the way collision and constraints reach bodies:
a: u64 = cast(cs[c].a, u64);
b: u64 = cast(cs[c].b, u64);
dx: f32 = px[b] - px[a];
...
Each body is somewhere else in memory. What matters is how many cache lines one body's data sits in, because each one is a separate trip to memory. SIMD does not help: there are no four neighbours to load at once.
Four layouts
aos [ x y z vx vy vz m r ][ x y z vx vy vz m r ] ... 32 bytes a body
columns positions [ x y z ][ x y z ][ x y z ] ...
velocities [ vx vy vz ][ vx vy vz ] ... 24 bytes a body
split px [ x x x x ... ] py [ y y y y ... ] pz ... vx ... vy ... vz ...
blocks [ x0..x7 | y0..y7 | z0..z7 | vx0..vx7 | vy0..vy7 | vz0..vz7 ][ x8..x15 ...
- aos keeps everything about a body together, two fields neither loop uses included.
- columns is one array per component, each component a small struct. This is what an ECS table stores, and what lessons 2 to 6 use.
- split is one array per float, lesson 1's
move_split. - blocks is eight bodies at a time, each coordinate a run of eight floats. It is split cut into pieces of eight, with the pieces of one body kept near each other. The usual name is AoSoA, an array of structs of arrays.
The program prints how many 64-byte cache lines one body's position sits in:
cache lines one lookup touches: aos 1.000, columns 1.125, split 3.000, blocks 2.000
A 12-byte vec3 crosses a line boundary 2 times in 16, hence 1.125. In split, x, y and z are in three different arrays. In a block, x and y share a line and z is in the next.
Measured
Release, clang -O2, best of three runs, on the 2.8 GHz Xeon these lessons were checked on: 4 MB of L2, 33 MB of L3. The body count was changed in the generated C for the first and last columns. At 65,536 bodies each layout fits in L2; at the lesson's 1,048,576 each is about the size of L3; at 8,388,608 each is 192 to 256 MB and comes from RAM.
Stream, ns per body:
| layout | 65,536 | 1,048,576 | 8,388,608 |
|---|---|---|---|
| aos | 1.67 | 4.20 | 4.09 |
| columns | 1.68 | 2.08 | 2.88 |
| split | 0.93 | 1.30 | 1.66 |
| blocks | 0.70 | 1.62 | 2.79 |
Lookup, ns per contact:
| layout | 65,536 | 1,048,576 | 8,388,608 |
|---|---|---|---|
| aos | 5.03 | 26.27 | 50.35 |
| columns | 3.42 | 11.94 | 40.35 |
| split | 6.54 | 28.22 | 95.01 |
| blocks | 5.98 | 25.70 | 61.20 |
clang with -mavx2 -mfma and gcc -O3 agree on the main points: split streams fastest from RAM, columns look up fastest at every size, and split looks up slowest from RAM. Where two numbers are within about 20 percent, the order changes from build to build; gcc -O3, for one, streamed columns fastest at a million bodies. The machine is shared; expect about 20 percent noise.
What it says:
- Split wins the stream and loses the lookup. From RAM it streams 1.7 times faster than columns, and looks up 2.4 times slower: three lines a body instead of one.
- Columns win the lookup everywhere and come second or third in the stream. AoS also has one line a body, but its bodies are 32 bytes, so the same lookups spread over more memory. At a million bodies, columns' positions are 12 MB and sit comfortably in L3; AoS spreads them over 32 MB, about all of it.
- Blocks are the compromise. In L2 they streamed fastest of all, and a little behind split at a million bodies. They look up better than split at every size. From RAM their stream fell back to about columns' speed on this machine, which I have not explained.
- Out of cache, a lookup costs ten to fifty times a streamed body. From RAM, 40 to 95 ns a contact against 2 to 4 ns a body. Turning lookups into something closer to a stream, by sorting contacts or walking space in order, usually saves more than any layout choice.
What the compiler makes of it
clang-cl /O2, Windows x64, from the object file:
| loop | bodies a pass | instructions a pass | shuffles |
|---|---|---|---|
integrate_aos | 1 | 16 | 1 |
integrate_columns | 4 | 77 | 30 |
integrate_split | 4 | 20 | 0 |
integrate_blocks | 8 | 37 | 0 |
integrate_split, /arch:AVX2 | 8 | 14 | 0 |
integrate_blocks, /arch:AVX2 | 8 | 14 | 0 |
contacts_columns | 1 contact | 25 | 0 |
contacts_split | 1 contact | 32 | 0 |
Blocks vectorize as cleanly as split: the inner loop over eight lanes is a fixed-size loop over consecutive floats, which compilers turn into whole registers. The contact loops are one contact a pass in every layout.
In between
Engines that have both kinds of loop do one of these:
- Blocks, as above. One layout, decent at both.
- Gather, work, scatter. Storage stays in columns or AoS, good for lookups. The hot loop copies a batch of bodies into a wide layout, x's in one register and y's in another, does its math, and copies them back. The copy pays off when the same batch is worked on many times, as a constraint solver does over several iterations. Box2D v3 solves contacts this way, four or eight at a time.
- Two layouts for two jobs. A particle system that only ever streams stores split. The bodies that collision looks up stay in columns.
How an ECS decides
An ECS does not decide where each float goes. The component is the unit it stores together, and the programmer decides what a component is.
- Tables. kayan, Unity's entities, Bevy and flecs keep one table per archetype, with one column per component. That is this lesson's columns layout: an array per component, and each component's fields side by side inside it. Unity cuts its tables into 16 KB chunks, each holding every component array for as many entities as fit, which is close to blocks at the level of components.
- Sparse sets. EnTT stores each component in its own dense array with a lookup table from entity to index, and can keep several components in the same order ("groups") so a system over them streams. Bevy lets each component choose: table storage, or a sparse set for components added and removed often.
- Systems are streams. A system is a query and a kernel, as in lesson 6: the ECS finds the tables that match, and hands the kernel each table's columns. That stream over columns is the fast path every ECS is built for. Systems that say which columns they read and which they write are what lets a scheduler run them in parallel.
- Lookups go through the handle. Reaching one entity is handle, slot, table and row, then the column:
kayan_getin lesson 4. That is this lesson's lookup loop with an extra hop in front. ECS code keeps lookups for the places that need them, such as relationships between entities and collision, and finds the rest by query.
So the decisions that are yours:
- Keep components small, to what one stream system reads. A component a system reads half of is AoS's wasted bytes again.
- Leave components as small structs,
position { x, y, z }, unless a profiler shows a stream over them is the bottleneck. They are the best layout for lookups and good enough for most streams. - For a hot stream over huge counts, particles above all, store split or blocks, or gather into them for the hot loop.
- Make lookups rarer and less random before tuning their layout.
Try
- Sort
csbyabefore the timing loops and run--timeagain. Which lookup numbers change, and by how much? - Set
body_countandcontact_countto 8388608 and run--timein Release. Compare with the tables above. - Make the contacts push velocities too,
vx,vyandvzof both bodies. How many lines does a lookup touch in each layout now? - Add
/arch:AVX2tobuild_c_flagsand compare the stream column. - Write
contacts_gather: for every eight contacts, copy the sixteen bodies' positions into[8]f32arrays, do the arithmetic over the arrays, and copy back. Contacts that share a body in the same eight break it; find the case and decide what to do about it. - Make
body64 bytes with fields neither loop uses, and watch aos in both tables.
1// 7. Layout follows access2//3// Lesson 1 found the layout SIMD wants: one array per float. This lesson asks4// what that layout costs everywhere else. The same bodies are stored four ways,5// and two kinds of loop run over each:6//7// stream every body, in order, the same math: integration.8// lookup bodies picked by index, two at a time, in no order: contacts,9// the way collision and constraints reach them.10//11// The four layouts:12//13// aos []body, everything about a body together14// columns []vec3 per field, as an ECS table stores components15// split []f32 per coordinate, as lesson 1's move_split16// blocks []body_block, eight bodies per block, one [8]f32 per coordinate17//18// A stream wants consecutive floats, so SIMD can load four or eight at once.19// A lookup wants one body's fields close together, so one cache line brings20// them all. No layout is best at both; the program prints how many cache lines21// a lookup touches in each, and with `--time` how long each loop takes.22 23cinclude "time.h"24import "std/print.rin"25import "std/memops.rin"26 27clock: proc[external]() -> long = {}28CLOCKS_PER_SEC: const long;29 30body_count: u64 = 1048576;31contact_count: u64 = 1048576;32dt: static f32 = 1.0 / 60.0;33gravity: static f32 = -9.8;34stiffness: static f32 = 0.01;35 36vec3: struct = {37 x: f32;38 y: f32;39 z: f32;40}41 42// aos: 32 bytes, the two fields both loops use and two they do not.43body: struct = {44 position: vec3;45 velocity: vec3;46 inverse_mass: f32;47 radius: f32;48}49 50// blocks: eight bodies, each coordinate a run of eight floats. 192 bytes.51body_block: struct = {52 px: [8]f32;53 py: [8]f32;54 pz: [8]f32;55 vx: [8]f32;56 vy: [8]f32;57 vz: [8]f32;58}59 60// Two bodies a contact pushes apart.61contact: struct = {62 a: u32;63 b: u32;64}65 66columns: struct = {67 positions: []vec3;68 velocities: []vec3;69}70 71split: struct = {72 px: []f32;73 py: []f32;74 pz: []f32;75 vx: []f32;76 vy: []f32;77 vz: []f32;78}79 80// Cache-line aligned, so the line counts below are the same on every run.81push_slice: proc<T>(arena: *memops_arena, count: u64) -> []T = {82 raw: ^T = cast(memops_arena_push_zero(arena, sizeof(T) * count, 64), ^T);83 return raw[0..count];84}85 86random_state: u64 = 12345;87 88random01: proc() -> f32 = {89 random_state = random_state * 6364136223846793005'u64 + 1442695040888963407'u64;90 return cast(random_state shr 40, f32) / 16777216.0;91}92 93random_index: proc(n: u64) -> u32 = {94 random_state = random_state * 6364136223846793005'u64 + 1442695040888963407'u64;95 return cast((random_state shr 33) % n, u32);96}97 98// ---------------------------------------------------------------- stream99// Every body, in order, the same math. noalias on what each kernel writes.100 101integrate_aos: proc(bodies: noalias []body, dt: f32) -> void = {102 for (i: u64 = 0; i < bodies.length; i += 1) {103 b: *body = bodies[i]&;104 b*.velocity.y += gravity * dt;105 b*.position.x += b*.velocity.x * dt;106 b*.position.y += b*.velocity.y * dt;107 b*.position.z += b*.velocity.z * dt;108 }109}110 111integrate_columns: proc(positions: noalias []vec3, velocities: noalias []vec3, dt: f32) -> void = {112 for (i: u64 = 0; i < positions.length; i += 1) {113 velocities[i].y += gravity * dt;114 positions[i].x += velocities[i].x * dt;115 positions[i].y += velocities[i].y * dt;116 positions[i].z += velocities[i].z * dt;117 }118}119 120integrate_split: proc(px: noalias []f32, py: noalias []f32, pz: noalias []f32,121 vx: []const f32, vy: noalias []f32, vz: []const f32, dt: f32) -> void = {122 for (i: u64 = 0; i < px.length; i += 1) {123 vy[i] += gravity * dt;124 px[i] += vx[i] * dt;125 py[i] += vy[i] * dt;126 pz[i] += vz[i] * dt;127 }128}129 130// The inner loop is eight floats per coordinate: one AVX register, or two SSE.131integrate_blocks: proc(blocks: noalias []body_block, dt: f32) -> void = {132 for (b: u64 = 0; b < blocks.length; b += 1) {133 k: *body_block = blocks[b]&;134 for (l: u64 = 0; l < 8; l += 1) {135 k*.vy[l] += gravity * dt;136 k*.px[l] += k*.vx[l] * dt;137 k*.py[l] += k*.vy[l] * dt;138 k*.pz[l] += k*.vz[l] * dt;139 }140 }141}142 143// ---------------------------------------------------------------- lookup144// Each contact moves its two bodies a little toward each other. The bodies are145// anywhere in the arrays, so every contact is two jumps through memory. The146// arithmetic is the same in every layout, in the same order, so the results147// match to the bit.148 149contacts_aos: proc(bodies: noalias []body, cs: []const contact) -> void = {150 for (c: u64 = 0; c < cs.length; c += 1) {151 p: *vec3 = bodies[cast(cs[c].a, u64)].position&;152 q: *vec3 = bodies[cast(cs[c].b, u64)].position&;153 dx: f32 = q*.x - p*.x;154 dy: f32 = q*.y - p*.y;155 dz: f32 = q*.z - p*.z;156 p*.x += dx * stiffness;157 p*.y += dy * stiffness;158 p*.z += dz * stiffness;159 q*.x -= dx * stiffness;160 q*.y -= dy * stiffness;161 q*.z -= dz * stiffness;162 }163}164 165contacts_columns: proc(positions: noalias []vec3, cs: []const contact) -> void = {166 for (c: u64 = 0; c < cs.length; c += 1) {167 p: *vec3 = positions[cast(cs[c].a, u64)]&;168 q: *vec3 = positions[cast(cs[c].b, u64)]&;169 dx: f32 = q*.x - p*.x;170 dy: f32 = q*.y - p*.y;171 dz: f32 = q*.z - p*.z;172 p*.x += dx * stiffness;173 p*.y += dy * stiffness;174 p*.z += dz * stiffness;175 q*.x -= dx * stiffness;176 q*.y -= dy * stiffness;177 q*.z -= dz * stiffness;178 }179}180 181contacts_split: proc(px: noalias []f32, py: noalias []f32, pz: noalias []f32, cs: []const contact) -> void = {182 for (c: u64 = 0; c < cs.length; c += 1) {183 a: u64 = cast(cs[c].a, u64);184 b: u64 = cast(cs[c].b, u64);185 dx: f32 = px[b] - px[a];186 dy: f32 = py[b] - py[a];187 dz: f32 = pz[b] - pz[a];188 px[a] += dx * stiffness;189 py[a] += dy * stiffness;190 pz[a] += dz * stiffness;191 px[b] -= dx * stiffness;192 py[b] -= dy * stiffness;193 pz[b] -= dz * stiffness;194 }195}196 197contacts_blocks: proc(blocks: noalias []body_block, cs: []const contact) -> void = {198 for (c: u64 = 0; c < cs.length; c += 1) {199 a: u64 = cast(cs[c].a, u64);200 b: u64 = cast(cs[c].b, u64);201 p: *body_block = blocks[a / 8]&;202 q: *body_block = blocks[b / 8]&;203 la: u64 = a % 8;204 lb: u64 = b % 8;205 dx: f32 = q*.px[lb] - p*.px[la];206 dy: f32 = q*.py[lb] - p*.py[la];207 dz: f32 = q*.pz[lb] - p*.pz[la];208 p*.px[la] += dx * stiffness;209 p*.py[la] += dy * stiffness;210 p*.pz[la] += dz * stiffness;211 q*.px[lb] -= dx * stiffness;212 q*.py[lb] -= dy * stiffness;213 q*.pz[lb] -= dz * stiffness;214 }215}216 217// ---------------------------------------------------------------- measuring218 219// How many 64-byte lines hold one body's x, y and z, given each one's byte220// offset from a line-aligned start. Each float is 4 bytes at a multiple of 4,221// so none of them crosses a line on its own.222lines_for: proc(x: u64, y: u64, z: u64) -> u64 = {223 lx: u64 = x / 64;224 ly: u64 = y / 64;225 lz: u64 = z / 64;226 n: u64 = 1;227 if (ly != lx) { n += 1; }228 if (lz != lx && lz != ly) { n += 1; }229 return n;230}231 232// The average over the first 4096 bodies, which covers every repeating pattern.233lines_per_lookup: proc(layout: u32) -> f64 = {234 total: u64 = 0;235 for (i: u64 = 0; i < 4096; i += 1) {236 if (layout == 0) {237 base: u64 = i * sizeof(body);238 total += lines_for(base, base + 4, base + 8);239 } else if (layout == 1) {240 base: u64 = i * sizeof(vec3);241 total += lines_for(base, base + 4, base + 8);242 } else if (layout == 2) {243 // three separate line-aligned arrays: never the same line244 total += 3;245 } else {246 base: u64 = (i / 8) * sizeof(body_block) + (i % 8) * 4;247 total += lines_for(base, base + 32, base + 64);248 }249 }250 return cast(total, f64) / 4096.0;251}252 253checksum_aos: proc(bodies: []const body) -> f64 = {254 sum: f64 = 0.0;255 for (i: u64 = 0; i < bodies.length; i += 1) {256 sum += cast(bodies[i].position.x + bodies[i].position.y + bodies[i].position.z, f64);257 }258 return sum;259}260 261checksum_columns: proc(positions: []const vec3) -> f64 = {262 sum: f64 = 0.0;263 for (i: u64 = 0; i < positions.length; i += 1) {264 sum += cast(positions[i].x + positions[i].y + positions[i].z, f64);265 }266 return sum;267}268 269checksum_split: proc(xs: []const f32, ys: []const f32, zs: []const f32) -> f64 = {270 sum: f64 = 0.0;271 for (i: u64 = 0; i < xs.length; i += 1) {272 sum += cast(xs[i] + ys[i] + zs[i], f64);273 }274 return sum;275}276 277checksum_blocks: proc(blocks: []const body_block) -> f64 = {278 sum: f64 = 0.0;279 for (b: u64 = 0; b < blocks.length; b += 1) {280 for (l: u64 = 0; l < 8; l += 1) {281 sum += cast(blocks[b].px[l] + blocks[b].py[l] + blocks[b].pz[l], f64);282 }283 }284 return sum;285}286 287seconds: proc() -> f64 = {288 return cast(clock(), f64) / cast(CLOCKS_PER_SEC, f64);289}290 291main: proc(argc: i32, argv: ^^c8) -> i32 = {292 timing: b32 = argc > 1 && strcmp(argv[1], "--time") == 0;293 steps: u64 = timing ? 20 : 2;294 295 arena: memops_arena = {};296 memops_arena_initialize(arena&);297 298 aos: []body = push_slice<body>(arena&, body_count);299 col: columns = {};300 col.positions = push_slice<vec3>(arena&, body_count);301 col.velocities = push_slice<vec3>(arena&, body_count);302 spl: split = {};303 spl.px = push_slice<f32>(arena&, body_count);304 spl.py = push_slice<f32>(arena&, body_count);305 spl.pz = push_slice<f32>(arena&, body_count);306 spl.vx = push_slice<f32>(arena&, body_count);307 spl.vy = push_slice<f32>(arena&, body_count);308 spl.vz = push_slice<f32>(arena&, body_count);309 blocks: []body_block = push_slice<body_block>(arena&, body_count / 8);310 311 for (i: u64 = 0; i < body_count; i += 1) {312 p: vec3 = { .x = random01() * 100.0, .y = random01() * 100.0, .z = random01() * 100.0 };313 v: vec3 = { .x = random01() - 0.5, .y = random01() * 4.0, .z = random01() - 0.5 };314 aos[i] = { .position = p, .velocity = v, .inverse_mass = 1.0, .radius = 0.5 };315 col.positions[i] = p;316 col.velocities[i] = v;317 spl.px[i] = p.x; spl.py[i] = p.y; spl.pz[i] = p.z;318 spl.vx[i] = v.x; spl.vy[i] = v.y; spl.vz[i] = v.z;319 k: *body_block = blocks[i / 8]&;320 l: u64 = i % 8;321 k*.px[l] = p.x; k*.py[l] = p.y; k*.pz[l] = p.z;322 k*.vx[l] = v.x; k*.vy[l] = v.y; k*.vz[l] = v.z;323 }324 cs: []contact = push_slice<contact>(arena&, contact_count);325 for (c: u64 = 0; c < contact_count; c += 1) {326 cs[c] = { .a = random_index(body_count), .b = random_index(body_count) };327 }328 329 // stream, then lookup, for each layout in turn330 t: [9]f64 = {};331 t[0] = seconds();332 for (s: u64 = 0; s < steps; s += 1) { integrate_aos(aos, dt); }333 t[1] = seconds();334 for (s: u64 = 0; s < steps; s += 1) { contacts_aos(aos, cs); }335 t[2] = seconds();336 for (s: u64 = 0; s < steps; s += 1) { integrate_columns(col.positions, col.velocities, dt); }337 t[3] = seconds();338 for (s: u64 = 0; s < steps; s += 1) { contacts_columns(col.positions, cs); }339 t[4] = seconds();340 for (s: u64 = 0; s < steps; s += 1) {341 integrate_split(spl.px, spl.py, spl.pz, spl.vx, spl.vy, spl.vz, dt);342 }343 t[5] = seconds();344 for (s: u64 = 0; s < steps; s += 1) { contacts_split(spl.px, spl.py, spl.pz, cs); }345 t[6] = seconds();346 for (s: u64 = 0; s < steps; s += 1) { integrate_blocks(blocks, dt); }347 t[7] = seconds();348 for (s: u64 = 0; s < steps; s += 1) { contacts_blocks(blocks, cs); }349 t[8] = seconds();350 351 printfmt("{} bodies, {} contacts, {} steps\n", body_count, contact_count, steps);352 printfmt("bytes per body: aos {}, columns {}, split {}, blocks {}\n",353 sizeof(body), sizeof(vec3) * 2, sizeof(f32) * 6, sizeof(body_block) / 8);354 printf("cache lines one lookup touches: aos %.3f, columns %.3f, split %.3f, blocks %.3f\n",355 lines_per_lookup(0), lines_per_lookup(1), lines_per_lookup(2), lines_per_lookup(3));356 357 sums: [4]f64 = {358 checksum_aos(aos),359 checksum_columns(col.positions),360 checksum_split(spl.px, spl.py, spl.pz),361 checksum_blocks(blocks),362 };363 printf("checksum aos %.3f\n", sums[0]);364 printf("checksum columns %.3f\n", sums[1]);365 printf("checksum split %.3f\n", sums[2]);366 printf("checksum blocks %.3f\n", sums[3]);367 same: b32 = sums[0] == sums[1] && sums[1] == sums[2] && sums[2] == sums[3];368 printfmt("same result: {}\n", same ? "yes" : "no");369 370 if (timing) {371 n_stream: f64 = cast(body_count * steps, f64);372 n_lookup: f64 = cast(contact_count * steps, f64);373 printf("ns per body stream ns per contact lookup\n");374 printf(" aos %6.2f %6.2f\n", (t[1] - t[0]) * 1.0e9 / n_stream, (t[2] - t[1]) * 1.0e9 / n_lookup);375 printf(" columns %6.2f %6.2f\n", (t[3] - t[2]) * 1.0e9 / n_stream, (t[4] - t[3]) * 1.0e9 / n_lookup);376 printf(" split %6.2f %6.2f\n", (t[5] - t[4]) * 1.0e9 / n_stream, (t[6] - t[5]) * 1.0e9 / n_lookup);377 printf(" blocks %6.2f %6.2f\n", (t[7] - t[6]) * 1.0e9 / n_stream, (t[8] - t[7]) * 1.0e9 / n_lookup);378 }379 return same ? 0 : 1;380}What access_patterns prints. tools/check.py fails the lesson if a single byte differs.
1048576 bodies, 1048576 contacts, 2 steps
bytes per body: aos 32, columns 24, split 24, blocks 24
cache lines one lookup touches: aos 1.000, columns 1.125, split 3.000, blocks 2.000
checksum aos 157337021.562
checksum columns 157337021.562
checksum split 157337021.562
checksum blocks 157337021.562
same result: yes
Advanced · library · built in lesson 4
kayan.rin
includes/kayan.rin · 417 lines
1// kayan: archetype tables, entity handles, and moving entities between tables.2//3// This is the world lesson 4 builds, kept as a library so lessons 5 and 6 can4// use it as it is. Read it with lesson 4: the lesson explains the why, this5// file is the how.6//7// What a program using it provides:8//9// * its components, as structs, and an enum with one member per component;10// * `component_of: proc<T>() -> <its enum>` for each component type, which11// the generic procs here call to find a type's id;12// * a call to `kayan_register` for each component, with its reflect record.13//14// Ids are u32 here, so the library does not need to know the program's enum.15// Up to 64 components, because a signature is a u64.16//17// Not here, on purpose: threads, change tracking, chunked storage, relations.18// Each is worth a lesson of its own and none is needed to see the idea.19 20import "std/print.rin"21import "std/memops.rin"22import "std/vec.rin"23import "std/hashmap.rin"24 25// A handle. `index` picks a slot in the world's slot list and `generation` says26// which life of that slot the handle was made for. Despawning bumps the slot's27// generation, so every handle to the dead entity stops matching, even after the28// index is reused. Generations start at 1, so a zeroed handle, `{}`, never29// names a live entity and can stand for "none".30kayan_entity: struct = {31 index: u32;32 generation: u32;33}34 35// Where an entity lives now: which table, which row. Rows move (swap-remove,36// lesson 2), so this is updated whenever one does. Handles never move. A free37// slot already holds its next generation, which no handle has yet, so the38// generation check alone says whether a handle is alive.39kayan_slot: struct = {40 generation: u32;41 table: u32;42 row: u32;43}44 45// One archetype. `entities[row]` is the handle of the entity in that row, which46// is how a swap-remove finds the slot to fix. `columns[id]` is null unless the47// signature has bit `id`.48kayan_table: struct = {49 signature: u64;50 count: u64;51 capacity: u64;52 entities: ^kayan_entity;53 columns: [64]^u8;54}55 56kayan_query: struct = {57 all: u64;58 none: u64;59}60 61// What a structural change reports. `dead` means the handle no longer names a62// live entity, so nothing was changed.63kayan_status: enum[must_use] = {64 ok,65 dead,66}67 68kayan_world: struct = {69 arena: *memops_arena;70 infos: [64]*const reflect;71 component_count: u32;72 // Tables are referred to by index. A pointer from kayan_table_at is good73 // until the next table is created, which can move the list.74 tables: vec<kayan_table>;75 by_signature: hashmap<u64, u32>;76 slots: vec<kayan_slot>;77 free: vec<u32>;78 live: u64;79}80 81// ---------------------------------------------------------------------------82// Components83// ---------------------------------------------------------------------------84 85kayan_bit: proc(id: u32) -> u64 = {86 return 1'u64 shl cast(id, u64);87}88 89kayan_id_of: proc<T>() -> u32 = {90 return cast(component_of<T>(), u32);91}92 93kayan_mask_of: proc<T>() -> u64 = {94 return kayan_bit(kayan_id_of<T>());95}96 97kayan_world_init: proc(w: *kayan_world, arena: *memops_arena) -> void = {98 w* = {};99 w*.arena = arena;100}101 102kayan_register: proc(w: *kayan_world, id: u32, info: *const reflect) -> void = {103 if (id >= 64) {104 printfmt("kayan: component id {} is past the 64 a signature can hold\n", id);105 exit(1);106 }107 w*.infos[id] = info;108 if (id + 1 > w*.component_count) {109 w*.component_count = id + 1;110 }111}112 113kayan_component_name: proc(w: *kayan_world, id: u32) -> ^const char = {114 return w*.infos[id]*.name;115}116 117// ---------------------------------------------------------------------------118// Tables119// ---------------------------------------------------------------------------120 121kayan_table_at: proc(w: *kayan_world, index: u32) -> *kayan_table = {122 return vec<kayan_table>at(w*.tables&, cast(index, u64));123}124 125// Double the table's columns. The old columns stay in the arena unused: an126// arena does not free single allocations, and these lessons accept the waste.127kayan_table_grow: proc(w: *kayan_world, t: *kayan_table) -> void = {128 capacity: u64 = t*.capacity == 0 ? 16 : t*.capacity * 2;129 entities: ^kayan_entity = cast(memops_arena_push_zero(w*.arena,130 sizeof(kayan_entity) * capacity, alignof(kayan_entity)), ^kayan_entity);131 if (t*.count > 0) {132 memcpy(entities, t*.entities, sizeof(kayan_entity) * t*.count);133 }134 t*.entities = entities;135 for (id: u32 = 0; id < w*.component_count; id += 1) {136 if ((t*.signature & kayan_bit(id)) != 0) {137 info: *const reflect = w*.infos[id];138 column: ^u8 = cast(memops_arena_push_zero(w*.arena, info*.size * capacity,139 info*.align), ^u8);140 if (t*.count > 0) {141 memcpy(column, t*.columns[id], info*.size * t*.count);142 }143 t*.columns[id] = column;144 }145 }146 t*.capacity = capacity;147}148 149// The table for a signature, made the first time anything needs it.150kayan_table_for: proc(w: *kayan_world, signature: u64) -> u32 = {151 found: *u32 = hashmap<u64, u32>get(w*.by_signature&, signature);152 if (found != null) {153 return found*;154 }155 for (id: u32 = 0; id < 64; id += 1) {156 if ((signature & kayan_bit(id)) != 0 && (id >= w*.component_count || w*.infos[id] == null)) {157 printfmt("kayan: signature {} uses component {}, which was never registered\n", signature, id);158 exit(1);159 }160 }161 t: kayan_table = {};162 t.signature = signature;163 index: u32 = cast(w*.tables.length, u32);164 vec<kayan_table>append(w*.arena, w*.tables&, t);165 hashmap<u64, u32>put(w*.arena, w*.by_signature&, signature, index);166 return index;167}168 169// A new row with zeroed components, owned by `e`.170kayan_table_add_row: proc(w: *kayan_world, t: *kayan_table, e: kayan_entity) -> u64 = {171 if (t*.count == t*.capacity) {172 kayan_table_grow(w, t);173 }174 row: u64 = t*.count;175 t*.entities[row] = e;176 for (id: u32 = 0; id < w*.component_count; id += 1) {177 if (t*.columns[id] != null) {178 size: u64 = w*.infos[id]*.size;179 memset(t*.columns[id][row * size]&, 0, size);180 }181 }182 t*.count += 1;183 return row;184}185 186// Swap-remove: the last row moves into `row`, and the slot of the entity that187// moved is told its new row. This is the line lesson 2 could not write.188kayan_table_remove_row: proc(w: *kayan_world, t: *kayan_table, row: u64) -> void = {189 last: u64 = t*.count - 1;190 if (row != last) {191 for (id: u32 = 0; id < w*.component_count; id += 1) {192 if (t*.columns[id] != null) {193 size: u64 = w*.infos[id]*.size;194 memcpy(t*.columns[id][row * size]&, t*.columns[id][last * size]&, size);195 }196 }197 moved: kayan_entity = t*.entities[last];198 t*.entities[row] = moved;199 vec<kayan_slot>at(w*.slots&, cast(moved.index, u64))*.row = cast(row, u32);200 }201 t*.count = last;202}203 204kayan_matches: proc(t: *kayan_table, q: kayan_query) -> b32 = {205 return (t*.signature & q.all) == q.all && (t*.signature & q.none) == 0;206}207 208// T's column, cut to the rows in use. Asking for a column the table does not209// have is a bug in the caller, so it stops the program.210kayan_column: proc<T>(t: *kayan_table) -> []T = {211 id: u32 = kayan_id_of<T>();212 if (t*.columns[id] == null) {213 printfmt("kayan: table {} has no column for component {}\n", t*.signature, id);214 exit(1);215 }216 raw: ^T = cast(t*.columns[id], ^T);217 return raw[0..t*.count];218}219 220kayan_table_entities: proc(t: *kayan_table) -> []kayan_entity = {221 return t*.entities[0..t*.count];222}223 224// ---------------------------------------------------------------------------225// Entities226// ---------------------------------------------------------------------------227 228kayan_alive: proc(w: *kayan_world, e: kayan_entity) -> b32 = {229 if (cast(e.index, u64) >= w*.slots.length) {230 return 0;231 }232 return vec<kayan_slot>at(w*.slots&, cast(e.index, u64))*.generation == e.generation;233}234 235// A new entity with the components in `signature`, all zeroed. A freed index is236// reused before a new one is made, with the generation its despawn left behind.237kayan_spawn: proc(w: *kayan_world, signature: u64) -> kayan_entity = {238 e: kayan_entity = {};239 if (!vec<u32>is_empty(w*.free&)) {240 e.index = vec<u32>pop(w*.free&);241 } else {242 e.index = cast(w*.slots.length, u32);243 vec<kayan_slot>append(w*.arena, w*.slots&, { .generation = 1 });244 }245 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));246 e.generation = slot*.generation;247 248 table_index: u32 = kayan_table_for(w, signature);249 t: *kayan_table = kayan_table_at(w, table_index);250 row: u64 = kayan_table_add_row(w, t, e);251 252 slot*.table = table_index;253 slot*.row = cast(row, u32);254 w*.live += 1;255 return e;256}257 258// A dead handle reports `dead`, so despawning twice is harmless.259kayan_despawn: proc(w: *kayan_world, e: kayan_entity) -> kayan_status = {260 if (!kayan_alive(w, e)) {261 return kayan_status.dead;262 }263 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));264 kayan_table_remove_row(w, kayan_table_at(w, slot*.table), cast(slot*.row, u64));265 slot*.generation += 1;266 vec<u32>append(w*.arena, w*.free&, e.index);267 w*.live -= 1;268 return kayan_status.ok;269}270 271kayan_signature: proc(w: *kayan_world, e: kayan_entity) -> u64 = {272 if (!kayan_alive(w, e)) {273 return 0;274 }275 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));276 return kayan_table_at(w, slot*.table)*.signature;277}278 279// Move `e` to the table for `signature`. Components both tables have are280// copied; components only the new table has start zeroed; components only the281// old table has are dropped.282kayan_move: proc(w: *kayan_world, e: kayan_entity, signature: u64) -> kayan_status = {283 if (!kayan_alive(w, e)) {284 return kayan_status.dead;285 }286 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));287 from_index: u32 = slot*.table;288 if (kayan_table_at(w, from_index)*.signature == signature) {289 return kayan_status.ok;290 }291 // kayan_table_for may add a table, so the table pointers are taken after it.292 to_index: u32 = kayan_table_for(w, signature);293 from: *kayan_table = kayan_table_at(w, from_index);294 to: *kayan_table = kayan_table_at(w, to_index);295 from_row: u64 = cast(slot*.row, u64);296 to_row: u64 = kayan_table_add_row(w, to, e);297 for (id: u32 = 0; id < w*.component_count; id += 1) {298 if (from*.columns[id] != null && to*.columns[id] != null) {299 size: u64 = w*.infos[id]*.size;300 memcpy(to*.columns[id][to_row * size]&, from*.columns[id][from_row * size]&, size);301 }302 }303 kayan_table_remove_row(w, from, from_row);304 slot*.table = to_index;305 slot*.row = cast(to_row, u32);306 return kayan_status.ok;307}308 309// e's T, or null if e is dead or has no T. The row is checked against the310// rows in use, so a slot pointing past the end stops the program in Debug.311kayan_get: proc<T>(w: *kayan_world, e: kayan_entity) -> *T = {312 if (!kayan_alive(w, e)) {313 return null;314 }315 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));316 t: *kayan_table = kayan_table_at(w, slot*.table);317 if (t*.columns[kayan_id_of<T>()] == null) {318 return null;319 }320 rows: []T = kayan_column<T>(t);321 return rows[cast(slot*.row, u64)]&;322}323 324// e's component `id` as its bytes, or an empty slice if e is dead or has no325// such component. For tools that work from reflection records rather than from326// types, like lesson 5's inspector.327kayan_get_bytes: proc(w: *kayan_world, e: kayan_entity, id: u32) -> []u8 = {328 none: []u8 = {};329 if (!kayan_alive(w, e) || id >= w*.component_count) {330 return none;331 }332 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));333 t: *kayan_table = kayan_table_at(w, slot*.table);334 if (t*.columns[id] == null) {335 return none;336 }337 size: u64 = w*.infos[id]*.size;338 column: []u8 = t*.columns[id][0..t*.count * size];339 row: u64 = cast(slot*.row, u64);340 return column[row * size..(row + 1) * size];341}342 343// Give e a T, moving it to a new table if it did not have one.344kayan_add: proc<T>(w: *kayan_world, e: kayan_entity, value: T) -> kayan_status = {345 if (kayan_move(w, e, kayan_signature(w, e) | kayan_mask_of<T>()) == kayan_status.dead) {346 return kayan_status.dead;347 }348 kayan_get<T>(w, e)* = value;349 return kayan_status.ok;350}351 352// Take e's T away, moving it to the table without it.353kayan_remove: proc<T>(w: *kayan_world, e: kayan_entity) -> kayan_status = {354 return kayan_move(w, e, kayan_signature(w, e) & ~kayan_mask_of<T>());355}356 357// Every row of every table must belong to a live entity whose slot points back358// at that table and row, and the rows must add up to the live count. Returns359// how many of those checks failed, printing each.360kayan_verify: proc(w: *kayan_world) -> u64 = {361 problems: u64 = 0;362 rows: u64 = 0;363 for (ti: u64 = 0; ti < w*.tables.length; ti += 1) {364 t: *kayan_table = vec<kayan_table>at(w*.tables&, ti);365 ids: []kayan_entity = kayan_table_entities(t);366 for (row: u64 = 0; row < ids.length; row += 1) {367 e: kayan_entity = ids[row];368 if (!kayan_alive(w, e)) {369 printfmt("kayan_verify: table {} row {} holds dead entity {}:{}\n",370 ti, row, e.index, e.generation);371 problems += 1;372 continue;373 }374 slot: *kayan_slot = vec<kayan_slot>at(w*.slots&, cast(e.index, u64));375 if (cast(slot*.table, u64) != ti || cast(slot*.row, u64) != row) {376 printfmt("kayan_verify: {}:{} is in table {} row {}, its slot says table {} row {}\n",377 e.index, e.generation, ti, row, slot*.table, slot*.row);378 problems += 1;379 }380 }381 rows += t*.count;382 }383 if (rows != w*.live) {384 printfmt("kayan_verify: {} rows in tables, {} entities alive\n", rows, w*.live);385 problems += 1;386 }387 return problems;388}389 390// ---------------------------------------------------------------------------391// Printing392// ---------------------------------------------------------------------------393 394kayan_print_signature: proc(w: *kayan_world, signature: u64) -> void = {395 printf("{");396 first: b32 = 1;397 for (id: u32 = 0; id < w*.component_count; id += 1) {398 if ((signature & kayan_bit(id)) != 0) {399 printf(first ? "%s" : " %s", kayan_component_name(w, id));400 first = 0;401 }402 }403 printf("}");404}405 406kayan_print_tables: proc(w: *kayan_world) -> void = {407 for (ti: u64 = 0; ti < w*.tables.length; ti += 1) {408 t: *kayan_table = vec<kayan_table>at(w*.tables&, ti);409 printf(" table %d ", cast(ti, i32));410 kayan_print_signature(w, t*.signature);411 printf(":");412 for (row: u64 = 0; row < t*.count; row += 1) {413 printf(" %u:%u", t*.entities[row].index, t*.entities[row].generation);414 }415 printf("\n");416 }417}Tooling
Build and check
tools/check.py runs every program and diffs its output
tools/check.py
1"""Build every lesson, run it, and compare what it prints with its .expected file.2 3 python tools/check.py every lesson4 python tools/check.py archetype only lessons whose path contains "archetype"5 python tools/check.py --bless rewrite the .expected files from this run6 python tools/check.py --sanitize also build with AddressSanitizer and UBSan7 8A lesson passes only if all four steps are clean:9 10 1. `rin compile` prints nothing but its "generated" line,11 2. the C compiler reports no warnings (-Wall -Wextra -Werror),12 3. the program exits with 0,13 4. its output matches the .expected file beside it, byte for byte.14 15The lessons are the list in build.rin, so there is one list to keep. The16dojo's playgrounds are in it too. Each dojo block is also checked on its own17(`python tools/check.py dojo` for only them): its solution.rin, a whole program18that imports the harness, must build and print its harness.expected;19level1.rin must build and fail, since it has a gap; level2.rin must build.20 21rin is found through RIN, then RIN_HOME, then PATH; its std is the folder beside22it, as `rin build` assumes. The C compiler is CC, or clang.23"""24 25from __future__ import annotations26 27import os28import re29import shutil30import subprocess31import sys32from pathlib import Path33 34ROOT = Path(__file__).resolve().parent.parent35OUT = ROOT / "build" / "check"36 37 38def find_rin() -> Path:39 names = ["rin.exe", "rin"] if os.name == "nt" else ["rin", "rin.exe"]40 if os.environ.get("RIN"):41 return Path(os.environ["RIN"])42 if os.environ.get("RIN_HOME"):43 for name in names:44 candidate = Path(os.environ["RIN_HOME"]) / name45 if candidate.exists():46 return candidate47 for name in names:48 found = shutil.which(name)49 if found:50 return Path(found)51 sys.exit("check: rin not found; set RIN or RIN_HOME, or put rin on PATH")52 53 54def list_from_build_rin(name: str) -> list[str] | None:55 text = (ROOT / "build.rin").read_text(encoding="utf-8")56 block = re.search(rf"^{name}:[^{{]*\{{(.*?)\}}", text, re.S | re.M)57 if not block:58 return None59 return re.findall(r'"([^"]+)"', block.group(1))60 61 62def lessons_from_build_rin() -> list[Path]:63 entries = list_from_build_rin("build_entries")64 if entries is None:65 sys.exit("check: no build_entries in build.rin")66 return [ROOT / entry for entry in entries]67 68 69def run(cmd: list[str], cwd: Path = ROOT,70 env: dict[str, str] | None = None) -> subprocess.CompletedProcess[str]:71 return subprocess.run(cmd, cwd=cwd, env=env, text=True, stdout=subprocess.PIPE,72 stderr=subprocess.STDOUT, timeout=300)73 74 75# On Windows a sanitized program loads clang_rt.asan_dynamic-x86_64.dll when it76# starts, and exits with 0xC0000135 before main if the DLL is not on PATH. It77# lives in the C compiler's own resource directory, so ask the compiler.78def sanitizer_env(cc: str) -> dict[str, str] | None:79 if os.name != "nt":80 return None81 resource = run([cc, "-print-resource-dir"]).stdout.strip()82 runtime = Path(resource) / "lib" / "windows"83 if not runtime.is_dir():84 return None85 env = dict(os.environ)86 env["PATH"] = str(runtime) + os.pathsep + env.get("PATH", "")87 return env88 89 90def check(rin: Path, cc: str, lesson: Path, bless: bool, sanitize: bool,91 env: dict[str, str] | None) -> tuple[bool, str]:92 stem = lesson.stem93 work = OUT / stem94 work.mkdir(parents=True, exist_ok=True)95 c_path = work / f"{stem}.c"96 exe = work / (f"{stem}.exe" if os.name == "nt" else stem)97 98 translate = run([str(rin), "compile", str(lesson), "-o", str(c_path),99 "--header", str(work / f"{stem}.h")])100 noise = [line for line in translate.stdout.splitlines()101 if line.strip() and not line.startswith("rin: generated")]102 if translate.returncode != 0 or noise:103 return False, "rin compile:\n" + translate.stdout104 105 flags = ["-std=c11", "-g", "-Wall", "-Wextra", "-Werror"]106 flags += [f"-D{define}" for define in list_from_build_rin("build_defines") or []]107 if sanitize:108 flags += ["-fsanitize=address,undefined", "-fno-sanitize-recover=all"]109 includes = ["-I", str(rin.parent), "-I", str(rin.parent / "std")]110 libs = [] if os.name == "nt" else ["-lm"]111 build = run([cc, *flags, *includes, str(c_path), "-o", str(exe), *libs])112 if build.returncode != 0 or build.stdout.strip():113 return False, f"{cc}:\n" + build.stdout114 115 ran = run([str(exe)], env=env)116 if ran.returncode != 0:117 return False, f"exit {ran.returncode}:\n{ran.stdout}"118 119 actual = ran.stdout.replace("\r\n", "\n")120 expected_path = lesson.with_suffix(".expected")121 if bless:122 expected_path.write_text(actual, encoding="utf-8", newline="\n")123 return True, "blessed"124 if not expected_path.exists():125 return False, f"no {expected_path.name}; run with --bless after reading the output:\n{actual}"126 expected = expected_path.read_text(encoding="utf-8").replace("\r\n", "\n")127 if actual != expected:128 got, want = actual.splitlines(), expected.splitlines()129 for i in range(max(len(got), len(want))):130 g = got[i] if i < len(got) else "<missing>"131 w = want[i] if i < len(want) else "<missing>"132 if g != w:133 return False, f"line {i + 1}\n got: {g}\n want: {w}"134 return True, ""135 136 137# ---------------------------------------------------------------- the dojo138 139DOJO = ROOT / "dojo"140# A kata in progress is full of unused parameters and locals.141KATA_QUIET = ["-Wno-unused-parameter", "-Wno-unused-variable", "-Wno-unused-but-set-variable",142 "-Wno-unused-function"]143 144 145def dojo_blocks() -> list[Path]:146 return sorted((p for p in DOJO.iterdir() if p.is_dir() and re.match(r"\d+\.", p.name)),147 key=lambda p: int(p.name.split(".")[0]))148 149 150# A kata file is a whole program: it imports what it uses and the harness,151# which has main, so each of the solution and the levels builds where it is.152def dojo_build(rin: Path, cc: str, block: Path, kata: Path, where: str, sanitize: bool,153 quiet: bool) -> tuple[Path | None, str]:154 folder = OUT / "dojo" / block.name155 folder.mkdir(parents=True, exist_ok=True)156 c_path = folder / f"{where}.c"157 exe = folder / (f"{where}.exe" if os.name == "nt" else where)158 translate = run([str(rin), "compile", str(kata), "-o", str(c_path),159 "--header", str(folder / f"{where}.h")])160 noise = [line for line in translate.stdout.splitlines()161 if line.strip() and not line.startswith("rin: generated")]162 if translate.returncode != 0 or noise:163 return None, "rin compile:\n" + translate.stdout164 flags = ["-std=c11", "-g", "-Wall", "-Wextra", "-Werror", *(KATA_QUIET if quiet else [])]165 flags += [f"-D{define}" for define in list_from_build_rin("build_defines") or []]166 if sanitize:167 flags += ["-fsanitize=address,undefined", "-fno-sanitize-recover=all"]168 includes = ["-I", str(rin.parent), "-I", str(rin.parent / "std")]169 libs = [] if os.name == "nt" else ["-lm"]170 build = run([cc, *flags, *includes, str(c_path), "-o", str(exe), *libs])171 if build.returncode != 0 or build.stdout.strip():172 return None, f"{cc}:\n" + build.stdout173 return exe, ""174 175 176def check_dojo(rin: Path, cc: str, block: Path, bless: bool, sanitize: bool,177 env: dict[str, str] | None) -> tuple[bool, str]:178 exe, err = dojo_build(rin, cc, block, block / "solution.rin", "solution", sanitize, False)179 if not exe:180 return False, "solution.rin, " + err181 ran = run([str(exe)], env=env)182 if ran.returncode != 0:183 return False, f"the solution fails its harness, exit {ran.returncode}:\n{ran.stdout}"184 actual = ran.stdout.replace("\r\n", "\n")185 expected_path = block / "harness.expected"186 detail = ""187 if bless:188 expected_path.write_text(actual, encoding="utf-8", newline="\n")189 detail = "blessed"190 elif not expected_path.exists():191 return False, f"no harness.expected; run with --bless after reading the output:\n{actual}"192 elif actual != expected_path.read_text(encoding="utf-8").replace("\r\n", "\n"):193 return False, "the harness's output with the solution is not harness.expected"194 for level in (1, 2):195 exe, err = dojo_build(rin, cc, block, block / f"level{level}.rin", f"level{level}", sanitize, True)196 if not exe:197 return False, f"level{level}.rin, " + err198 if level == 1 and run([str(exe)], env=env).returncode == 0:199 return False, "level 1 passes the harness as given; its gap is not a gap"200 return True, detail201 202 203def main() -> int:204 args = sys.argv[1:]205 bless = "--bless" in args206 sanitize = "--sanitize" in args207 filters = [a for a in args if not a.startswith("--")]208 209 rin = find_rin()210 cc = os.environ.get("CC", "clang")211 lessons = [l for l in lessons_from_build_rin()212 if not filters or any(f in str(l) for f in filters)]213 214 env = sanitizer_env(cc) if sanitize else None215 216 failed = 0217 for lesson in lessons:218 ok, detail = check(rin, cc, lesson, bless, sanitize, env)219 name = lesson.relative_to(ROOT).as_posix()220 print(f"{'ok ' if ok else 'FAIL'} {name}" + (f" ({detail})" if ok and detail else ""))221 if not ok:222 failed += 1223 print(" " + detail.replace("\n", "\n "))224 print(f"{len(lessons) - failed} of {len(lessons)} passed")225 226 blocks = [b for b in dojo_blocks()227 if not filters or any(f in b.relative_to(ROOT).as_posix() for f in filters)]228 dojo_failed = 0229 for block in blocks:230 ok, detail = check_dojo(rin, cc, block, bless, sanitize, env)231 print(f"{'ok ' if ok else 'FAIL'} {block.relative_to(ROOT).as_posix()}, the solution and the levels"232 + (f" ({detail})" if ok and detail else ""))233 if not ok:234 dojo_failed += 1235 print(" " + detail.replace("\n", "\n "))236 if blocks:237 print(f"{len(blocks) - dojo_failed} of {len(blocks)} dojo blocks passed")238 return 1 if failed or dojo_failed else 0239 240 241if __name__ == "__main__":242 sys.exit(main())tools/dojo.py
1"""The dojo's runner: start a kata, run it against its harness, keep your times.2 3 python tools/dojo.py the blocks, your reps, and what is due4 python tools/dojo.py start slot_map 2 a fresh kata.rin from level 2; the clock starts5 python tools/dojo.py slot_map build the harness with your kata.rin and run it6 python tools/dojo.py slot_map --seed 42 --stop 13377 the same fuzz run, stopping before operation 13378 python tools/dojo.py slot_map --sanitize with AddressSanitizer and UBSan9 python tools/dojo.py play slot_map the block's playground10 11A block can be named by its number, its name, or the start of its name:12`3`, `slot_map` and `slot` are the same block.13 14When a started kata passes, the time since `start` is added to dojo/log.md,15and the block is due again after 1, 3, 7, 14, then 30 days, counting its16passes. Spaced reps are what make a block stick, more than many in one day.17 18If the harness crashes rather than failing a check (an out-of-bounds index,19say), the runner runs it again with --trace to show the operation that was20running.21 22Your kata.rin files are yours: git ignores them, and `start` will not23overwrite one you have changed unless you add --force. A kata is a whole24program: it imports what it uses and the harness, which has main, so the25runner builds kata.rin, and the harness comes with it. rin and the C compiler26are found as tools/check.py finds them.27"""28 29from __future__ import annotations30 31import datetime as dt32import json33import os34import re35import shutil36import subprocess37import sys38from pathlib import Path39 40sys.path.insert(0, str(Path(__file__).resolve().parent))41from check import find_rin, list_from_build_rin # noqa: E40242 43ROOT = Path(__file__).resolve().parent.parent44DOJO = ROOT / "dojo"45OUT = ROOT / "build" / "dojo"46LOG = DOJO / "log.md"47STARTED = DOJO / ".started.json"48INTERVALS = [1, 3, 7, 14, 30]49 50 51def blocks() -> list[Path]:52 return sorted((p for p in DOJO.iterdir() if p.is_dir() and re.match(r"\d+\.", p.name)),53 key=lambda p: int(p.name.split(".")[0]))54 55 56def name_of(block: Path) -> str:57 return block.name.split(".", 1)[1]58 59 60def find(word: str) -> Path:61 found = [b for b in blocks() if word in (b.name, name_of(b), b.name.split(".")[0])]62 found = found or [b for b in blocks() if name_of(b).startswith(word)]63 if len(found) != 1:64 sys.exit(f"dojo: no single block called {word!r}; the blocks are "65 + ", ".join(name_of(b) for b in blocks()))66 return found[0]67 68 69# The target times, from the level table in each block's README:70# | 2 | every proc | 8:00 |71def targets(block: Path) -> dict[int, str]:72 readme = block / "README.md"73 text = readme.read_text(encoding="utf-8") if readme.exists() else ""74 return {int(m.group(1)): m.group(2) for m in re.finditer(r"^\| (\d) \|.*\| (\d+:\d\d) \|$", text, re.M)}75 76 77# rin writes C, and the C compiler does the rest, as tools/check.py does. A78# kata in progress is full of unused parameters and locals, so the warnings for79# those are off and warnings are not errors; tools/check.py builds the80# solutions with -Werror and every warning on.81def build(source: Path, exe: Path, sanitize: bool) -> None:82 exe.parent.mkdir(parents=True, exist_ok=True)83 rin = find_rin()84 c_path = exe.with_suffix(".c")85 translate = subprocess.run([str(rin), "compile", str(source), "-o", str(c_path),86 "--header", str(exe.with_suffix(".h"))], text=True,87 stdout=subprocess.PIPE, stderr=subprocess.STDOUT)88 noise = [l for l in translate.stdout.splitlines() if l.strip() and not l.startswith("rin: generated")]89 if translate.returncode != 0 or noise:90 print(translate.stdout, end="")91 sys.exit(1)92 cc = os.environ.get("CC", "clang")93 flags = ["-std=c11", "-g", "-Wall", "-Wextra", "-Wno-unused-parameter", "-Wno-unused-variable",94 "-Wno-unused-but-set-variable", "-Wno-unused-function"]95 flags += [f"-D{define}" for define in list_from_build_rin("build_defines") or []]96 if sanitize:97 flags += ["-fsanitize=address,undefined", "-fno-sanitize-recover=all"]98 includes = ["-I", str(rin.parent), "-I", str(rin.parent / "std")]99 libs = [] if os.name == "nt" else ["-lm"]100 if subprocess.run([cc, *flags, *includes, str(c_path), "-o", str(exe), *libs]).returncode != 0:101 sys.exit(1)102 103 104def load_started() -> dict:105 try:106 return json.loads(STARTED.read_text(encoding="utf-8"))107 except (OSError, ValueError):108 return {}109 110 111def save_started(started: dict) -> None:112 STARTED.write_text(json.dumps(started, indent=2) + "\n", encoding="utf-8")113 114 115def log_rows() -> list[list[str]]:116 if not LOG.exists():117 return []118 rows = []119 for line in LOG.read_text(encoding="utf-8").splitlines():120 cells = [c.strip() for c in line.strip().strip("|").split("|")]121 if len(cells) == 4 and re.match(r"\d{4}-\d\d-\d\d", cells[0]):122 rows.append(cells)123 return rows124 125 126def minutes(seconds: float) -> str:127 s = int(round(seconds))128 return f"{s // 60}:{s % 60:02d}"129 130 131def seconds(text: str) -> int:132 m, s = text.split(":")133 return int(m) * 60 + int(s)134 135 136def due(passes: list[dt.date]) -> dt.date | None:137 if not passes:138 return None139 return max(passes) + dt.timedelta(days=INTERVALS[min(len(passes), len(INTERVALS)) - 1])140 141 142def status() -> int:143 rows = log_rows()144 today = dt.date.today()145 started = load_started()146 print(f"{'block':<16}{'reps':>5} {'best time, level':<18}{'last':<12}next")147 for b in blocks():148 mine = [r for r in rows if r[1] == name_of(b)]149 passes = [dt.date.fromisoformat(r[0]) for r in mine]150 best = min(mine, key=lambda r: (-int(r[2]), seconds(r[3])), default=None)151 best_text = f"{best[3]}, level {best[2]}" if best else "-"152 next_day = due(passes)153 next_text = "now" if next_day is None or next_day <= today else next_day.isoformat()154 if name_of(b) in started:155 next_text = f"started, level {started[name_of(b)]['level']}"156 last = max(passes).isoformat() if passes else "-"157 print(f"{b.name:<16}{len(mine):>5} {best_text:<18}{last:<12}{next_text}")158 print()159 print("start one with: python tools/dojo.py start <block> <level 1 to 3>")160 return 0161 162 163def start(word: str, level: str, force: bool) -> int:164 b = find(word)165 source = b / f"level{level}.rin"166 if not source.exists():167 sys.exit(f"dojo: {name_of(b)} has no level {level}; the levels are "168 + ", ".join(p.stem[len("level"):] for p in sorted(b.glob("level*.rin"))))169 kata = b / "kata.rin"170 untouched = [p.read_text(encoding="utf-8") for p in b.glob("level*.rin")]171 if kata.exists() and kata.read_text(encoding="utf-8") not in untouched and not force:172 sys.exit(f"dojo: {kata.relative_to(ROOT)} has your work in it; add --force to replace it")173 shutil.copyfile(source, kata)174 started = load_started()175 started[name_of(b)] = {"level": int(level), "at": dt.datetime.now().isoformat(timespec="seconds")}176 save_started(started)177 target = targets(b).get(int(level))178 print(f"{kata.relative_to(ROOT)} is level {level}" + (f"; the target is {target}" if target else ""))179 print(f"the clock is running. Test it with: python tools/dojo.py {name_of(b)}")180 return 0181 182 183# Run a program, printing what it prints as it prints it, and keep the text to184# look for PASS in.185def stream(cmd: list[str]) -> tuple[int, str]:186 lines = []187 with subprocess.Popen(cmd, cwd=ROOT, text=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) as p:188 for line in p.stdout:189 print(line, end="", flush=True)190 lines.append(line)191 return p.returncode, "".join(lines)192 193 194def check(word: str, args: list[str]) -> int:195 b = find(word)196 if not (b / "kata.rin").exists():197 sys.exit(f"dojo: no kata.rin in {b.relative_to(ROOT)} yet; start one with: "198 f"python tools/dojo.py start {name_of(b)} 1")199 passthrough = [a for a in args if a != "--sanitize"]200 exe = OUT / name_of(b) / ("harness.exe" if os.name == "nt" else "harness")201 build(b / "kata.rin", exe, "--sanitize" in args)202 code, out = stream([str(exe), *passthrough])203 if code != 0 and not any(line.startswith("FAIL") for line in out.splitlines()):204 traced = subprocess.run([str(exe), *passthrough, "--trace", "1"], cwd=ROOT, text=True,205 stdout=subprocess.PIPE, stderr=subprocess.STDOUT)206 ops = [l for l in traced.stdout.splitlines() if l.startswith("operation ")]207 print("the harness crashed" + (f"; the last operation it started was\n {ops[-1]}" if ops208 else " in the scenario, after the last step printed above"))209 if code != 0 or "PASS" not in out.splitlines():210 return 1211 started = load_started()212 entry = started.pop(name_of(b), None)213 if entry and not passthrough:214 took = (dt.datetime.now() - dt.datetime.fromisoformat(entry["at"])).total_seconds()215 if not LOG.exists():216 LOG.write_text("# Dojo log\n\n| date | block | level | time |\n|---|---|---|---|\n", encoding="utf-8")217 with LOG.open("a", encoding="utf-8") as f:218 f.write(f"| {dt.date.today().isoformat()} | {name_of(b)} | {entry['level']} | {minutes(took)} |\n")219 save_started(started)220 target = targets(b).get(entry["level"])221 print(f"\nlevel {entry['level']} in {minutes(took)}" + (f", target {target}" if target else ""))222 passes = [dt.date.fromisoformat(r[0]) for r in log_rows() if r[1] == name_of(b)]223 print(f"logged in {LOG.relative_to(ROOT)}; {name_of(b)} is due again on {due(passes).isoformat()}")224 return 0225 226 227def play(word: str) -> int:228 b = find(word)229 exe = OUT / name_of(b) / ("playground.exe" if os.name == "nt" else "playground")230 build(b / f"play_{name_of(b)}.rin", exe, False)231 return subprocess.run([str(exe)], cwd=ROOT).returncode232 233 234def main() -> int:235 args = sys.argv[1:]236 if not args:237 return status()238 if args[0] in ("-h", "--help", "help"):239 print(__doc__)240 return 0241 if args[0] == "start":242 rest = [a for a in args[1:] if a != "--force"]243 if len(rest) != 2:244 sys.exit("dojo: start <block> <level>")245 return start(rest[0], rest[1], "--force" in args)246 if args[0] == "play":247 if len(args) != 2:248 sys.exit("dojo: play <block>")249 return play(args[1])250 return check(args[0], args[1:])251 252 253if __name__ == "__main__":254 sys.exit(main())dojo/dojo.rin
1// What every harness in the dojo shares: the run's settings, a random number2// generator that gives the same numbers everywhere, how a step is checked, and3// how a failure is reported. It also holds the two types the katas take as4// given: a handle, and a status.5//6// The harnesses import it, and so does every kata file that uses a handle or7// a status:8//9// import "../dojo.rin"10 11import "std/print.rin"12import "std/memops.rin"13import "std/vec.rin"14import "std/fmt.rin"15 16// A generational handle: an index, and which life of that index it was made17// for. Generations start at 1, so a zeroed handle, `{}`, is never alive and can18// mean "no handle".19dojo_handle: struct = {20 index: u32;21 generation: u32;22}23 24// What a change reports. `dead` means the handle no longer names anything25// alive, so nothing changed.26dojo_status: enum[must_use] = {27 ok,28 dead,29}30 31dojo_handle_eq: proc(a: dojo_handle, b: dojo_handle) -> b32 = {32 return a.index == b.index && a.generation == b.generation;33}34 35// Writes `index:generation`.36dojo_fmt_handle: proc(f: *fmt, h: dojo_handle) -> void = {37 fmt_u64(f, cast(h.index, u64));38 fmt_char(f, ':');39 fmt_u64(f, cast(h.generation, u64));40}41 42dojo_status_name: proc(s: dojo_status) -> ^const char = {43 return s == dojo_status.ok ? "ok" : "dead";44}45 46dojo_bool_name: proc(b: b32) -> ^const char = {47 return b ? "true" : "false";48}49 50// One harness run: which block, the fuzz seed, how many operations, where to51// stop for the debugger, and whether to print each operation. The command line52// sets them:53//54// harness --seed 42 --ops 5000 --stop 1337 --trace 155dojo_run: struct = {56 block: ^const char;57 seed: u64;58 ops: u64;59 stop: u64;60 trace: u64;61}62 63dojo_parse_u64: proc(text: ^const c8, out: *u64) -> b32 = {64 value: u64 = 0;65 i: u64 = 0;66 while (text[i] >= '0' && text[i] <= '9') {67 value = value * 10 + cast(text[i] - '0', u64);68 i += 1;69 }70 out* = value;71 return i > 0 && text[i] == 0;72}73 74dojo_run_init: proc(argc: i32, argv: ^^c8, block: ^const char, title: ^const char) -> dojo_run = {75 run: dojo_run = {};76 run.block = block;77 run.seed = 1;78 run.ops = 20000;79 for (i: i32 = 1; i < argc; i += 2) {80 value: u64 = 0;81 if (i + 1 >= argc || !dojo_parse_u64(argv[i + 1], value&)) {82 printfmt("usage: harness [--seed N] [--ops N] [--stop N] [--trace 1]\n");83 exit(2);84 }85 if (strcmp(argv[i], "--seed") == 0) {86 run.seed = value;87 } else if (strcmp(argv[i], "--ops") == 0) {88 run.ops = value;89 } else if (strcmp(argv[i], "--stop") == 0) {90 run.stop = value;91 } else if (strcmp(argv[i], "--trace") == 0) {92 run.trace = value;93 } else {94 printfmt("usage: harness [--seed N] [--ops N] [--stop N] [--trace 1]\n");95 exit(2);96 }97 }98 printfmt("dojo: {}\n", title);99 return run;100}101 102// A step of the scenario: what was done, what came back, and what should have.103// Printed when it matches; the end of the run when it does not.104dojo_same: proc(what: ^const char, got: ^const char, want: ^const char) -> void = {105 if (strcmp(got, want) != 0) {106 printfmt("FAIL in the scenario: {} gave {}, it should give {}\n", what, got, want);107 exit(1);108 }109 printfmt(" {}: {}\n", what, got);110}111 112dojo_fuzz: proc(run: *dojo_run) -> void = {113 printfmt("fuzz: {} operations from seed {}, checked after each\n", run*.ops, run*.seed);114}115 116// Set a breakpoint here, and run with --stop N.117dojo_stopped_at: u64 = 0;118 119dojo_stop_here: proc(op: u64) -> void = {120 dojo_stopped_at = op;121}122 123// Call before operation `op`. It returns 1 when the operation should be124// described, and the caller prints the description and a newline: with125// --trace, every operation is, so a crash shows the last one that started;126// with --stop at this operation, it is, after dojo_stop_here has run, so a127// breakpoint there lands just before it.128dojo_before: proc(run: *dojo_run, op: u64) -> b32 = {129 if (run*.stop == op) {130 dojo_stop_here(op);131 printfmt("stopped before operation {}: ", op);132 return 1;133 }134 if (run*.trace != 0) {135 printfmt("operation {}: ", op);136 return 1;137 }138 return 0;139}140 141// A failure in the fuzz run, in three parts: dojo_fail starts the line, the142// caller prints what went wrong and a newline, and dojo_failed says how to get143// back here in the debugger and ends the run.144dojo_fail: proc(run: *dojo_run, op: u64) -> void = {145 printfmt("FAIL at operation {} (seed {}): ", op, run*.seed);146}147 148dojo_failed: proc(run: *dojo_run, op: u64) -> void = {149 printfmt(" debug it: build/dojo/{}/harness --seed {} --stop {}\n", run*.block, run*.seed, op);150 printfmt(" with a breakpoint on dojo_stop_here, which runs just before that operation\n");151 exit(1);152}153 154dojo_pass: proc() -> void = {155 printfmt("PASS\n");156}157 158// splitmix64: a few lines, fast, and the same numbers on every machine and in159// rust-dod's dojo, so a seed means the same run everywhere. u64 arithmetic160// wraps, as splitmix64 wants.161dojo_rng: struct = {162 state: u64;163}164 165dojo_rng_next: proc(r: *dojo_rng) -> u64 = {166 r*.state = r*.state + 11400714819323198485'u64;167 z: u64 = r*.state;168 z = (z ^ (z shr 30)) * 13787848793156543929'u64;169 z = (z ^ (z shr 27)) * 10723151780598845931'u64;170 return z ^ (z shr 31);171}172 173// A number in 0..n.174dojo_rng_below: proc(r: *dojo_rng, n: u64) -> u64 = {175 return dojo_rng_next(r) % n;176}build.rin
1// How the lessons are built. Read by `rin build`, not executed.2//3// Every lesson is its own program, named after its file. tools/check.py reads4// this same list, builds each lesson, runs it and compares its output with the5// .expected file beside it.6build_name: ^const char = "rin-dod";7build_dir: ^const char = "build";8build_generator: ^const char = "Ninja";9build_compiler: ^const char = "clang-cl";10build_type: ^const char = "Debug";11 12// std's string8 calls fopen, which the Windows SDK marks deprecated.13build_defines: [_]^const char = { "_CRT_SECURE_NO_WARNINGS" };14 15build_entries: [_]^const char = {16 // Build an ECS: start here, README.md17 "ecs/src/1.entities/1.1.ids/ids.rin",18 "ecs/src/1.entities/1.2.despawn/despawn.rin",19 "ecs/src/1.entities/1.3.generations/generations.rin",20 "ecs/src/1.entities/1.4.entity_index/entity_index.rin",21 "ecs/src/2.components/2.1.components/components.rin",22 "ecs/src/2.components/2.2.columns/columns.rin",23 "ecs/src/2.components/2.3.tables/tables.rin",24 "ecs/src/2.components/2.4.archetypes/archetypes.rin",25 "ecs/src/2.components/2.5.typed_access/typed_access.rin",26 "ecs/src/2.components/2.6.add_remove/add_remove.rin",27 "ecs/src/2.components/2.7.inspecting/inspecting.rin",28 "ecs/src/3.systems/3.1.queries/queries.rin",29 "ecs/src/3.systems/3.2.systems/systems.rin",30 "ecs/src/3.systems/3.3.commands/commands.rin",31 "ecs/src/3.systems/3.4.game/game.rin",32 33 // Advanced: layout and SIMD, docs/README.md34 "src/1.aos_vs_soa/aos_vs_soa.rin",35 "src/2.archetype_table/archetype_table.rin",36 "src/3.many_archetypes/many_archetypes.rin",37 "src/4.entity_handles/entity_handles.rin",38 "src/5.inspector/inspector.rin",39 "src/6.kayan/kayan.rin",40 "src/7.access_patterns/access_patterns.rin",41 42 // The dojo's playgrounds, dojo/README.md43 "dojo/1.free_list/play_free_list.rin",44 "dojo/2.swap_remove/play_swap_remove.rin",45 "dojo/3.slot_map/play_slot_map.rin",46 "dojo/4.column/play_column.rin",47 "dojo/5.commands/play_commands.rin",48 "dojo/6.bitset/play_bitset.rin",49 "dojo/7.arena/play_arena.rin",50 "dojo/8.index/play_index.rin",51 "dojo/9.interning/play_interning.rin",52 "dojo/10.edges/play_edges.rin",53 "dojo/11.reflect/play_reflect.rin",54 "dojo/12.watermark/play_watermark.rin",55 "dojo/13.dispatch/play_dispatch.rin",56};.gitignore
1build/2tools/__pycache__/3# The dojo: your katas, and the clock of the one you started.4dojo/*/kata.rin5dojo/.started.jsonLICENSE
1MIT License2 3Copyright (c) 2026 Ibrahim Hindawi4 5Permission is hereby granted, free of charge, to any person obtaining a copy6of this software and associated documentation files (the "Software"), to deal7in the Software without restriction, including without limitation the rights8to use, copy, modify, merge, publish, distribute, sublicense, and/or sell9copies of the Software, and to permit persons to whom the Software is10furnished to do so, subject to the following conditions:11 12The above copyright notice and this permission notice shall be included in all13copies or substantial portions of the Software.14 15THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR16IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,17FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE18AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER19LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,20OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE21SOFTWARE.