Stencil testing

The depth buffer we just looked at has a companion. In the format window_create_depth creates, DXGI_FORMAT_D24_UNORM_S8_UINT, each pixel holds 24 bits of depth and, beside them, 8 more bits: the stencil buffer. Those 8 bits give every pixel a small integer, 0 to 255, which shaders never read or write. Instead, draws can be told to write values into it, and other draws can be told to keep or drop their pixels depending on what is there. That second part is the stencil test, and it runs in the output merger together with the depth test; as we will see, what a draw writes into the stencil can depend on the outcome of both.

Here is the idea in LearnOpenGL's picture. The stencil buffer is cleared to 0, a frame of 1s is written into it, and then a scene is drawn that only keeps the pixels where the stencil holds 1:

A colour buffer, a stencil buffer holding a rectangular frame of 1s, and the result after the stencil test: only the frame shows (LearnOpenGL's diagram)

Whatever the effect, the recipe usually has the same shape:

  1. Draw some objects with stencil writes on, so their shape is recorded in the stencil buffer.
  2. Turn stencil writes off.
  3. Draw other objects with a stencil test that keeps or drops their pixels according to what step 1 left behind.

The stencil buffer is cleared with the depth buffer. The clear in window_begin, which every lesson calls at the start of a frame, already includes it, with 0 as the value:

movss xmm3, 1.0
vcall(context, ID3D11DeviceContextVtbl.ClearDepthStencilView, depth_dsv,
      D3D11_CLEAR_FLAG.D3D11_CLEAR_DEPTH | D3D11_CLEAR_FLAG.D3D11_CLEAR_STENCIL, 0, 0)

The 1.0 in xmm3 is the depth to clear to (the fourth argument counting this, so its float travels in the fourth xmm register, and the first 0 only holds its place), and the last 0 is the stencil value.

Stencil functions

OpenGL configures stencil testing with three calls: glEnable(GL_STENCIL_TEST), glStencilFunc for the test and glStencilOp for the writes, plus glStencilMask for which bits may change. In Direct3D all of it, together with the depth settings from the previous chapter, is one immutable object: the same depth-stencil state. The lesson builds its states with one procedure. What its three states have in common is written once, as the initialiser of the description, a global named after the procedure that uses it:

; create_stencil_state's. What the three states share is the initialiser;
; the code fills in the depth test, the write mask and the comparison. The
; stencil passes REPLACE: where a draw passes, the reference value given at
; OMSetDepthStencilState is written (if the write mask lets it).
create_stencil_state_desc: D3D11_DEPTH_STENCIL_DESC = {
    .DepthWriteMask = D3D11_DEPTH_WRITE_MASK.D3D11_DEPTH_WRITE_MASK_ALL,
    .DepthFunc = D3D11_COMPARISON_FUNC.D3D11_COMPARISON_LESS,
    .StencilEnable = 1,
    .StencilReadMask = D3D11_DEFAULT_STENCIL_READ_MASK,
    .FrontFace = {
        .StencilFailOp = D3D11_STENCIL_OP.D3D11_STENCIL_OP_KEEP,
        .StencilDepthFailOp = D3D11_STENCIL_OP.D3D11_STENCIL_OP_KEEP,
        .StencilPassOp = D3D11_STENCIL_OP.D3D11_STENCIL_OP_REPLACE,
    },
    .BackFace = {                          ; the back faces the same
        .StencilFailOp = D3D11_STENCIL_OP.D3D11_STENCIL_OP_KEEP,
        .StencilDepthFailOp = D3D11_STENCIL_OP.D3D11_STENCIL_OP_KEEP,
        .StencilPassOp = D3D11_STENCIL_OP.D3D11_STENCIL_OP_REPLACE,
    },
}
create_stencil_state_out: *ID3D11DepthStencilState = {}

; ecx: stencil comparison, edx: 1 to write the stencil, r8d: 1 to test depth
; -> rax: the state
create_stencil_state: proc(func: D3D11_COMPARISON_FUNC, write: b32, depth: b32) -> *ID3D11DepthStencilState = {
    sub rsp, 40                            ; 32 shadow, and rsp back on 16
    mov create_stencil_state_desc.DepthEnable, r8d
    mov create_stencil_state_desc.FrontFace.StencilFunc, ecx
    mov create_stencil_state_desc.BackFace.StencilFunc, ecx
    ; StencilWriteMask is a UINT8, so its store is a byte: al
    xor eax, eax
    test edx, edx
    jz no_write
    mov eax, D3D11_DEFAULT_STENCIL_WRITE_MASK
  no_write:
    mov create_stencil_state_desc.StencilWriteMask, al
    vcall(device, ID3D11DeviceVtbl.CreateDepthStencilState, create_stencil_state_desc&, create_stencil_state_out&)
    assert_hr()
    mov rax, create_stencil_state_out
    add rsp, 40
    ret
}

Piece by piece:

Two things are different from OpenGL. The reference value is not in the state at all. It is given each time the state is bound, as the last argument of OMSetDepthStencilState, so one state object serves for any reference. And Direct3D has a separate set of operations for front-facing and back-facing triangles, FrontFace and BackFace (OpenGL has the same in glStencilOpSeparate). We draw with culling off, so both kinds of triangle reach the test, and the description gives both the same operations.

Every store names its field, and rasm takes the offset and the size from the description's type, which is the SDK's own D3D11_DEPTH_STENCIL_DESC from the binding generated from its headers; the C compiler then checks every offset used against the header. That is why the write mask goes through al: StencilWriteMask is a UINT8, the store is one byte, and a 32-bit register there would be a size error rather than a silent overwrite of the next field. The operations that never change are not stored at all; they are in the initialiser, back faces and all, so where a hand-written description would copy FrontFace into BackFace, this one simply spells both out.

The operations a stencil write can perform are these:

OperationWhat is written
D3D11_STENCIL_OP_KEEPnothing: the stored value stays
D3D11_STENCIL_OP_ZERO0
D3D11_STENCIL_OP_REPLACEthe reference value
D3D11_STENCIL_OP_INCR_SATthe stored value plus 1, stopping at 255
D3D11_STENCIL_OP_DECR_SATthe stored value minus 1, stopping at 0
D3D11_STENCIL_OP_INVERTthe stored value with every bit flipped
D3D11_STENCIL_OP_INCRthe stored value plus 1, wrapping from 255 to 0
D3D11_STENCIL_OP_DECRthe stored value minus 1, wrapping from 0 to 255
The names of the counting operations are swapped between the two APIs. OpenGL's GL_INCR stops at the maximum and GL_INCR_WRAP wraps; in Direct3D, INCR_SAT stops and plain INCR wraps. Porting GL_INCR as D3D11_STENCIL_OP_INCR gives code that works until a pixel is counted 256 times.

So with a state from create_stencil_state, a pixel that passes both tests writes the reference value into the stencil (if writes are on), and a pixel that fails either test leaves the stencil as it was.

Object outlining

Descriptions of stencil state only really make sense once you see them used, so let's build something with it: an outline around objects, the kind an editor draws around a selected object or a strategy game around selected units.

A scene in which the selected objects are surrounded by a coloured border (LearnOpenGL's diagram)

The trick is to draw each object twice. The first time, normally, and while doing so we write 1 into the stencil wherever the object lands. The second time, a little larger and in one flat colour, but only where the stencil is not 1. Everywhere the bigger copy covers the original, it is rejected; what is left is the rim that sticks out past the original's silhouette. In steps:

  1. Draw the floor, which gets no outline, without touching the stencil.
  2. Draw the cubes with a stencil test that always passes and a write of 1.
  3. Scale the cubes up slightly and draw them with a flat-colour pixel shader, a stencil test that passes only where the stencil is not 1, no stencil writes and no depth test.

The flat colour comes from a second shader file, single_color.hlsl, whose vertex shader is the usual one and whose pixel shader is as short as they come:

float4 ps_main(vs_out p) : SV_Target
{
    return float4(0.04, 0.28, 0.26, 1.0);
}
invoke(shader_create, s&, "src/4.advanced_d3d11/2.stencil_testing/stencil_testing.hlsl", cube_layout&, CUBE_LAYOUT_COUNT)
invoke(shader_create, single_color&, "src/4.advanced_d3d11/2.stencil_testing/single_color.hlsl", cube_layout&, CUBE_LAYOUT_COUNT)

Each step's configuration is one depth-stencil state, created once at startup:

invoke(create_stencil_state, D3D11_COMPARISON_FUNC.D3D11_COMPARISON_ALWAYS, 0, 1)
mov plain_state, rax
invoke(create_stencil_state, D3D11_COMPARISON_FUNC.D3D11_COMPARISON_ALWAYS, 1, 1)
mov mark_state, rax
invoke(create_stencil_state, D3D11_COMPARISON_FUNC.D3D11_COMPARISON_NOT_EQUAL, 0, 0)
mov outside_state, rax

The render loop then reads almost like the list of steps. The two cubes' positions are a small array, walked with rsi pointing at the current vec3 and ebx counting; both are registers a call leaves alone, which is why main pushes them first thing, and its frame is sized with the pushes counted:

main: proc() -> i32 = {
    push rbx
    push rsi
    sub rsp, 40                            ; draw_vertices takes 5: 32 + 8; entry 8 + two pushes 16 + 40 lands on 16
    ; the floor, leaving the stencil alone
    vcall(context, ID3D11DeviceContextVtbl.OMSetDepthStencilState, plain_state, 0)
    invoke(bind_texture, 0, floor_texture)
    invoke(m4_identity, model&)
    invoke(draw_vertices, transforms_buffer, t&, plane_vb, 6, model&)

    ; 1. the cubes, writing 1 into the stencil
    vcall(context, ID3D11DeviceContextVtbl.OMSetDepthStencilState, mark_state, 1)
    invoke(bind_texture, 0, cube_texture)
    lea rsi, positions
    xor ebx, ebx
  cubes:
    invoke(m4_translate, model&, rsi)
    invoke(draw_vertices, transforms_buffer, t&, cube_vb, 36, model&)
    add rsi, sizeof(vec3)
    inc ebx
    cmp ebx, 2
    jb cubes

    ; 2. a little larger, flat, only where the stencil is not 1: the rim
    vcall(context, ID3D11DeviceContextVtbl.OMSetDepthStencilState, outside_state, 1)
    invoke(shader_use, single_color&)
    lea rsi, positions
    xor ebx, ebx
  outlines:
    lea rcx, model
    mov rdx, rsi
    movss xmm2, 1.1                        ; the scale, the third argument
    call m4_translate_scale
    invoke(draw_vertices, transforms_buffer, t&, cube_vb, 36, model&)
    add rsi, sizeof(vec3)
    inc ebx
    cmp ebx, 2
    jb outlines
    vcall(context, ID3D11DeviceContextVtbl.OMSetDepthStencilState, 0, 0)

The reference value 1 appears twice, once when binding mark_state, where it is the value written, and once when binding outside_state, where it is the value compared against. The outline cubes are scaled by 1.1 about their own centres: m4_translate_scale in math.rasm builds translate(position) * scale(1.1) in one go, which scales first and then moves, as in Transformations. Its scale is a float, the third argument, so it goes in xmm2, and the call is written out by hand rather than with invoke. At the end the state is reset to null, the default, so that nothing drawn later is affected by accident.

Why switch the depth test off for the outline? With it on, the outline would lose its bottom edge, which lies below the floor (the scaled cubes reach down to y = -0.55), and it would vanish wherever anything nearer covers it. With it off, the outline shows through everything, which is the look an editor wants for a selection: you can find the object even when it is behind a wall. (A disabled depth test also writes no depth, so the outline does not hide anything drawn after it.)

The two marble cubes on the metal floor, each with a dark teal outline around it

The outline is not the same width all round: scaling about the centre pushes each face out by a fixed distance in the world, and perspective makes that look wider where the cube is nearer the camera. A scale is not a true constant-width outline, but for convex shapes like a cube it is convincing.

Because both cubes write into the same stencil before either outline is drawn, the outlines merge where the cubes overlap on screen: one border around the pair, not a border on top of each cube. For a group of selected units that is usually what you want. For separate borders you would clear the stencil, or use a different reference value, per object. The full program is here, and its shaders here and here.

Outlines are only one use. A stencil can limit drawing to the inside of a mirror or a portal, so a second view of the scene appears exactly there. Counting with INCR and DECR as shadow geometry is drawn is the basis of stencil shadow volumes. And the stencil is a cheap way to mark which pixels need a costly effect, so a later full-screen pass can skip the others.

Additional resources

Exercises

  1. Turn the depth test back on for the outline pass (the last argument of create_stencil_state for outside_state) and fly around the cubes. Where does the outline disappear?
  2. Give each cube its own border: write reference 1 for the first cube and 2 for the second, and draw each outline with the matching reference.
  3. Draw the floor so that it only shows through a window: before drawing it, write 1 into the stencil with a small square (without drawing its colour, using a blend state whose RenderTargetWriteMask is 0), then draw the floor with EQUAL and reference 1.
Credit: this chapter follows LearnOpenGL's Stencil testing by Joey de Vries, rewritten for Direct3D 11 and x64 assembly in rasm. LearnOpenGL's text and code are licensed CC BY-NC 4.0, and so is this page; the wording, code and screenshots here are new, and diagrams marked as LearnOpenGL's are its own.