CodeCrash

At the moment I’m working on a game project, written in Rust, using pure OpenGL for the graphics backend.

Whilst I’ve become far more confident with OpenGL once I found the amazing RenderDoc, writing plain GLSL code is still annoying. Code is often duplicated, libraries don’t really exist, and sometimes constants need to be known at compile time (like the size of an array).

This is especially problematic if these constants actually originate in your games logic (like the number of player types). Updating these values manually in your shader code is repetitive and prone to both error and simple forgetfulness.

For all these reasons it’s really helpful to build some kind of preprocessor for your GLSL code that can include other files, so you can organize your Code into manageable chunks.¹

Enter: Tera

Thanks to the amazing Rust ecosystem, we don’t actually have to write our own preprocessor. Because Rust is also often used for web projects, which need a lot of templated web-pages, a Jinja-like templating engine already exists: Tera.

A Tera template doesn’t just allow you to include other files, but can also receive a context from Rust which allows you to provide values to your OpenGL code at compile time. It even runs a simple scripting language for dynamic template instantiation. Which means we could even write simple macros. This is just what we need! So how do we integrate Tera into our own OpenGL engine?

Well, this depends on what your requirements for the preprocessing you have…

To build.rs or not to build.rs

There are basically two points in our program’s lifetime at which we can run our preprocessor:

• cargo build i.e. when we compile our Rust program
• cargo run i.e. when we run our program and compile our OpenGL shaders

For me this boils down to one question:

Why do at runtime, what you can do at compile time?

This question prods at the fact that, as developers, we sometimes tend to calculate a lot of stuff at runtime that we could have just as well hard-coded at compile-time. Whilst the idea of hard-coding might sound vile to some, it actually has a lot of merits. It’s much less error-prone, the result is already there and can be double-checked, it’s faster for the end user, as no processing is needed at runtime. Furthermore we don’t have to test nearly as much, there’s simply less to go wrong at runtime. This blog post goes into more detail on this idea.

So why would we do our preprocessing at runtime? Sometimes we might have to. Especially if we want to pass Rust values to our preprocessor that we either only know at runtime, or only know once part of our game is already compiled. This is most likely the case if your game’s game logic and rendering aren’t separated into multiple crates. If so, I recommend you just run the GLSL preprocessing right before compiling your GLSL shaders during the runtime of your game.

However in my case, the client’s rendering and game logic are in separate crates, so I can access the game logic even before compiling the client. So I’ll use the power of Cargo’s build scripts to precompile my shaders at compile time.

A build script is simply a small rust script (called build.rs) that is compiled and run by Cargo before compiling your actual crate. In my case, it looks like this:

// the shared crate contains my game logic
use shared::game::components::modules::MODULE_DATA;
use std::{env, error::Error, fs};
use tera::*;

// All my shaders reside in the 'src/shaders' directory
fn generate_shaders() -> std::result::Result<(), Box <dyn Error>> {
    let tera = Tera::new("src/shaders/*")?;
    println!("cargo:rerun-if-changed=src/shaders/");

    let mut context = Context::new();
    // You can basically insert any data you want here, as long as it's Serialize+Deserialize.
    // In my case this is the data for all types of unit modules.
    context.insert("module_data", &MODULE_DATA);

    let output_path = env::var("OUT_DIR")?;
    fs::create_dir_all(format!("{}/shaders/", output_path))?;

    for file in fs::read_dir("src/shaders")? {
        let file = file?.file_name();
        let file_name = file.to_str().unwrap();

        let result = tera.render(file_name, &context)?;
        fs::write(format!("{}/shaders/{}", output_path, file_name), result)?;
        println!("cargo:rerun-if-changed=src/shaders/{}", file_name);
    }
    Ok(())
}

fn main() {
    if let Err(err) = generate_shaders() {
         // panic here for a nicer error message, otherwise it will 
        // be flattened to one line for some reason
        panic!("Unable to generate shaders\n{}", err);
    }
}

And to make Tera available in the build.rs script, we have to add this to our Cargo.toml:

[build-dependencies]
# Shader preprocessing
tera="1"
# Game logic (use your own crate here)
shared={path="../shared"}

This code will iterate over all files in the src/shaders directory, instantiate (render) the file as a Tera template and write it to wherever the OUT_DIR environment variable points to. Unfortunately we can’t write our generated templates back out into our src directory. This is mandated by convention as described in the Cargo docs. As a consequence we can’t easily inspect the generated source code though. We’ll fix this later.

Also note the generous amount of ?-Operators in this code. Because we’re doing this at compile-time, we can freely abort the build should something go wrong. We don’t have to try to fix the problem at runtime or to build a nice error message for the user. This output is for the developer and can be a bit crude². If you are using a similar system at runtime, you should probably build something more sophisticated though.

Using the preprocessor

In any shader in the src/shaders directory, we can now use Tera’s {% include "myfile.glsl" %} directive to include other glsl code. We can also access our game’s constants at compile time using the Tera Context. In my case I use {{ module_data | length }} to define the length of some of my uniform arrays. I’ll never have to update that number by hand again!

And there’s a lot more we can do, as we have the full power of Tera available. We can generate code conditionally, in loops, and even inherit code from other templates. Really, your imagination is the limit here!

So let’s get those shaders compiling, shall we…

Getting the shaders into Rust

Similar to how we preprocess the shaders at compile time, we can also take care of this step during compile time. Rust provides the <a href="https://doc.rust-lang.org/std/macro.include_str.html">include_str!</a> macro, to include an external file as a &'static str. To make this even easier, I wrote a small wrapper macro:

#[macro_export]
macro_rules! include_shader {
    ($path:literal) => {
        include_str!(concat!(env!("OUT_DIR"), "/shaders/", $path))
    };
}

Now we can just write include_shader!("myshader.glsl") to get our shader source into our Rust program. No external file loading required!

As this will hard-code our shaders into our Rust executable, we don’t even have to ship them externally with the project. But this also means we have to recompile our Rust code every time our shader code changes. Have you noticed the cargo:rerun-if-changed= prints in build.rs yet?

Printing these lines for every shader file won’t actually be visible to the developer, but notifies cargo that it needs to recompile, should the file at the provided path change. We also add such a print for the entire directory as this might notify cargo of newly added/removed files, but this behavior is platform-dependent.

Now that our shader source is in Rust, we can finally compile it:

pub fn create_whitespace_cstring_with_len(len: usize) -> CString {
    // allocate buffer of correct size
    let mut buffer: Vec<u8> = Vec::with_capacity(len + 1);
    // fill it with len spaces
    buffer.extend([b' '].iter().cycle().take(len));
    // convert buffer to CString
    unsafe { CString::from_vec_unchecked(buffer) }
}

fn shader_from_source(source: &CStr, kind: gl::types::GLenum) -> Result <gl::types::GLuint, String> {
    let id = unsafe { gl::CreateShader(kind) };
    unsafe {
        gl::ShaderSource(id, 1, &source.as_ptr(), std::ptr::null());
        gl::CompileShader(id);
    }

    let mut success: gl::types::GLint = 1;
    unsafe {
        gl::GetShaderiv(id, gl::COMPILE_STATUS, &mut success);
    }

    if success == 0 {
        let mut len: gl::types::GLint = 0;
        unsafe {
            gl::GetShaderiv(id, gl::INFO_LOG_LENGTH, &mut len);
        }

        let error = create_whitespace_cstring_with_len(len as usize);

        unsafe {
            gl::GetShaderInfoLog(
                id,
                len,
                std::ptr::null_mut(),
                error.as_ptr() as *mut gl::types::GLchar,
            );
        }

        let shader_source = source
            .to_str()
            .unwrap_or("???")
            .lines()
            .enumerate()
            .map(|(i, line)| format!("{:03} {}", i + 1, line))
            .collect::< Vec<_> >();
        let shader_source = shader_source.join("\n");

        return Err(format!(
            "{}\n\n--- Shader source ---\n{}",
            error.to_string_lossy().into_owned(),
            shader_source
        ));
    }

    Ok(id)
}

let vert_shader = shader_from_source(&CString::new(include_shader!("module.vert"))?, gl::VERTEX_SHADER)?;

You might notice that if we receive an error from the GLSL compiler, we append the shader source to the error message. Because now the GLSL code we compile might be very different from the data in our shaders directory, with all the included files and instantiated templates, that the line numbers in the OpenGL error message are very likely to be off, making them even more useless.

And as I mentioned earlier, our generated GLSL code is only available in the OUT_DIR, which can be hard to find.

For this reason, we split the GLSL source into lines, add line numbers to it and add it to the error message. So instead of trying to find our error here:

#version 330 core

uniform mat3 mvp;

layout (location = 0) in vec2 Position;
layout (location = 1) in vec3 UV;

out vec3 uv;

{% include "test.glsl" %}

void main()
{
    vec2 position = (mvp * vec3(Position, 1.0f)).xy;
    gl_Position = vec4(position, 0.5, 1.0);
    uv = UV;
}

We can clearly see what went wrong:

Conclusion

Combining Cargo’s build system and tera to precompile GLSL shader code gives us a lot of flexibility when writing shaders. Code can be organized better, is easier to maintain, easier to integrate with data available in Rust whilst not compromising runtime performance or complexity.

And if your requirements are different, it should be easy to modify this system to work at runtime instead. Simply don’t write your shaders back into the file system, but directly compile them after the tera.render call and maybe add more thorough error checking/handling. And maybe leave a comment showing off your code 😉.

Lastly I’d be very interested in how you write/organize your shaders? Do you use a similar system? Plain GLSL? What drawbacks did you find? (Pretty sure this will confuse some tooling, e.g. syntax highlighters). What would you improve? Has this post helped you out?

Leave a comment, I’d love to hear from you.