previously we created our first real abstraction, a Handle type that represents a javascript object in rust, and keeps the object alive until it’s dropped.

but if we benchmark it, we find that creating and destroying a Handle takes well over 1500ns, which is incredibly inefficient. what gives?

#![cfg_attr(feature = "bench", feature(test))]
#[cfg(feature = "bench")]
extern crate test;

#[cfg(feature = "bench")]
#[bench]
fn bench(bencher: &mut test::bench::Bencher) {
    unsafe {
        let ctx = Context::create();
        ctx.push_c_function(Some(print), DUK_VARARGS);
        ctx.put_global_string(c"print".as_ptr());

        ctx.eval_string(cr#"
            var HANDLES = {};
            var NEXT_HANDLE_KEY = 0;
        "#.as_ptr());
        ctx.pop();

        bencher.iter(|| {
            ctx.push_string(c"ao!!".as_ptr());
            let handle = Handle::new_from_stack(&ctx, -1);
            ctx.pop();
        });
    }
}

$ cargo +nightly bench --features bench
test handle_new_drop ... bench:       1,709.24 ns/iter (+/- 14.79)

8. optimising handles with samply

samply is a sampling profiler that integrates with the excellent firefox profiler.

i used to use cargo-flamegraph, and flamegraph.pl before that, but the firefox profiler UI is so much more powerful (and unlike the perfetto UI, i actually understand how to use it effectively). samply itself is also both faster and easier to use, in large part because it resolves stacks and symbols on the fly, no postprocessing needed.

let’s get samply and perf (required by samply on linux), and while we’re at it, let’s set up mold for faster linking, since we’re gonna be iterating a lot.

with import <nixpkgs> {};
(mkShell.override { stdenv = clangStdenv; }) {
  name = "duktest-shell";
  LIBCLANG_PATH = lib.makeLibraryPath [ llvmPackages.clang-unwrapped.lib ];
  buildInputs = [ samply perf mold ];
  RUSTFLAGS = "-Clink-arg=-fuse-ld=mold";
}

we’ll need a couple of linux kernel settings too. the former will probably be familiar to anyone who has done profiling on linux, but the latter is also needed to avoid a “Failed to start profiling: mmap failed” error when running samply.

$ echo 1 | sudo tee /proc/sys/kernel/perf_event_paranoid
$ sudo sysctl kernel.perf_event_mlock_kb=2048

now let’s take a profile. for each profile i take, i’ll resolve everything and upload it, so you can see what i’m seeing.

$ cargo +nightly bench --no-run --features bench
$ samply record -- cargo +nightly bench --features bench

8.1. avoiding bytecode compilation (1709ns → 331ns)

focusing on the bencher.iter() closure, 88% of the time is spent in duk_eval_raw(), and 80pp of that is in duk_compile_raw(). we’re compiling NEXT_HANDLE_KEY++ over and over, every single time we create a handle!

https://share.firefox.dev/44UWvlj

let’s turn that into a function.

diff --git a/src/context.rs b/src/context.rs
index c8090f0..c192483 100644
--- a/src/context.rs
+++ b/src/context.rs
@@ -25,2 +25,5 @@ impl Context {
                 var NEXT_HANDLE_KEY = 0;
+                function next_handle_key() {
+                    return NEXT_HANDLE_KEY++;
+                }
             "#.as_ptr());
diff --git a/src/main.rs b/src/main.rs
index 4d13455..7a8ecb4 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -43,3 +43,4 @@ impl Handle<'_> {
             ctx.get_global_string(c"HANDLES".as_ptr());
-            ctx.eval_string(c"NEXT_HANDLE_KEY++".as_ptr());
+            ctx.get_global_string(c"next_handle_key".as_ptr());
+            ctx.call(0);
             let key = ctx.get_uint(-1);

$ cargo +nightly bench --features bench
test handle_new_drop ... bench:         331.94 ns/iter (+/- 3.03)

8.2. caching heap pointers (331ns → 245ns)

now we’re spending 29% of our time in duk_get_global_string(), and 22pp of that in duk_get_prop(). looking up properties is not fast!

https://share.firefox.dev/4gqvc9A

let’s cache the heap pointers for HANDLES and next_handle_key().

diff --git a/src/context.rs b/src/context.rs
index c192483..8d02f0d 100644
--- a/src/context.rs
+++ b/src/context.rs
@@ -7,2 +7,4 @@ pub struct Context {
     ctx: *mut duk_context,
+    pub(crate) handles: *mut c_void,
+    pub(crate) next_handle_key: *mut c_void,
 }
@@ -31,3 +33,9 @@ impl Context {

-            Self { ctx }
+            duk_get_global_string(ctx, c"HANDLES".as_ptr());
+            duk_get_global_string(ctx, c"next_handle_key".as_ptr());
+            let handles = duk_get_heapptr(ctx, -2);
+            let next_handle_key = duk_get_heapptr(ctx, -1);
+            duk_pop_2(ctx);
+
+            Self { ctx, handles, next_handle_key }
         }
diff --git a/src/main.rs b/src/main.rs
index ca3c650..07005f1 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -28,3 +28,3 @@ impl Drop for Handle<'_> {
             // `delete HANDLES[rust self.key]`
-            self.ctx.get_global_string(c"HANDLES".as_ptr());
+            self.ctx.push_heapptr(self.ctx.handles);
             self.ctx.del_prop_index(-1, self.key);
@@ -42,4 +42,4 @@ impl Handle<'_> {
             let index = ctx.normalize_index(index);
-            ctx.get_global_string(c"HANDLES".as_ptr());
-            ctx.get_global_string(c"next_handle_key".as_ptr());
+            ctx.push_heapptr(ctx.handles);
+            ctx.push_heapptr(ctx.next_handle_key);
             ctx.call(0);

$ cargo +nightly bench --features bench
test handle_new_drop ... bench:         245.00 ns/iter (+/- 8.03)

8.3. avoiding js function calls (245ns → 152ns)

now we’re spending 33% of our time in duk__handle_call_raw(), 33% of our time in duk_put_prop(), and 22% of our time in duk_del_prop().

https://share.firefox.dev/4vmq0qY

let’s move the steps for generating the next handle key out of javascript entirely. with a u64, we should be able to create 2³² new handles per second for 136 years, which is probably good enough. unfortunately duktape doesn’t support bigint yet, so we’ll only be able to create 2⁵³ handles without collisions, which comes out to 2²⁴ handles per second (59ns each) for 17 years. i don’t feel as comfortable with this, but we’ll replace it with something better soon anyway.

diff --git a/src/context.rs b/src/context.rs
index 8d02f0d..e921295 100644
--- a/src/context.rs
+++ b/src/context.rs
@@ -8,3 +8,3 @@ pub struct Context {
     pub(crate) handles: *mut c_void,
-    pub(crate) next_handle_key: *mut c_void,
+    pub(crate) next_handle_key: std::sync::atomic::AtomicU64,
 }
@@ -26,6 +26,2 @@ impl Context {
                 var HANDLES = {};
-                var NEXT_HANDLE_KEY = 0;
-                function next_handle_key() {
-                    return NEXT_HANDLE_KEY++;
-                }
             "#.as_ptr());
@@ -34,8 +30,6 @@ impl Context {
             duk_get_global_string(ctx, c"HANDLES".as_ptr());
-            duk_get_global_string(ctx, c"next_handle_key".as_ptr());
-            let handles = duk_get_heapptr(ctx, -2);
-            let next_handle_key = duk_get_heapptr(ctx, -1);
-            duk_pop_2(ctx);
+            let handles = duk_get_heapptr(ctx, -1);
+            duk_pop(ctx);

-            Self { ctx, handles, next_handle_key }
+            Self { ctx, handles, next_handle_key: 0.into() }
         }
diff --git a/src/main.rs b/src/main.rs
index 07005f1..3e321e8 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -20,3 +20,3 @@ struct Handle<'ctx> {
     ctx: &'ctx Context,
-    key: duk_uint_t,
+    key: u64,
 }
@@ -29,3 +29,4 @@ impl Drop for Handle<'_> {
             self.ctx.push_heapptr(self.ctx.handles);
-            self.ctx.del_prop_index(-1, self.key);
+            self.ctx.push_number(self.key as f64);
+            self.ctx.del_prop(-2);
             self.ctx.pop();
@@ -43,5 +44,4 @@ impl Handle<'_> {
             ctx.push_heapptr(ctx.handles);
-            ctx.push_heapptr(ctx.next_handle_key);
-            ctx.call(0);
-            let key = ctx.get_uint(-1);
+            let key = ctx.next_handle_key.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
+            ctx.push_number(key as f64);
             ctx.dup(index);

$ cargo +nightly bench --features bench
test handle_new_drop ... bench:         152.22 ns/iter (+/- 1.02)

8.4. avoiding stringifying integers (152ns → 103ns)

now we’re spending 44% of our time in duk_del_prop() and 39% of our time in duk_put_prop(), with 23pp plus 29pp of that in duk_to_string(). in other words, we’re spending over half of our time stringifying integers in decimal!

https://share.firefox.dev/4ym5iKO

let’s pack the bits of our u64 into a nine-byte string, of the form FF xx xx xx xx xx xx xx xx_h. why the leading FF_h?

duktape API strings are based on CESU-8, which is like UTF-8 but encodes values above U+FFFF differently and less efficiently (making it not a superset of UTF-8):

// U+10000 LINEAR B SYLLABLE B008 A
duk_eval_string(ctx, cr#""\uD800\uDC00""#.as_ptr());
let mut test_len = 0usize;
let test = duk_get_lstring(ctx, -1, &mut test_len as *mut _);

// CESU-8: `[ed, a0, 80, ed, b0, 80]`
eprintln!("{:02x?}", std::slice::from_raw_parts(test as *const u8, test_len));
duk_pop(ctx);

// UTF-8: `[f0, 90, 80, 80]`
eprintln!("{:02x?}", "\u{10000}".as_bytes());

but they also allow the encoding of:

• unpaired surrogates, which are necessary to represent javascript strings faithfully. duktape encodes these using the same technique as CESU-8, albeit technically out of spec. the newer, more efficient UTF-8-based counterpart to this approach is WTF-8, created by my (now) colleague Simon Sapin.

• Symbol values, using byte sequences that are not otherwise valid. for example, Symbol.toPrimitive is \x81Symbol.toPrimitive\xff.

  • and “hidden” Symbol values, which can only be created outside javascript, and need not ever be exposed to javascript. notably these have the form \xff followed by any sequence of bytes.

diff --git a/src/main.rs b/src/main.rs
index 3e321e8..61de949 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -29,3 +29,4 @@ impl Drop for Handle<'_> {
             self.ctx.push_heapptr(self.ctx.handles);
-            self.ctx.push_number(self.key as f64);
+            let key_symbol = Self::key_to_symbol(self.key);
+            self.ctx.push_lstring(key_symbol.as_ptr() as *const _, key_symbol.len());
             self.ctx.del_prop(-2);
@@ -37,2 +38,7 @@ impl Drop for Handle<'_> {
 impl Handle<'_> {
+    fn key_to_symbol(key: u64) -> [u8; 9] {
+        let key = key.to_ne_bytes();
+        [0xff, key[0], key[1], key[2], key[3], key[4], key[5], key[6], key[7]]
+    }
+
     /// create a handle for the object at `index` on the stack,
@@ -45,3 +51,4 @@ impl Handle<'_> {
             let key = ctx.next_handle_key.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
-            ctx.push_number(key as f64);
+            let key_symbol = Self::key_to_symbol(key);
+            ctx.push_lstring(key_symbol.as_ptr() as *const _, key_symbol.len());
             ctx.dup(index);

$ cargo +nightly bench --features bench
test handle_new_drop ... bench:         103.73 ns/iter (+/- 0.67)

8.5. array object fast path (103ns → 79ns)

now we’re spending 39% of our time in duk_push_lstring(), with 37pp of that in duk_heap_strtable_intern(), and we also spend 28% and 16% in duk_put_prop() and duk_del_prop() respectively. in general, javascript property keys are strings, and property accesses require both string interning and hash table lookup.

https://share.firefox.dev/3SWUsdR

to make most Array objects faster, duktape has a fast path for array index accesses on those objects, which stores values as an array internally. for that fast path to kick in though, we would need to meet a few requirements:

• the object needs to be an Array object
• the array needs to be dense, not sparse (no delete holes)
• all array index properties need to have default attributes

let’s rework HANDLES to meet these requirements. if we’re gonna store handles as an array, we’ll want to reuse low indices whenever possible, to keep the array only as large as the peak number of concurrent handles.

another reason to reuse low indices is that by definition, an array index is less than 2³². my rule of thumb for avoiding collisions in integer id types is that u32 requires reuse to avoid collisions, u64 can be sequentially issued without reuse (up to 2³² per second for 136 years), and u128 can be randomly issued without reuse (hence UUIDv4).

when we destroy handles, we can ensure that future handles reuse their keys in constant time and constant extra space, by encoding a free list in their place, and keeping track of the head of that linked list:

• initial state:
  HANDLES = [object, object, object]
head → none

• destroy handle 1:
  HANDLES = [object, null, object]
head → handle 1 → none

• destroy handle 0:
  HANDLES = [1, null, object]
head → handle 0 → handle 1 → none

• create handle (handle 0):
  HANDLES = [object, null, object]
head → handle 1 → none

• create handle (handle 1):
  HANDLES = [object, object, object]
head → none

• create handle (handle 2):
  HANDLES = [object, object, object, object]
head → none

diff --git a/src/context.rs b/src/context.rs
index e921295..d1f8b65 100644
--- a/src/context.rs
+++ b/src/context.rs
@@ -1,2 +1,2 @@
-use std::{ffi::{CStr, c_char, c_void}, ptr::null_mut};
+use std::{cell::Cell, ffi::{CStr, c_char, c_void}, ptr::null_mut, rc::Rc};

@@ -8,3 +8,4 @@ pub struct Context {
     pub(crate) handles: *mut c_void,
-    pub(crate) next_handle_key: std::sync::atomic::AtomicU64,
+    pub(crate) next_handle_key: Cell<u32>,
+    pub(crate) free_handle_key: Rc<Cell<Option<u32>>>,
 }
@@ -25,3 +26,3 @@ impl Context {
             duk_eval_string(ctx, cr#"
-                var HANDLES = {};
+                var HANDLES = [];
             "#.as_ptr());
@@ -33,3 +34,3 @@ impl Context {

-            Self { ctx, handles, next_handle_key: 0.into() }
+            Self { ctx, handles, next_handle_key: 0.into(), free_handle_key: Rc::new(None.into()) }
         }
diff --git a/src/main.rs b/src/main.rs
index 61de949..15204c4 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -20,3 +20,3 @@ struct Handle<'ctx> {
     ctx: &'ctx Context,
-    key: u64,
+    key: u32,
 }
@@ -27,7 +27,10 @@ impl Drop for Handle<'_> {
         unsafe {
-            // `delete HANDLES[rust self.key]`
+            // `HANDLES[rust self.key] = tombstone`
             self.ctx.push_heapptr(self.ctx.handles);
-            let key_symbol = Self::key_to_symbol(self.key);
-            self.ctx.push_lstring(key_symbol.as_ptr() as *const _, key_symbol.len());
-            self.ctx.del_prop(-2);
+            if let Some(old_free_handle_key) = self.ctx.free_handle_key.replace(Some(self.key)) {
+                self.ctx.push_uint(old_free_handle_key);
+            } else {
+                self.ctx.push_null();
+            }
+            self.ctx.put_prop_index(-2, self.key);
             self.ctx.pop();
@@ -38,7 +41,2 @@ impl Drop for Handle<'_> {
 impl Handle<'_> {
-    fn key_to_symbol(key: u64) -> [u8; 9] {
-        let key = key.to_ne_bytes();
-        [0xff, key[0], key[1], key[2], key[3], key[4], key[5], key[6], key[7]]
-    }
-
     /// create a handle for the object at `index` on the stack,
@@ -47,10 +45,18 @@ impl Handle<'_> {
         unsafe {
-            // `HANDLES[NEXT_HANDLE_KEY++] = stack[rust index]`
+            // `HANDLES[key] = stack[rust index]`
             let index = ctx.normalize_index(index);
             ctx.push_heapptr(ctx.handles);
-            let key = ctx.next_handle_key.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
-            let key_symbol = Self::key_to_symbol(key);
-            ctx.push_lstring(key_symbol.as_ptr() as *const _, key_symbol.len());
+            let key = if let Some(key) = ctx.free_handle_key.get() {
+                ctx.get_prop_index(-1, key.into());
+                let old_free_handle_key = ctx.get_uint(-1);
+                ctx.free_handle_key.set(Some(old_free_handle_key));
+                ctx.pop();
+                key
+            } else {
+                let key = ctx.next_handle_key.get();
+                ctx.next_handle_key.set(key.checked_add(1).expect("too many handles"));
+                key
+            };
             ctx.dup(index);
-            ctx.put_prop(-3);
+            ctx.put_prop_index(-2, key);
             ctx.pop();

$ cargo +nightly bench --features bench
test handle_new_drop ... bench:          79.64 ns/iter (+/- 6.64)

https://share.firefox.dev/4pgwq9P

just the beginning

we now have handles that can be created and destroyed in under 80ns.

there’s so much more we can do with handles though! see JSVM, by my friend Alicia, for another duktape binding where the handles:

• support canonicalisation (at least for java → javascript) – take another handle to the same object, you get the same handle

• support chaining – drill down into a nested object without creating handles for the intermediate levels, and improve the error messages for type errors

previously we looked at how to call javascript functions in rust, and found a relatively efficient way to export rust structs to javascript.

how do we access a javascript object from rust later, without keeping it on the value stack the whole time? we could store it in a js variable, and look it up later:

let ctx = duk_create_heap(None, None, None, null_mut(), None);
duk_push_c_function(ctx, Some(print), DUK_VARARGS);
duk_put_global_string(ctx, c"print".as_ptr());

duk_push_array(ctx);
duk_push_int(ctx, 0*0);
duk_put_prop_index(ctx, -2, 0);
duk_push_int(ctx, 1*1);
duk_put_prop_index(ctx, -2, 1);
duk_push_int(ctx, 2*2);
duk_put_prop_index(ctx, -2, 2);
duk_push_int(ctx, 3*3);
duk_put_prop_index(ctx, -2, 3);

// done with the array for now!
duk_put_global_string(ctx, c"squares".as_ptr());

assert_eq!(duk_get_top(ctx), 0);
// TODO: some other work
assert_eq!(duk_get_top(ctx), 0);

duk_get_global_string(ctx, c"print".as_ptr());
duk_get_global_string(ctx, c"squares".as_ptr());
duk_call(ctx, 1);
duk_pop(ctx);

println!();
duk_destroy_heap(ctx);

$ cargo run
0,1,4,9

this works, but property lookup can be slow. and for js objects that matter to rust, we really want to be able to interact with them as rust values, and tie their lifetimes to other values that depend on them.

6. js values must outlive their rust usage

((o) duktape has a garbage-collected heap, which makes allocations for strings, buffers, and (notably) objects. duktape also never moves the top-level allocations for those, so it would seem as though we could grab the heap pointer for the array, and push it back onto the stack later when we need it again. but in reality, we crash.

// done with the array for now!
let squares = duk_get_heapptr(ctx, -1);
duk_pop(ctx);

assert_eq!(duk_get_top(ctx), 0);
// TODO: some other work
assert_eq!(duk_get_top(ctx), 0);

duk_get_global_string(ctx, c"print".as_ptr());
duk_push_heapptr(ctx, squares);
duk_call(ctx, 1);
duk_pop(ctx);

println!();
duk_destroy_heap(ctx);

$ cargo run
Segmentation fault

we no longer store the object in a variable, so it’s vulnerable to garbage collection as soon as we duk_pop(). what we need is a rust value that:

1. can access a javascript value at any time, and
2. keeps that javascript value alive for as long as it lives.

that’s a handle! we can achieve those two things by:

1. saving the heap pointer in the handle with duk_get_heapptr(), then restoring it later with duk_push_heapptr() whenever we need to access it.

2. creating a strong reference to the object in javascript, by setting a property of a global variable, then deleting that property when the handle is dropped.

duk_eval_string(ctx, cr#"
    var HANDLES = {};
    var NEXT_HANDLE_KEY = 0;
"#.as_ptr());
duk_pop(ctx);

/// rooting reference to a javascript object.
#[derive(Debug)]
struct Handle {
    /// heap pointer of the object, to access it in [`Handle::push()`]
    ptr: *mut c_void,

    /// handle key, to release the handle in [`Drop`]
    ctx: *mut duk_context,
    key: duk_uint_t,
}

/// unroot the object, allowing it to be garbage collected.
impl Drop for Handle {
    fn drop(&mut self) {
        unsafe {
            // `delete HANDLES[rust self.key]`
            duk_get_global_string(self.ctx, c"HANDLES".as_ptr());
            duk_del_prop_index(self.ctx, -1, self.key);
            duk_pop(self.ctx);
        }
    }
}

impl Handle {
    /// create a handle for the object at `index` on the stack,
    /// rooting it to prevent garbage collection.
    fn new_from_stack(ctx: *mut duk_context, index: duk_idx_t) -> Handle {
        unsafe {
            // `HANDLES[NEXT_HANDLE_KEY++] = stack[rust index]`
            let index = duk_normalize_index(ctx, index);
            duk_get_global_string(ctx, c"HANDLES".as_ptr());
            duk_eval_string(ctx, c"NEXT_HANDLE_KEY++".as_ptr());
            let key = duk_get_uint(ctx, -1);
            duk_dup(ctx, index);
            duk_put_prop(ctx, -3);
            duk_pop(ctx);

            let ptr = duk_get_heapptr(ctx, index);
            Handle { ptr, ctx, key }
        }
    }

    /// push the object onto the value stack.
    fn push(&self) {
        unsafe {
            duk_push_heapptr(self.ctx, self.ptr);
        }
    }
}

these handles work a bit better, but the Drop impl causes a crash if it happens after duk_destroy_heap(), which is true for let bindings in the same scope.

// done with the array for now!
let squares = Handle::new_from_stack(ctx, -1);
duk_pop(ctx);

assert_eq!(duk_get_top(ctx), 0);
// TODO: some other work
assert_eq!(duk_get_top(ctx), 0);

duk_get_global_string(ctx, c"print".as_ptr());
squares.push();
duk_call(ctx, 1);
duk_pop(ctx);

println!();
duk_destroy_heap(ctx);

$ cargo run
0,1,4,9
Segmentation fault

7. the js heap must outlive all handles

we need to ensure that the duktape heap and context aren’t destroyed before we run the Drop impl of any Handle, because dropping a handle requires running instructions in the context. but ctx is a raw pointer (*mut duk_context), which has no lifetime.

to give the context a lifetime, we need to use references. but raw pointers are Copy, which means we can trivially sneak them out of any reference, so we’ll need to create a non-Copy wrapper struct…

use std::{ffi::{CStr, c_char, c_void}, ptr::null_mut};

use crate::sys::*;

#[derive(Debug)]
pub struct Context {
    ctx: *mut duk_context,
}

impl Drop for Context {
    fn drop(&mut self) {
        unsafe {
            duk_destroy_heap(self.ctx);
        }
    }
}

impl Context {
    pub fn create() -> Self {
        unsafe {
            let ctx = duk_create_heap(None, None, None, null_mut(), Some(fatal));
            Self { ctx }
        }
    }
}

macro_rules! fn_wrapper {
    ($call_name:ident, $method_name:ident(ctx: *mut duk_context $(, $arg_name:ident: $arg_ty:ty)* $(,)?) $(-> $ret_ty:ty)?) => {
        #[allow(unused)]
        pub unsafe fn $method_name(&self $(, $arg_name: $arg_ty)*) $(-> $ret_ty)? {
            unsafe {
                $call_name(self.ctx $(, $arg_name)*)
            }
        }
    };
}

impl Context {
    // wrappers for functions not supported by bindgen
    fn_wrapper!(duk_eval_string, eval_string(ctx: *mut duk_context, src: *const c_char));

    // wrappers mechanically derived from bindgen output for `duktape.h`
    fn_wrapper!(duk_push_undefined, push_undefined(ctx: *mut duk_context));
    fn_wrapper!(duk_push_null, push_null(ctx: *mut duk_context));
    fn_wrapper!(duk_push_boolean, push_boolean(ctx: *mut duk_context, val: duk_bool_t));
    fn_wrapper!(duk_push_true, push_true(ctx: *mut duk_context));
    fn_wrapper!(duk_push_false, push_false(ctx: *mut duk_context));
    fn_wrapper!(duk_push_number, push_number(ctx: *mut duk_context, val: duk_double_t));
    fn_wrapper!(duk_push_nan, push_nan(ctx: *mut duk_context));
    fn_wrapper!(duk_push_int, push_int(ctx: *mut duk_context, val: duk_int_t));
    fn_wrapper!(duk_push_uint, push_uint(ctx: *mut duk_context, val: duk_uint_t));
    fn_wrapper!(duk_push_string, push_string(ctx: *mut duk_context, str_: *const ::std::os::raw::c_char, ) -> *const ::std::os::raw::c_char);
    fn_wrapper!(duk_push_lstring, push_lstring(ctx: *mut duk_context, str_: *const ::std::os::raw::c_char, len: duk_size_t, ) -> *const ::std::os::raw::c_char);
    fn_wrapper!(duk_push_pointer, push_pointer(ctx: *mut duk_context, p: *mut ::std::os::raw::c_void));
    // ...
}

extern "C" fn fatal(_: *mut c_void, message: *const c_char) {
    unsafe {
        panic!("{:?}", CStr::from_ptr(message));
    }
}

…so we can borrow it in Handle. unfortunately (for my brevity) this also means we’ll have to rewrite all of the duktape calls from duk_foo(ctx, ...) to ctx.foo(...).

let ctx = Context::create();
ctx.push_c_function(Some(print), DUK_VARARGS);
ctx.put_global_string(c"print".as_ptr());

ctx.push_array();
ctx.push_int(0*0);
ctx.put_prop_index(-2, 0);
ctx.push_int(1*1);
ctx.put_prop_index(-2, 1);
ctx.push_int(2*2);
ctx.put_prop_index(-2, 2);
ctx.push_int(3*3);
ctx.put_prop_index(-2, 3);

ctx.eval_string(cr#"
    var HANDLES = {};
    var NEXT_HANDLE_KEY = 0;
"#.as_ptr());
ctx.pop();

/// rooting reference to a javascript object.
#[derive(Debug)]
struct Handle<'ctx> {
    /// heap pointer of the object, to access it in [`Handle::push()`]
    ptr: *mut c_void,

    /// handle key, to release the handle in [`Drop`]
    ctx: &'ctx Context,
    key: duk_uint_t,
}

/// unroot the object, allowing it to be garbage collected.
impl Drop for Handle<'_> {
    fn drop(&mut self) {
        unsafe {
            // `delete HANDLES[rust self.key]`
            self.ctx.get_global_string(c"HANDLES".as_ptr());
            self.ctx.del_prop_index(-1, self.key);
            self.ctx.pop();
        }
    }
}

impl Handle<'_> {
    /// create a handle for the object at `index` on the stack,
    /// rooting it to prevent garbage collection.
    fn new_from_stack(ctx: &Context, index: duk_idx_t) -> Handle<'_> {
        unsafe {
            // `HANDLES[NEXT_HANDLE_KEY++] = stack[rust index]`
            let index = ctx.normalize_index(index);
            ctx.get_global_string(c"HANDLES".as_ptr());
            ctx.eval_string(c"NEXT_HANDLE_KEY++".as_ptr());
            let key = ctx.get_uint(-1);
            ctx.dup(index);
            ctx.put_prop(-3);
            ctx.pop();

            let ptr = ctx.get_heapptr(index);
            Handle { ptr, ctx, key }
        }
    }

    /// push the object onto the value stack.
    fn push(&self) {
        unsafe {
            self.ctx.push_heapptr(self.ptr);
        }
    }
}

// done with the array for now!
let squares = Handle::new_from_stack(&ctx, -1);
ctx.pop();

assert_eq!(ctx.get_top(), 0);
// TODO: some other work
assert_eq!(ctx.get_top(), 0);

ctx.get_global_string(c"print".as_ptr());
squares.push();
ctx.call(1);
ctx.pop();

println!();
// impl Drop for Handle will run first
// then impl Drop for Context will call duk_destroy_heap()

with the Handle borrowing Context, rust ensures that the Context is dropped after the Handle when they both go out of scope. voilà!

$ cargo run
0,1,4,9

stay tuned!

the Handle impl shown above is very inefficient, taking over 1500ns to create and destroy a single handle. next time, we’ll optimise it, down to less than 80ns.

previously we created some rust bindings for ((o) duktape, a javascript runtime that turned out to be pretty easy to embed.

we want to be able to pass structs from rust to javascript, so that scripts can act on the same information as the rest of the program.

say we have a struct that represents the author of a blog post.

struct Author {
    name: String,
    link_url: String,
    avatar_url: String,
    email_address: String,
    phone_number: String,
    pronouns: String,
    bio: String,
}

let author = Author {
    name: "shuppy".to_owned(),
    link_url: "https://shuppy.org".to_owned(),
    avatar_url: "https://shuppy.org/avatar.png".to_owned(),
    email_address: "test@shuppy.org".to_owned(),
    phone_number: "+61 491 570 006".to_owned(),
    pronouns: "she/her".to_owned(),
    bio: "
        Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor
        incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud
        exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure
        dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
        Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt
        mollit anim id est laborum.
    ".repeat(100),
};

unsafe {
    todo!();
}

4. calling js functions in rust

we want to let the user write a linkify_author() function to customise the markdown we generate for links to the author. first we create a heap and context, then we run the user’s custom code, then we call the user’s function.

let ctx = duk_create_heap(None, None, None, null_mut(), None);
duk_push_c_function(ctx, Some(print), DUK_VARARGS);
duk_put_global_string(ctx, c"print".as_ptr());

// run the user’s custom code
duk_eval_string(ctx, cr#"
    function linkify_author(author) {
        return "[" + author.name + "](" + author.link_url + ")";
    }
"#.as_ptr());
duk_pop(ctx);

// TODO: call the user’s function

// `result = pop()`
println!("{:?}", CStr::from_ptr(duk_get_string(ctx, -1)));
duk_pop(ctx);

duk_destroy_heap(ctx);

duktape’s interpreter is a stack machine. we interact with that machine by feeding it instructions that push values onto, pop values off, read from, or reorder the stack.

to call linkify_author(), we push the function, then the arguments (the author), then call that function with one argument, leaving the return value on the stack. but the author is a rust struct, so how do we push that?

// `push(linkify_author(rust author))`
duk_get_global_string(ctx, c"linkify_author".as_ptr());
// TODO: duk_push_something(rust author);
duk_call(ctx, 1);

5. exporting rust structs to js

we need to build an object in javascript representing the Author value we have in rust.

let’s reorder our instructions a bit, to untangle building the object from calling the function. this is more readable, but probably less efficient.

// TODO: duk_push_something(rust author);

// `push(linkify_author(rust author))`
duk_get_global_string(ctx, c"linkify_author".as_ptr());
duk_pull(ctx, -2);
duk_call(ctx, 1);

the obvious approach would be to build a javascript object that has all of the same fields as the rust Author, converted to javascript types.

// `author = {};`
// `for (key in author) author[key] = convert rust author[key];`
// `push(author)`
duk_push_object(ctx);
duk_push_lstring(ctx, author.name.as_ptr() as *const _, author.name.len());
duk_put_prop_string(ctx, -2, c"name".as_ptr());
duk_push_lstring(ctx, author.link_url.as_ptr() as *const _, author.link_url.len());
duk_put_prop_string(ctx, -2, c"link_url".as_ptr());
duk_push_lstring(ctx, author.avatar_url.as_ptr() as *const _, author.avatar_url.len());
duk_put_prop_string(ctx, -2, c"avatar_url".as_ptr());
duk_push_lstring(ctx, author.email_address.as_ptr() as *const _, author.email_address.len());
duk_put_prop_string(ctx, -2, c"email_address".as_ptr());
duk_push_lstring(ctx, author.phone_number.as_ptr() as *const _, author.phone_number.len());
duk_put_prop_string(ctx, -2, c"phone_number".as_ptr());
duk_push_lstring(ctx, author.pronouns.as_ptr() as *const _, author.pronouns.len());
duk_put_prop_string(ctx, -2, c"pronouns".as_ptr());
duk_push_lstring(ctx, author.bio.as_ptr() as *const _, author.bio.len());
duk_put_prop_string(ctx, -2, c"bio".as_ptr());

this works!

$ cargo run
"[shuppy](https://shuppy.org)"

but if the javascript code doesn’t end up reading all of the fields, then we’ve wasted time copying a bunch of strings for nothing.

what if we instead build an object that wraps an “inner” rust pointer, with getters for each field that only copy strings into javascript when needed?

// `author = {inner: rust author};`
// `for (key in author) Object.defineProperty(author, key, {get});`
// `push(author)`
duk_push_object(ctx);
duk_push_pointer(ctx, &author as *const _ /* 🗞️ bad */ as *mut _);
duk_put_prop_string(ctx, -2, c"inner".as_ptr());
macro_rules! define_author_getter {
    ($ctx:ident, $key:literal, |author| author.$field:ident) => {{
        duk_push_string($ctx, $key.as_ptr());
        duk_push_c_function($ctx, Some(get), 1);
        duk_def_prop($ctx, -3, DUK_DEFPROP_HAVE_GETTER);
        extern "C" fn get(ctx: *mut duk_context) -> duk_ret_t {
            unsafe {
                duk_push_this(ctx);
                duk_get_prop_string(ctx, -1, c"inner".as_ptr());
                let author = duk_get_pointer(ctx, -1) as *const Author;
                duk_pop_2(ctx);
                let result = &(*author).$field;
                duk_push_lstring(ctx, result.as_ptr() as *const _, result.len());
            }
            1
        }
    }};
}
define_author_getter!(ctx, c"name", |author| author.name);
define_author_getter!(ctx, c"link_url", |author| author.link_url);
define_author_getter!(ctx, c"avatar_url", |author| author.avatar_url);
define_author_getter!(ctx, c"email_address", |author| author.email_address);
define_author_getter!(ctx, c"phone_number", |author| author.phone_number);
define_author_getter!(ctx, c"pronouns", |author| author.pronouns);
define_author_getter!(ctx, c"bio", |author| author.bio);

we’re defining getters for each field on the instance, and if we have more than one instance, that’s quite a few instructions per instance. let’s define them on a prototype, which many instances can cheaply reuse.

duk_eval_string(ctx, cr#"
    function Author(inner) {
        this.inner = inner;
    }
"#.as_ptr());
duk_pop(ctx);

// `for (key in author) Object.defineProperty(Author.prototype, key, {get})`
duk_eval_string(ctx, c"Author.prototype".as_ptr());
macro_rules! define_author_getter {
    ($ctx:ident, $key:literal, |author| author.$field:ident) => {{
        duk_push_string($ctx, $key.as_ptr());
        duk_push_c_function($ctx, Some(get), 1);
        duk_def_prop($ctx, -3, DUK_DEFPROP_HAVE_GETTER);
        extern "C" fn get(ctx: *mut duk_context) -> duk_ret_t {
            unsafe {
                duk_push_this(ctx);
                duk_get_prop_string(ctx, -1, c"inner".as_ptr());
                let author = duk_get_pointer(ctx, -1) as *const Author;
                duk_pop_2(ctx);
                let result = &(*author).$field;
                duk_push_lstring(ctx, result.as_ptr() as *const _, result.len());
            }
            1
        }
    }};
}
define_author_getter!(ctx, c"name", |author| author.name);
define_author_getter!(ctx, c"link_url", |author| author.link_url);
define_author_getter!(ctx, c"avatar_url", |author| author.avatar_url);
define_author_getter!(ctx, c"email_address", |author| author.email_address);
define_author_getter!(ctx, c"phone_number", |author| author.phone_number);
define_author_getter!(ctx, c"pronouns", |author| author.pronouns);
define_author_getter!(ctx, c"bio", |author| author.bio);
duk_pop(ctx);

// `push(new Author(rust author))`
duk_get_global_string(ctx, c"Author".as_ptr());
duk_push_pointer(ctx, &author as *const _ /* 🗞️ bad */ as *mut _);
duk_new(ctx, 1);

indeed this is the fastest approach! if we subtract the ~360ns of work that is common to all three approaches, they take ~1167ns, ~595ns, and ~251ns respectively.

$ cargo +nightly bench --features bench
test empty_object     ... bench:         360.78 ns/iter (+/- 24.98)
test obvious_approach ... bench:       1,527.54 ns/iter (+/- 11.06)
test with_getters     ... bench:         955.30 ns/iter (+/- 10.53)
test with_prototype   ... bench:         611.16 ns/iter (+/- 6.16)

stay tuned!

next time, we’ll look at building some abstractions over these bindings, plus some tools to help us manage the interpreter’s value stack correctly.

mod sys;

use std::{ffi::CStr, io::{Write, stdout}, ptr::null_mut};

use crate::sys::*;

fn main() {
    struct Author {
        name: String,
        link_url: String,
        avatar_url: String,
        email_address: String,
        phone_number: String,
        pronouns: String,
        bio: String,
    }

    let author = Author {
        name: "shuppy".to_owned(),
        link_url: "https://shuppy.org".to_owned(),
        avatar_url: "https://shuppy.org/avatar.png".to_owned(),
        email_address: "test@shuppy.org".to_owned(),
        phone_number: "+61 491 570 006".to_owned(),
        pronouns: "she/her".to_owned(),
        bio: "
            Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor
            incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud
            exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure
            dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
            Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt
            mollit anim id est laborum.
        ".repeat(100),
    };

    unsafe {
        let ctx = duk_create_heap(None, None, None, null_mut(), None);
        duk_push_c_function(ctx, Some(print), DUK_VARARGS);
        duk_put_global_string(ctx, c"print".as_ptr());

        // run the user’s custom code
        duk_eval_string(ctx, cr#"
            function linkify_author(author) {
                return "[" + author.name + "](" + author.link_url + ")";
            }
        "#.as_ptr());
        duk_pop(ctx);

        duk_eval_string(ctx, cr#"
            function Author(inner) {
                this.inner = inner;
            }
        "#.as_ptr());
        duk_pop(ctx);

        // `for (key in author) Object.defineProperty(Author.prototype, key, {get})`
        duk_eval_string(ctx, c"Author.prototype".as_ptr());
        macro_rules! define_author_getter {
            ($ctx:ident, $key:literal, |author| author.$field:ident) => {{
                duk_push_string($ctx, $key.as_ptr());
                duk_push_c_function($ctx, Some(get), 1);
                duk_def_prop($ctx, -3, DUK_DEFPROP_HAVE_GETTER);
                extern "C" fn get(ctx: *mut duk_context) -> duk_ret_t {
                    unsafe {
                        duk_push_this(ctx);
                        duk_get_prop_string(ctx, -1, c"inner".as_ptr());
                        let author = duk_get_pointer(ctx, -1) as *const Author;
                        duk_pop_2(ctx);
                        let result = &(*author).$field;
                        duk_push_lstring(ctx, result.as_ptr() as *const _, result.len());
                    }
                    1
                }
            }};
        }
        define_author_getter!(ctx, c"name", |author| author.name);
        define_author_getter!(ctx, c"link_url", |author| author.link_url);
        define_author_getter!(ctx, c"avatar_url", |author| author.avatar_url);
        define_author_getter!(ctx, c"email_address", |author| author.email_address);
        define_author_getter!(ctx, c"phone_number", |author| author.phone_number);
        define_author_getter!(ctx, c"pronouns", |author| author.pronouns);
        define_author_getter!(ctx, c"bio", |author| author.bio);
        duk_pop(ctx);

        // `push(new Author(rust author))`
        duk_get_global_string(ctx, c"Author".as_ptr());
        duk_push_pointer(ctx, &author as *const _ /* 🗞️ bad */ as *mut _);
        duk_new(ctx, 1);

        // `push(linkify_author(rust author))`
        duk_get_global_string(ctx, c"linkify_author".as_ptr());
        duk_pull(ctx, -2);
        duk_call(ctx, 1);

        // `result = pop()`
        println!("{:?}", CStr::from_ptr(duk_get_string(ctx, -1)));
        duk_pop(ctx);

        duk_destroy_heap(ctx);
    }
}

extern "C" fn print(ctx: *mut duk_context) -> duk_ret_t {
    unsafe {
        duk_push_string(ctx, c" ".as_ptr());
        duk_insert(ctx, 0);
        duk_join(ctx, duk_get_top(ctx) - 1);
        let text = duk_safe_to_string(ctx, -1);
        let text = CStr::from_ptr(text);
        let _ = stdout().write_all(text.to_bytes());
    }
    0
}

#![cfg_attr(feature = "bench", feature(test))]
#[cfg(feature = "bench")]
extern crate test;

#[cfg(feature = "bench")]
#[bench]
fn empty_object(bencher: &mut test::bench::Bencher) {
    struct Author {
        name: String,
        link_url: String,
        avatar_url: String,
        email_address: String,
        phone_number: String,
        pronouns: String,
        bio: String,
    }

    let author = Author {
        name: "shuppy".to_owned(),
        link_url: "https://shuppy.org".to_owned(),
        avatar_url: "https://shuppy.org/avatar.png".to_owned(),
        email_address: "test@shuppy.org".to_owned(),
        phone_number: "+61 491 570 006".to_owned(),
        pronouns: "she/her".to_owned(),
        bio: "
            Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor
            incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud
            exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure
            dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
            Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt
            mollit anim id est laborum.
        ".repeat(100),
    };

    unsafe {
        let ctx = duk_create_heap(None, None, None, null_mut(), None);
        duk_push_c_function(ctx, Some(print), DUK_VARARGS);
        duk_put_global_string(ctx, c"print".as_ptr());

        // run the user’s custom code
        duk_eval_string(ctx, cr#"
            function linkify_author(author) {
                return "[" + author.name + "](" + author.link_url + ")";
            }
        "#.as_ptr());
        duk_pop(ctx);

        bencher.iter(|| {
            // `push({})`
            duk_push_object(ctx);

            // `push(linkify_author(rust author))`
            duk_get_global_string(ctx, c"linkify_author".as_ptr());
            duk_pull(ctx, -2);
            duk_call(ctx, 1);

            // `result = pop()`
            let result = CStr::from_ptr(duk_get_string(ctx, -1));
            duk_pop(ctx);
            result
        });

        duk_destroy_heap(ctx);
    }
}

#[cfg(feature = "bench")]
#[bench]
fn obvious_approach(bencher: &mut test::bench::Bencher) {
    struct Author {
        name: String,
        link_url: String,
        avatar_url: String,
        email_address: String,
        phone_number: String,
        pronouns: String,
        bio: String,
    }

    let author = Author {
        name: "shuppy".to_owned(),
        link_url: "https://shuppy.org".to_owned(),
        avatar_url: "https://shuppy.org/avatar.png".to_owned(),
        email_address: "test@shuppy.org".to_owned(),
        phone_number: "+61 491 570 006".to_owned(),
        pronouns: "she/her".to_owned(),
        bio: "
            Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor
            incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud
            exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure
            dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
            Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt
            mollit anim id est laborum.
        ".repeat(100),
    };

    unsafe {
        let ctx = duk_create_heap(None, None, None, null_mut(), None);
        duk_push_c_function(ctx, Some(print), DUK_VARARGS);
        duk_put_global_string(ctx, c"print".as_ptr());

        // run the user’s custom code
        duk_eval_string(ctx, cr#"
            function linkify_author(author) {
                return "[" + author.name + "](" + author.link_url + ")";
            }
        "#.as_ptr());
        duk_pop(ctx);

        bencher.iter(|| {
            // `author = {};`
            // `for (key in author) author[key] = convert rust author[key];`
            // `push(author)`
            duk_push_object(ctx);
            duk_push_lstring(ctx, author.name.as_ptr() as *const _, author.name.len());
            duk_put_prop_string(ctx, -2, c"name".as_ptr());
            duk_push_lstring(ctx, author.link_url.as_ptr() as *const _, author.link_url.len());
            duk_put_prop_string(ctx, -2, c"link_url".as_ptr());
            duk_push_lstring(ctx, author.avatar_url.as_ptr() as *const _, author.avatar_url.len());
            duk_put_prop_string(ctx, -2, c"avatar_url".as_ptr());
            duk_push_lstring(ctx, author.email_address.as_ptr() as *const _, author.email_address.len());
            duk_put_prop_string(ctx, -2, c"email_address".as_ptr());
            duk_push_lstring(ctx, author.phone_number.as_ptr() as *const _, author.phone_number.len());
            duk_put_prop_string(ctx, -2, c"phone_number".as_ptr());
            duk_push_lstring(ctx, author.pronouns.as_ptr() as *const _, author.pronouns.len());
            duk_put_prop_string(ctx, -2, c"pronouns".as_ptr());
            duk_push_lstring(ctx, author.bio.as_ptr() as *const _, author.bio.len());
            duk_put_prop_string(ctx, -2, c"bio".as_ptr());

            // `push(linkify_author(rust author))`
            duk_get_global_string(ctx, c"linkify_author".as_ptr());
            duk_pull(ctx, -2);
            duk_call(ctx, 1);

            // `result = pop()`
            let result = CStr::from_ptr(duk_get_string(ctx, -1));
            duk_pop(ctx);
            result
        });

        duk_destroy_heap(ctx);
    }
}

#[cfg(feature = "bench")]
#[bench]
fn with_getters(bencher: &mut test::bench::Bencher) {
    struct Author {
        name: String,
        link_url: String,
        avatar_url: String,
        email_address: String,
        phone_number: String,
        pronouns: String,
        bio: String,
    }

    let author = Author {
        name: "shuppy".to_owned(),
        link_url: "https://shuppy.org".to_owned(),
        avatar_url: "https://shuppy.org/avatar.png".to_owned(),
        email_address: "test@shuppy.org".to_owned(),
        phone_number: "+61 491 570 006".to_owned(),
        pronouns: "she/her".to_owned(),
        bio: "
            Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor
            incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud
            exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure
            dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
            Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt
            mollit anim id est laborum.
        ".repeat(100),
    };

    unsafe {
        let ctx = duk_create_heap(None, None, None, null_mut(), None);
        duk_push_c_function(ctx, Some(print), DUK_VARARGS);
        duk_put_global_string(ctx, c"print".as_ptr());

        // run the user’s custom code
        duk_eval_string(ctx, cr#"
            function linkify_author(author) {
                return "[" + author.name + "](" + author.link_url + ")";
            }
        "#.as_ptr());
        duk_pop(ctx);

        bencher.iter(|| {
            // `author = {inner: rust author};`
            // `for (key in author) Object.defineProperty(author, key, {get});`
            // `push(author)`
            duk_push_object(ctx);
            duk_push_pointer(ctx, &author as *const _ /* 🗞️ bad */ as *mut _);
            duk_put_prop_string(ctx, -2, c"inner".as_ptr());
            macro_rules! define_author_getter {
                ($ctx:ident, $key:literal, |author| author.$field:ident) => {{
                    duk_push_string($ctx, $key.as_ptr());
                    duk_push_c_function($ctx, Some(get), 1);
                    duk_def_prop($ctx, -3, DUK_DEFPROP_HAVE_GETTER);
                    extern "C" fn get(ctx: *mut duk_context) -> duk_ret_t {
                        unsafe {
                            duk_push_this(ctx);
                            duk_get_prop_string(ctx, -1, c"inner".as_ptr());
                            let author = duk_get_pointer(ctx, -1) as *const Author;
                            duk_pop_2(ctx);
                            let result = &(*author).$field;
                            duk_push_lstring(ctx, result.as_ptr() as *const _, result.len());
                        }
                        1
                    }
                }};
            }
            define_author_getter!(ctx, c"name", |author| author.name);
            define_author_getter!(ctx, c"link_url", |author| author.link_url);
            define_author_getter!(ctx, c"avatar_url", |author| author.avatar_url);
            define_author_getter!(ctx, c"email_address", |author| author.email_address);
            define_author_getter!(ctx, c"phone_number", |author| author.phone_number);
            define_author_getter!(ctx, c"pronouns", |author| author.pronouns);
            define_author_getter!(ctx, c"bio", |author| author.bio);

            // `push(linkify_author(rust author))`
            duk_get_global_string(ctx, c"linkify_author".as_ptr());
            duk_pull(ctx, -2);
            duk_call(ctx, 1);

            // `result = pop()`
            let result = CStr::from_ptr(duk_get_string(ctx, -1));
            duk_pop(ctx);
            result
        });

        duk_destroy_heap(ctx);
    }
}

#[cfg(feature = "bench")]
#[bench]
fn with_prototype(bencher: &mut test::bench::Bencher) {
    struct Author {
        name: String,
        link_url: String,
        avatar_url: String,
        email_address: String,
        phone_number: String,
        pronouns: String,
        bio: String,
    }

    let author = Author {
        name: "shuppy".to_owned(),
        link_url: "https://shuppy.org".to_owned(),
        avatar_url: "https://shuppy.org/avatar.png".to_owned(),
        email_address: "test@shuppy.org".to_owned(),
        phone_number: "+61 491 570 006".to_owned(),
        pronouns: "she/her".to_owned(),
        bio: "
            Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor
            incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud
            exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure
            dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
            Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt
            mollit anim id est laborum.
        ".repeat(100),
    };

    unsafe {
        let ctx = duk_create_heap(None, None, None, null_mut(), None);
        duk_push_c_function(ctx, Some(print), DUK_VARARGS);
        duk_put_global_string(ctx, c"print".as_ptr());

        // run the user’s custom code
        duk_eval_string(ctx, cr#"
            function linkify_author(author) {
                return "[" + author.name + "](" + author.link_url + ")";
            }
        "#.as_ptr());
        duk_pop(ctx);

        duk_eval_string(ctx, cr#"
            function Author(inner) {
                this.inner = inner;
            }
        "#.as_ptr());
        duk_pop(ctx);

        // `for (key in author) Object.defineProperty(Author.prototype, key, {get})`
        duk_eval_string(ctx, c"Author.prototype".as_ptr());
        macro_rules! define_author_getter {
            ($ctx:ident, $key:literal, |author| author.$field:ident) => {{
                duk_push_string($ctx, $key.as_ptr());
                duk_push_c_function($ctx, Some(get), 1);
                duk_def_prop($ctx, -3, DUK_DEFPROP_HAVE_GETTER);
                extern "C" fn get(ctx: *mut duk_context) -> duk_ret_t {
                    unsafe {
                        duk_push_this(ctx);
                        duk_get_prop_string(ctx, -1, c"inner".as_ptr());
                        let author = duk_get_pointer(ctx, -1) as *const Author;
                        duk_pop_2(ctx);
                        let result = &(*author).$field;
                        duk_push_lstring(ctx, result.as_ptr() as *const _, result.len());
                    }
                    1
                }
            }};
        }
        define_author_getter!(ctx, c"name", |author| author.name);
        define_author_getter!(ctx, c"link_url", |author| author.link_url);
        define_author_getter!(ctx, c"avatar_url", |author| author.avatar_url);
        define_author_getter!(ctx, c"email_address", |author| author.email_address);
        define_author_getter!(ctx, c"bio", |author| author.bio);
        duk_pop(ctx);

        bencher.iter(|| {
            // `push(new Author(rust author))`
            duk_get_global_string(ctx, c"Author".as_ptr());
            duk_push_pointer(ctx, &author as *const _ /* 🗞️ bad */ as *mut _);
            duk_new(ctx, 1);

            // `push(linkify_author(rust author))`
            duk_get_global_string(ctx, c"linkify_author".as_ptr());
            duk_pull(ctx, -2);
            duk_call(ctx, 1);

            // `result = pop()`
            let result = CStr::from_ptr(duk_get_string(ctx, -1));
            duk_pop(ctx);
            result
        });

        duk_destroy_heap(ctx);
    }
}

say you write a program in rust, with all of the benefits that entails (performance, type safety, etc), but you want to make parts of it programmable by the user without recompiling. you want to embed some kind of interpreted language, like lua or javascript, but surely that would be a pain? it’s not as bad as you might think!

which language? which runtime?

lua is famously embeddable, but i honestly don’t enjoy writing lua. that might be because i haven’t done it much. javascript is also embeddable, and after twenty years of writing javascript, i find it… tolerable. so let’s go with javascript.

there are many, many javascript runtimes out there. V8 and spidermonkey are cutting-edge options, very fast, but also very complex.

spidermonkey is what firefox and servo use, and it even has rust bindings, so you might think this is what i would pick. after all, servo is my job, and i’ve used html5ever and cssparser in hobby projects, and they’re great!

but i’ve never embedded an interpreter before, and the five months i had working on something adjacent to spidermonkey in servo tells me that a newbie like me should cut my teeth on something simpler.

enter: duktape

((o) duktape is a javascript runtime that’s about as easy to embed as it gets: one .c file, one .h file, and a straightforward C API. but there are some big drawbacks:

• it only really supports ES5. this is (or was) a common target for compilers like babel, but if you’re writing it by hand, that means you can use things like bind() on Function and map() on Array, but not things like const or arrow functions.

• it has no JIT, so it’s much, much slower than V8 and spidermonkey.

xkcd standards but it’s crates.io

i searched for “duktape” on crates.io and found 25 packages, most of which appeared to be unmaintained or immature. one even admits:

Aren't there about a dozen of these crates?
Yes. They all seem to address slightly different needs so I've added to the pile. Sorry.

so we’re shit outta luck right? not necessarily! we can roll our own duktape bindings without an unreasonable amount of effort. let’s do that here, in this post!

the code below is compatible with duktape 2.7.0.

1. low-level bindings

let’s create a rust program first, then grab a copy of duktape. while the duktape releases include the actual sources (not all in a single .c and .h) and build scripts we can use to create a customised flavour of duktape, we can just use the default flavour in src this time:

$ cargo new duktest
$ cd duktest
$ curl -O https://duktape.org/duktape-2.7.0.tar.xz
$ tar xf duktape-2.7.0.tar.xz
$ ls duktape-2.7.0/src
duk_config.h  duk_source_meta.json  duktape.c  duktape.h

we’re gonna use bindgen to generate most of our low-level bindings.

[build-dependencies]
bindgen = "0.72.1"

bindgen will use libclang to parse duktape.h. with nix, that looks like this:

with import <nixpkgs> {};
(mkShell.override { stdenv = clangStdenv; }) {
  name = "duktest-shell";
  LIBCLANG_PATH = lib.makeLibraryPath [ llvmPackages.clang-unwrapped.lib ];
}

the build script is almost entirely based on this example in the bindgen docs, but one important thing is to tell bindgen to only generate bindings for duktape.h, not a bunch of other standard library headers that will just cause us problems:

// `error[E0428]: the name `FP_NAN` is defined multiple times`
// `error[E0428]: the name `FP_INFINITE` is defined multiple times`
// `error[E0428]: the name `FP_ZERO` is defined multiple times`
// `error[E0428]: the name `FP_SUBNORMAL` is defined multiple times`
// `error[E0428]: the name `FP_NORMAL` is defined multiple times`
// `warning: `extern` block uses type `u128`, which is not FFI-safe`
.allowlist_file(".*/duktape[.]h")

use std::env;
use std::path::PathBuf;

fn main() {
    let libdir_path = PathBuf::from("duktape-2.7.0/src")
        .canonicalize()
        .expect("cannot canonicalize path");

    let h_path = libdir_path.join("duktape.h");
    let c_path = libdir_path.join("duktape.c");
    let obj_path = libdir_path.join("duktape.o");
    let lib_path = libdir_path.join("libduktape.a");
    let (h_path, c_path, obj_path, lib_path) = (
        h_path.to_str().expect("unsupported path"),
        c_path.to_str().expect("unsupported path"),
        obj_path.to_str().expect("unsupported path"),
        lib_path.to_str().expect("unsupported path"),
    );

    // Tell cargo to look for shared libraries in the specified directory
    println!("cargo:rustc-link-search={}", libdir_path.to_str().unwrap());

    // Tell cargo to tell rustc to link our `duktape` library. Cargo will
    // automatically know it must look for a `libduktape.a` file.
    println!("cargo:rustc-link-lib=duktape");

    // Run `clang` to compile the `duktape.c` file into a `duktape.o` object file.
    if !std::process::Command::new("clang")
        .args(["-c", "-o", obj_path, c_path])
        .output()
        .expect("failed to spawn `clang`")
        .status
        .success()
    {
        panic!("failed to compile object file");
    }

    // Run `ar` to generate the `libhello.a` file from the `hello.o` file.
    if !std::process::Command::new("ar")
        .args(["rcs", lib_path, obj_path])
        .output()
        .expect("failed to spawn `ar`")
        .status
        .success()
    {
        panic!("failed to emit library file");
    }

    let bindings = bindgen::Builder::default()
        // `error[E0428]: the name `FP_NAN` is defined multiple times`
        // `error[E0428]: the name `FP_INFINITE` is defined multiple times`
        // `error[E0428]: the name `FP_ZERO` is defined multiple times`
        // `error[E0428]: the name `FP_SUBNORMAL` is defined multiple times`
        // `error[E0428]: the name `FP_NORMAL` is defined multiple times`
        // `warning: `extern` block uses type `u128`, which is not FFI-safe`
        .allowlist_file(".*/duktape[.]h")
        .header(h_path)
        // Tell cargo to invalidate the built crate whenever any of the
        // included header files changed.
        .parse_callbacks(Box::new(bindgen::CargoCallbacks::new()))
        .generate()
        .expect("failed to generate bindings");

    // Write the bindings to the $OUT_DIR/bindings.rs file.
    let out_path = PathBuf::from(env::var("OUT_DIR").unwrap());
    bindings
        .write_to_file(out_path.join("bindings.rs"))
        .expect("failed to write bindings!");
}

now we pull in those bindings from $OUT_DIR/bindings.rs into a module:

#![allow(unused)]
#![allow(nonstandard_style)]
include!(concat!(env!("OUT_DIR"), "/bindings.rs"));

mod sys;

2. missing pieces

bindgen is great, but there are some things it just can’t do. for these things, we’ll have to write the bindings for by hand, translating from C to rust.

a big one is function-like macros. duktape uses a modest amount of these, thankfully not too many as of version 2.7.0, but they also reserve the right to change any API to a macro (or back to a real function) in any release.

so far, i’ve had to translate duk_eval, duk_eval_noresult, duk_peval, duk_peval_noresult, duk_eval_string, duk_peval_string, duk_eval_lstring, duk_peval_lstring, and duk_safe_to_string. here are a few of those:

#define duk_eval_string(ctx,src)  \
    ((void) duk_eval_raw((ctx), (src), 0, 0 /*args*/ | DUK_COMPILE_EVAL | DUK_COMPILE_NOSOURCE | DUK_COMPILE_STRLEN | DUK_COMPILE_NOFILENAME))

#define duk_peval_lstring(ctx,buf,len)  \
    (duk_eval_raw((ctx), buf, len, 0 /*args*/ | DUK_COMPILE_EVAL | DUK_COMPILE_NOSOURCE | DUK_COMPILE_SAFE | DUK_COMPILE_NOFILENAME))

#define duk_safe_to_string(ctx,idx) \
    duk_safe_to_lstring((ctx), (idx), NULL)

use std::{ffi::c_char, ptr::null_mut};

pub unsafe fn duk_eval_string(ctx: *mut duk_context, src: *const c_char) {
    unsafe {
        duk_eval_raw(
            ctx,
            src,
            0,
            0 /*args*/ | DUK_COMPILE_EVAL | DUK_COMPILE_NOSOURCE | DUK_COMPILE_STRLEN | DUK_COMPILE_NOFILENAME,
        );
    }
}

pub unsafe fn duk_peval_lstring(ctx: *mut duk_context, buf: *const c_char, len: duk_size_t) -> duk_int_t {
    unsafe {
        duk_eval_raw(ctx, buf, len, 0 /*args*/ | DUK_COMPILE_EVAL | DUK_COMPILE_NOSOURCE | DUK_COMPILE_SAFE | DUK_COMPILE_NOFILENAME)
    }
}

pub unsafe fn duk_safe_to_string(ctx: *mut duk_context, idx: duk_idx_t) -> *const c_char {
    unsafe {
        duk_safe_to_lstring(ctx, idx, null_mut())
    }
}

bindgen does support simple preprocessor constants, but for reasons unclear to me, not all of them make it through. the only one i’ve run into so far is DUK_VARARGS:

/* Indicates that a native function does not have a fixed number of args,
 * and the argument stack should not be capped/extended at all.
 */
#define DUK_VARARGS                       ((duk_int_t) (-1))

pub const DUK_VARARGS: duk_int_t = -1;

3. hello world

with this, we have enough to start running some javascript!

mod sys;

use std::{ffi::CStr, io::{Write, stdout}, ptr::null_mut};

use crate::sys::*;

fn main() {
    unsafe {
        let ctx = duk_create_heap(None, None, None, null_mut(), None);
        duk_push_c_function(ctx, Some(nop), DUK_VARARGS);
        duk_put_global_string(ctx, c"nop".as_ptr());
        duk_push_c_function(ctx, Some(print), DUK_VARARGS);
        duk_put_global_string(ctx, c"print".as_ptr());

        duk_eval_string(ctx, c"nop()".as_ptr());
        duk_eval_string(ctx, cr#"print("hello js world\n")"#.as_ptr());

        duk_pop(ctx);
        duk_destroy_heap(ctx);

        println!("goodbye rust world");
    }
}

extern "C" fn nop(_: *mut duk_context) -> duk_ret_t {
    0
}

extern "C" fn print(ctx: *mut duk_context) -> duk_ret_t {
    unsafe {
        duk_push_string(ctx, c" ".as_ptr());
        duk_insert(ctx, 0);
        duk_join(ctx, duk_get_top(ctx) - 1);
        let text = duk_safe_to_string(ctx, -1);
        let text = CStr::from_ptr(text);
        let _ = stdout().write_all(text.to_bytes());
    }
    0
}

$ cargo run
hello js world
goodbye rust world

stay tuned!

next time, we’ll look at building some abstractions over these bindings, plus some tools to help us manage the interpreter’s value stack correctly.

how do we add caching to a static site generator? quoth last episode:

we need a way to know if a post has changed, or any of its attachments have changed, since it was last rendered or its metadata was last cached, and we also need to know what tag pages will need to be rerendered.

after a bunch of experimentation, i think nix is pretty much what we want, but nix doesn’t really work on windows, so it can’t be nix, so let’s build our own nix :3

disclaimer again

none of this has landed on the main branch yet, for good reason. it’s probably buggy as shit, so don’t run it on your real blog yet.

juicy numbers

in the autost#54 patch as of 218670e77f08e:

this blog (4803 threads)

this blog is my cohost archive plus a couple hundred newer posts. that’s 4803 posts (9816 if you count posts referenced by replies), and ~4.7 GB of attachments.

$ export RUST_LOG=autost=warn
$ time autost render --skip-attachments
9.36s user 1.30s system 1287% cpu 0.828 total
$ time autost render --skip-attachments --use-cache
10.26s user 3.97s system 821% cpu 1.731 total
$ time autost render --skip-attachments --use-cache
1.02s user 1.53s system 661% cpu 0.385 total

$ unset RUST_LOG
$ time autost render --skip-attachments --use-cache
INFO build{function=ReadFile id=b0143e6e233ed...}: autost::cache: building
INFO build{function=RenderMarkdown id=e0c542ce051e2...}: autost::cache: building
INFO build{function=FilteredPost id=5ad850559c540...}: autost::cache: building
INFO build{function=Thread id=799f2c033be6a...}: autost::cache: building
INFO build{function=RenderedThread id=844c6195470bd...}: autost::cache: building
INFO build{function=Thread id=9dda7da0c21ae...}: autost::cache: building
INFO build{function=RenderedThread id=eac9c7f17c6f8...}: autost::cache: building
INFO autost::cache: writing cache pack 5ad
INFO autost::cache: writing cache pack 9dd
INFO autost::cache: writing cache pack 799
INFO autost::cache: writing cache pack e0c
INFO autost::cache: writing cache pack eac
INFO autost::cache: writing cache pack 844
INFO autost::cache: writing cache pack b01
1.27s user 1.72s system 681% cpu 0.438 total

$ unset RUST_LOG
$ time autost cache test --list-threads-in-tag usb3sun --use-cache
37 threads in tag "usb3sun":
- "2023-01-06T18:08:35.092Z", "posts/787278.html", "SPARCstations have a unique serial-like interface "
- "2023-06-23T17:15:54.361Z", "posts/1742287.html", "usb3sun is an adapter that lets you connect usb ke"
- "2023-06-24T17:50:23.538Z", "posts/1650431.html", "SPARCstations have a unique serial-like interface "
[...]
INFO autost::cache: writing cache pack 372
0.49s user 0.93s system 410% cpu 0.347 total

$ rm -R cache
$ export RUST_LOG=autost=warn
$ time autost cache test --list-threads-in-tag usb3sun
1.77s user 0.27s system 864% cpu 0.236 total
$ time autost cache test --list-threads-in-tag usb3sun --use-cache
2.39s user 2.77s system 493% cpu 1.045 total
$ time autost cache test --list-threads-in-tag usb3sun --use-cache
0.07s user 0.30s system 412% cpu 0.090 total

big archive (146752 threads)

ok but what if i merged the cohost archives of everyone i followed into one very big archive? that’s 146752 posts (309140 if you count posts referenced by replies), and ~64 GB of attachments.

$ export RUST_LOG=autost=warn
$ time autost render --skip-attachments
258.96s user 49.31s system 1237% cpu 24.902 total
$ time autost render --skip-attachments --use-cache
309.85s user 72.00s system 1067% cpu 35.755 total
$ time autost render --skip-attachments --use-cache
33.57s user 44.87s system 701% cpu 11.188 total

$ unset RUST_LOG
$ time autost cache test --list-threads-in-tag usb3sun
37 threads in tag "usb3sun":
- "2023-01-06T18:08:35.092Z", "posts/787278.html", "SPARCstations have a unique serial-like interface "
- "2023-06-23T17:15:54.361Z", "posts/1742287.html", "usb3sun is an adapter that lets you connect usb ke"
- "2023-06-24T17:50:23.538Z", "posts/1650431.html", "SPARCstations have a unique serial-like interface "
[...]
2025-08-28T09:39:27.095295Z  INFO autost::cache: writing cache pack 857
14.77s user 18.37s system 381% cpu 8.692 total

$ rm -R cache
$ export RUST_LOG=autost=warn
$ time autost cache test --list-threads-in-tag usb3sun
34.12s user 5.86s system 720% cpu 5.551 total
$ time autost cache test --list-threads-in-tag usb3sun --use-cache
57.26s user 32.47s system 768% cpu 11.673 total
$ time autost cache test --list-threads-in-tag usb3sun --use-cache
1.47s user 5.72s system 464% cpu 1.547 total

anyway! i repeat, how do we add caching to a static site generator?

bad answer: timestamps

make is an incremental build system, where “incremental” means it only rebuilds things that have changed. it does this with file timestamps and a dependency graph: a file needs to be rebuilt if any of its dependencies are newer than it.

this is inadequate, because you can easily change the timestamp without changing the contents or change the contents without changing the timestamp. and good luck if you want to simultaneously cache more than one version of a build product.

this is also unnecessarily frugal, except for attachments, as we’ll see shortly.

better answer: hashes

nowadays reading files is pretty fast, as long as they’re stored on an ssd. let’s write a little microbenchmark to see just how fast it can be. all of these results were taken on my framework 13 with an AMD 7840U and a charger connected.

$ autost cache benchmark <posts|posts-recursive|attachments> sum-paths-len <10..=100>
$ autost cache benchmark <posts|posts-recursive|attachments> sum-read-len <10..=100>
$ autost cache benchmark <posts|posts-recursive|attachments> blake3 <10..=100>
$ autost cache benchmark <posts|posts-recursive|attachments> blake3-mmap-rayon <10..=100>

this blog (4803 threads)

it looks like reading and hashing all of the posts doesn’t even double the time taken to walk the tree, but doing all of the attachments would incur an over 30x time penalty.

big archive (146752 threads)

again it’s less than a 2x penalty if we read and hash the posts, but an over 30x penalty for attachments.

another way of looking at it is, if we’re gonna have to read a bunch of files, we might as well hash them so we can easily check if they’ve changed. again this doesn’t make sense for attachments, whose contents are never actually read by autost render. i’m not sure i have a good solution for attachments yet.

best answer: nix?

rendering your site involves many steps:

• making a list of posts
• rendering the posts
  • reading the post sources
  • for posts in markdown, rendering the markdown
  • parsing the post html into a dom tree
  • extracting metadata
    • and rendering the referenced posts (if any)
    • and scanning for references to attachments
  • applying transformations (e.g. making all images lazy loaded)
  • serialising the dom tree back to html
  • sanitising the html to “safe” tags and attributes
• writing the output files
  • copying the static files (e.g. style.css, script.js)
  • hard linking the referenced attachments
  • writing a page for each thread, containing that thread
  • writing a page for each tag, containing all of its threads
  • writing an atom feed for each tag, containing all of its threads
  • writing a page for each built-in collection (e.g. index.html)

we don’t just want to cache the final build outputs (html pages and atom feeds), because the intermediate build steps are also useful for things like querying metadata. so how do we build a cache for all of the different kinds of build steps without descending into cache invalidation and ad-hoc serialisation hell?

many of these steps are pretty much functions (in the mathematical sense) that take some input and transform it to some output. in fact, all of them are, although some of them are easier to describe that way than others.

nix is a build system and package manager that uses this observation to build the largest ever single repository of software with efficient binary caching. how can we apply its ideas here?

imagine the process of loading a thread, just enough to know its metadata like its tags and what attachments it references:

• for the last post in the thread
  • read the post sources
  • if the post is in markdown, render the markdown
  • parse the html and extract metadata
• repeat for each of the posts it references

now let’s say we described each of these steps as a function. note that for the caching to work correctly, it’s very important that aside from readFile(), all of the other functions are pure. they can only use their input arguments, which is why makeThread() can’t take one post and load all of the other posts it references from disk.

• readFile(path, hash) → content reads path with expected hash hash, and returns the content of that file with that hash (more on the hash later)
• renderMarkdown(markdown) → html renders the given markdown, and returns the rendered html
• loadPost(html) → post (html, metadata) parses the given html into a dom tree, extracts the metadata, and returns a post
• makeThread(posts…) → thread (posts…) takes the given posts, combines them into a thread, and returns that thread

each of those functions just Builds The Thing. to make them cacheable and allow us to run them with maximum parallelism, we want to be able to make a “build plan” that completely describes how to Build The Thing without doing all of the work of actually Building The Thing.

let’s describe the build planning as a second set of functions:

• readFilePlan(path) → “readFile(path, hash)”
• renderMarkdownPlan(path) → “renderMarkdown(readFile(path, hash))”
• loadPostPlan(path) →
  • | “loadPost(renderMarkdown(readFile(path, hash))” if markdown
  • | “loadPost(readFile(path, hash))” if html
• makeThreadPlan(path) → “makeThread(loadPost(…), loadPost(…), …)”

one problem we run into with makeThreadPlan() is that we need the metadata of one of the posts to know what other loadPost() calls to include in the result. this is unfortunate, because it means we can’t entirely avoid Building The Thing when doing build planning.

now here’s the magic!

let’s say we’re building the thread for 10000216.md, which replies to 10000215.md. makeThreadPlan(10000216.md) returns the build plan below. it was relatively easy to compute, and it completely describes how to load the thread from the files that make up the thread.

“makeThread(loadPost(renderMarkdown(readFile(10000216.md, 44062ae08bb67…))), loadPost(renderMarkdown(readFile(10000215.md, 8adba2770e58c…))))”

remember the “(more on hash later)” from earlier? if the build plan for a thread includes the hash of each of its posts, then we can hash the build plan and get an ID that changes if and only if any of the posts have changed.

and with that kind of unique ID for each thing we build, caching is easy! since the contents of the build plan for a given ID can’t possibly change (by definition), and the contents of the output really shouldn’t change (if we wrote our build logic correctly), we never need to delete or update anything, except maybe to free up disk space.

to me, this is the essence of nix, and i had great fun capturing that essence in our funny little static site generator. if you wanna see how this ersatz nix works, check out src/cache.rs and src/cache/drv.rs ^w^

that’s the post. but if you’re hungry for even more detail, read on…

performant store

processing thousands or even hundreds of thousands of posts is hard work. some things that helped with cache storage bottlenecks:

• staring at hundreds of flamegraphs
• switching from sqlite to plain old files
• switching from json to bincode
• switching from tokio async to rayon sync
  • dedicated rayon thread pools for writing files with 4x the threads, since they will spend most of their time in syscalls waiting for i/o
  • explicitly creating a thread pool for normal work, since rayon otherwise just uses the thread pool of the innermost fork-join scope
• caching build products in memory, then writing that to disk
  • storing native rust types in the memory cache, so we can take build product serialisation off the critical path
  • offloading writes to dedicated thread pools, so we can take build product file writing off the critical path
• moving from one file per build product, to 4096 “packs” of cache data
  • splitting the memory cache into 4096 “packs” as well
  • splitting the dirty bit into 4096 dirty bits as well
• reducing overheads of syncing memory caches with the disk store
  • switching from DashMap to HashMap, because it’s cheaper when we have thousands of them, and still performs well enough when sharded
  • lazy deserialisation of cache items, loading as Vec<u8> until needed

tag indexes

autost cache test --list-threads-in-tag builds a “tag index” to more efficiently look up posts having the given tag, but it was hard to do this in a way that wasn’t slower than just computing the metadata from scratch. two things made it possible.

the build plans themselves were expensive, because for each evaluation we had to deserialise a big tree of inputs from ThreadDrv to FilteredPostDrv to RenderMarkdownDrv and finally down to ReadFileDrv. plus load their outputs into the memory cache for no reason. we fixed this by hiding everything other than ReadFileDrv from the build plan, inferring them within the TagIndexDrv builder. idk about this… it feels like cheating or subverting how nix is supposed to work?

the obvious representation of the tag index is a HashMap or BTreeMap, but this has to be deserialised in full on first use without any parallelism, which is a waste unless our query involves the whole dataset. instead we can build a tiny sqlite database and serialise it to Vec<u8>, then serialise that as the build output. since sqlite can obviously query a table without building a whole map in memory, this is much faster.

i’ve started working on autost again, with the hope i can tackle some of the problems i wrote about in april. so far, i’ve been cautiously experimenting with adding a database, which again, seems to be unavoidable if we want autost to be anything more than a Cohost Archive You Can Keep Posting To. but this can solve other problems too!

disclaimer

none of this has landed on the main branch yet, for good reason. it’s probably buggy as shit, so don’t run it on your real blog yet.

post ids

to publish a new post, we need to choose a new post id, and our current approach is pretty minimalist (derogatory):

1. let id = 10000000
2. try to create posts/{id}.md
3. if we can’t because it already exists:
   1. let id = id + 1
   2. go to step 2

this is obviously inefficient, because it gets slower and slower the more posts you’ve made, and it doesn’t even work correctly if your last post was a .html post.

we could cache the last post id in a text file, and that sounds easy at first, but storing data in text files gets complicated fast. now you have to worry about things like

• how to avoid data loss if a power outage happens while writing
• how to version and migrate the file format when you need to extend it
• how to make related changes to multiple files together or not at all

and all of these are solved problems if we use an actual database. here’s a simple sqlite database that solves our post id problems:

CREATE TABLE "post" (
    "post_id" INTEGER PRIMARY KEY AUTOINCREMENT  -- e.g. 10000250
    , "path" TEXT NOT NULL                       -- e.g. '10000250.md'
    , "rendered_path" TEXT NULL UNIQUE           -- e.g. '10000250.html'
);

since the "rendered_path" is UNIQUE, we can’t accidentally create a "path" = '10000250.md' and a "path" = '10000250.html' that both have "rendered_path" = '10000250.html'. and choosing the next post id so we can publish a new post is easy:

BEGIN;
-- no "post_id", so we generate it with AUTOINCREMENT.
-- empty "path", because we don’t know it yet.
INSERT INTO "post" ("path") VALUES ('');
-- now we check the “last insert id”, which is done outside sql.
-- let’s say it’s 10000250.
UPDATE "post" SET "path" = '10000250.md' WHERE "post_id" = 10000250;
COMMIT;

check out autost#53 for the complete patch.

caching

right now, answering questions like “what posts are tagged #birds?” takes about as long as rendering your whole site, because in both cases we have to read all of your posts, parse the html, and extract the metadata. and when you publish or edit a post, we have to render your whole site from scratch.

this is generally how static site generators work, and Computer Touchers like that because having your whole blog be a bunch of text files is (a) elegant and (b) works well with version control. but the average Computer Toucher also has one (1) blog post, titled “New Year, New Blog, or: How I Learned To Stop Worrying and Switch From Twelvety to Sext.js”, so we should take their opinions with a grain of salt.

static site generators do have other benefits though. they’ve got good data portability, and static sites are easy to host on any web server, even web servers that are free because Big Coding is using them as a loss leader.

so maybe we want something in between, where your post files are still the primary source of truth, but we cache the metadata that we extract from them in the database, or we cache the fact that the post has already been rendered.

that will leave us with one of the two hard problems in computer science: now we need a way to know if a post has changed, or any of its attachments have changed, since it was last rendered or its metadata was last cached, and we also need to know what tag pages will need to be rerendered. hmm. is this just nix?

autost was written in a month-long daze after cohost announced their shutdown. it started out as a way for me to archive my own chosts on shuppy.org. since the plan all along was for me to continue chosting from there, that came next. then it gained the ability to import posts from other blogs, archive the chosts of everyone you follow, and archive chosts liked by you and your friends.

buried deep within the core of autost is some gnarly logic that decides which posts should be rendered, which posts should be unlisted or publicly visible (“interesting” posts), and which tags should have their own tag pages (“interesting” tags).

the problems i wanted to solve initially were:

• i want to archive my own chosts and continue chosting on the same site
• i want to curate which of my archived chosts are publicly visible
• we need a way to store the posts that are being replied to

but the way i solved them was a bunch of hardcoded rules:

• posts you make with autost are publicly visible
  • in practice, this means posts authored by your [self_author] href
• your own archived chosts in “interesting” tags are public too
  • in practice, this means posts authored by your other_self_authors
  • it’s “your own” if you authored the last post in the thread
  • the “last post” does not include transparent shares
  • “interesting” tags are those linked in the top nav section
• specific archived chosts can be made public or unlisted
• posts in subdirectories are ignored except for rendering replies

there are so many problems with this model:

• you can’t put any posts in subdirectories
• you can’t opt out of the rules for archived chosts. autost cohost-archive hacks around this by setting the [self_author] href of each archive to a cohost url
• you can’t organise posts in any way other than tags. autost cohost-archive hacks around this by making a separate site for each project you follow
• there’s no clear way to know why a post exists, or what you want to do with it, other than this big pile of emergent behaviour
• since every site has to abide by the same rules, any changes to the rules could break existing sites or change how they behave

we need to move away from these hardcoded rules. i think it’s time to sit down and actually define what we want autost to be capable of, and design a data model around that. this will require changes to existing sites, but we can build automated migrations or migration tools to upgrade those sites.

when you render your site, we should ask questions like “do we render this post?”, “is this post publicly visible?”, and “which tags have tag pages?”, but let whoever created the post (autost server, autost cohost2autost, autost import, or you) decide the answers.

we’ll probably want to store those answers in a database. it’s all files right now, which is like a database, except harder to extend and harder to safely mix human changes and machine changes. having a database may make it easier to extend autost to do other things with archived chosts, like organise the chosts of everyone you follow into a single browsable archive, or generate a page of your liked chosts.

we may even want to move posts that only exist for rendering replies out of the posts directory and into a separate store, like the attachment store. this will require changes to the <link rel=references> in existing posts though, so it would need to be done with extra care.

this update (full changelog) includes several minor improvements that landed after the big cohost shutdown scramble (rip eggbug). most notably:

• all <img> elements are now lazy loaded, massively reducing the size of large threads pages
• you can now reblog posts from akkoma instances, including those on the website league

• autost server now lets you click [+] on any tag to post in it
• autost server is no longer broken on windows (@ar1a, #39)

so… real talk. a handful of you actually use (or are considering using) autost for more than just archiving your chosts, and i think that’s wonderful. but maybe at this point you’re wondering if the project’s days are numbered.

development has been pretty slow the past couple of months, because i fell in love with a fox, and i needed a break after the last crunch. but i don’t consider autost finished, and i intend to keep working on it, just at a healthier pace.

higher priorities include being able to upload images from the post composer and compose posts from another device. but longer term, i wonder if there’s a niche for autost to be an easier kind of blog engine.

most blogging solutions are either saas, complex self-hosted (wordpress), or require you to be a computer-toucher and have frontend knowledge (static site generators). what if you could just run an app on your computer and get an editor for your blog? what if you could compose posts, follow and reblog your friends, and deploy your blog to neocities or some other host, all without touching the command line?

if that interests you, let me know at autost@shuppy.org or on the league :)

only a couple days left until cohost goes down! why not archive the chosts you care about, so you can look back on them without relying on wayback machine?

cohost-dl will do your chosts plus your liked chosts, while autost will do both of those plus chosts by people you follow! here’s how you can use autost for that:

1. install autost

2. log in to cohost, press F12, then grab your “connect.sid” cookie (see screenshot or consult someone with stickers on their laptop)

3. if you own more than one page, switch to the one you want to archive

4. get archiving!

5. repeat from step 3 until you’ve archived all of your pages

please email me on autost@shuppy.org if you have any questions! good luck ^w^

autost is a cohost-compatible archive and blog engine, which i use right here on shuppy.org. for another autost blog, check out LotteMakesStuff dot PINK :D

your posts are now rebloggable, thanks to the microformats2 h-entry format! for the most consistent experience, be sure to set your [self_author] display_handle setting to a domain name.

you can now reblog posts with autost import. for example, to reblog Natalie’s post Reblogging posts with h-entry:

$ autost import https://nex-3.com/blog/reblogging-posts-with-h-entry/
2024-10-03T15:30:33.221087Z  INFO run_migrations: autost::migrations: hard linking attachments out of site/attachments
2024-10-03T15:30:33.411823Z  INFO autost::command::import: writing RelativePath { inner: "posts/imported/1.html", kind: Other }
2024-10-03T15:30:33.422458Z  INFO autost::command::import: click here to reply: http://[::1]:8420/posts/compose?reply_to=imported/1.html

you can now create attachments from local files with autost attach. for example:

$ autost attach ~/Downloads/atom{1,2}.png
2024-10-11T11:47:39.031943Z  INFO run_migrations: autost::migrations: hard linking attachments out of site/attachments
2024-10-11T11:47:39.137328Z  INFO autost::command::attach: created attachment: <attachments/d9c42006-07ca-4027-8abb-dfa670a7e637/atom1.png>
2024-10-11T11:47:39.137477Z  INFO autost::command::attach: created attachment: <attachments/887df41f-a1ce-4e66-8a9f-09ef1f6fedca/atom2.png>

warning: autost attach does not strip any exif data yet, so be careful not to leak your gps location in any photos you upload! use exiftool or a photo editor to check.

the atom output is also way better now! most importantly, threads with multiple posts are no longer an unreadable mess:

→

note that your subscribers will need to convince their clients to redownload your posts, or unsubscribe and resubscribe, to see these effects on existing posts.

subscribe here for updates, or email me on autost@shuppy.org if you have any questions. enjoy :3

i’ve started adding h-entry support to autost! still need to rewrite relative urls and cache embedded images, but i can already import and share/reply to posts:

$ autost import https://nex-3.com/blog/reblogging-posts-with-h-entry/
2024-10-03T15:30:33.221087Z  INFO run_migrations: autost::migrations: hard linking attachments out of site/attachments
2024-10-03T15:30:33.411823Z  INFO autost::command::import: writing RelativePath { inner: "posts/imported/1.html", kind: Other }
2024-10-03T15:30:33.422458Z  INFO autost::command::import: click here to reply: http://[::1]:8420/posts/compose?reply_to=imported/1.html

autost is a cohost-compatible archive and blog engine. it’s pretty much a static site generator that can import chosts, but i plan to eventually add a rss/atom feed reader and chronological timeline.

you no longer need to know how to use git or cargo to use it! and now you can write posts and reply to posts in the composer!

cohost is shutting down soon, so go here for updates or email me on autost@shuppy.org if you have any questions. enjoy :3

rebug for the morning crew! please let me know if autost works or doesn't work for you, and go here for updates ^w^

some big usability improvements in this one, plus a couple of breaking changes!

breaking changes: cohost2autost no longer takes the ./posts and site/attachments arguments, and render no longer takes the site argument. these paths were always ./posts, site/attachments, and site (#8).

when viewing your site with the server, there’s a compose button now (#7).

in the server, you also get a link to your site in the terminal now, and you no longer get a NotFound error if your base_url setting is not /.

you no longer have to type RUST_LOG=info to see what’s happening (#5).

attachment filenames containing % and ? are now handled correctly (#4).

questions and contributions are welcome! if you run into any problems, or want to help out, let me know on github or email me on autost@shuppy.org.

oh and go here for updates once cohost noposts :)

…and hopefully feed reader?

want to archive your chosts on your website but have too many for the cohost web component? want something like cohost-dl except you can keep posting?

autost is my take on that. you can see what it looks like on shuppy.org:

it’s pretty much a static site generator that can import chosts. some features:

• perfect html rendering of your chosts and css crimes
  (css things like bounce and spin work too, but not yet 100%)
• include and exclude specific tags or chosts in your archive
• tag your archived chosts without editing them on cohost
• automatically rename and add tags based on the existing tags
• make new posts with mostly-cohost-compatible markdown
  (there’s even a post composer with live preview! but it’s very much a wip)

i’ve only had a couple weeks to work on this, so it’s not the most polished, and for now you need a bit of command line knowledge to get it going.

but my goal here is to make a blog engine with the same posting and reading experience as cohost, where you can follow people with rss/atom feeds, see their posts on a chronological timeline, and share their posts on yours.

does that vibe with anyone? should i keep working on this? let me know!

i gotta pay down this tech debt asafp or this thing is gonna burst into treats.

also, clicking “publish” in the composer redirects you to the post now, but i figured this would be funnier than posting a screenshot of a redirect.

thanks, i’ve fixed that now!

fluent-reader seems to need a <base> tag in each <content> to tell it how to resolve relative urls. it doesn’t support xml:base, which i’ve now reported to the devs, but it turns out my feeds didn’t have that either. my bad!

i’ve been working on something similar that will rely on this (see #shuppy.org, autost). my posts are in an html-fragment-based format, which looks something like this:

<link rel="references" href="7801740/7742758.html">
<link rel="references" href="7801740/7800234.html">
<meta name="title" content="the post hole works">
<meta name="published" content="2024-09-22T14:24:07.963Z">
<link rel="author" href="https://cohost.org/delan" name="shuppy (@delan)">
<meta name="tags" content="shuppy.org">

[post content in markdown or html here]

this is not the best interchange format for others to reblog, though, because it relies on you rendering markdown the same way as me, and you would need to fetch the posts i’m replying to separately. your idea would really help with that!

one suggestion: it would be nice if the format you came up with was also usable in the “whole blog” feed, where each <atom:entry> is a post, including the posts it replies to. i started thinking about atom-compatible ways of doing that in this chost.

>>> https://shuppy.org/posts/10000000.html <<<

is this thing on?

to better curate my chosts even after cohost shuts down, i can not only tag chosts without editing them, but also rename tags and add related tags automatically. for example:

• #photography now includes all of my birds and wildlife shots
• #watching now includes all of my tv and film chosting
• #reverse engineering now includes all of my ghidra chosts

(the blog software is here, if you wanna play around with it)

thanks for the heads up! i've been testing with freshrss so far, if you can tell me what reader you use that would be really helpful

next stop: tag synonyms and tag implications

what if atom had a way to embed a copy of the post you're sharing or replying to, not just link to it? i would say just add a second <content>, but that’s forbidden :(

edit: hmm how about nesting <entry>, or putting an <entry> inside the <link> (instead of self-closing <link/>). per § 6.4.2, we may be able to nest <entry> as long as we have a foreign element in between, like <atom:entry><cohost:reply><atom:entry>

oh and i’ve updated my contact details!! say hi and send me stuff :3

i had an old website, but for a variety of reasons it really didn’t feel like a good successor to my cohost page. i feel like what i really want is to Continue Chosting next month, just on my own website? (well no, what i really want is cohost, but failing that…)

so i made this! and many of my old chosts are there now too! i’ll need a bit more time to figure out how to actually make new posts (lol) and have them play nice with my old chosts, but hey, now you know where to find me once i do o/