shuppy — autost

embedding a javascript runtime in rust, part 4: faster handles

2026-07-10T11:46:36.090Z

previously we created our first real abstraction, a Handle type that represents a javascript object in rust, and keeps the object alive until it’s dropped.

but if we benchmark it, we find that creating and destroying a Handle takes well over 1500ns, which is incredibly inefficient. what gives?

src/main.rs

#![cfg_attr(feature = "bench", feature(test))]
#[cfg(feature = "bench")]
extern crate test;

#[cfg(feature = "bench")]
#[bench]
fn bench(bencher: &mut test::bench::Bencher) {
    unsafe {
        let ctx = Context::create();
        ctx.push_c_function(Some(print), DUK_VARARGS);
        ctx.put_global_string(c"print".as_ptr());

        ctx.eval_string(cr#"
            var HANDLES = {};
            var NEXT_HANDLE_KEY = 0;
        "#.as_ptr());
        ctx.pop();

        bencher.iter(|| {
            ctx.push_string(c"ao!!".as_ptr());
            let handle = Handle::new_from_stack(&ctx, -1);
            ctx.pop();
        });
    }
}

$ cargo +nightly bench --features bench
test handle_new_drop ... bench:       1,709.24 ns/iter (+/- 14.79)

8. optimising handles with samply

samply is a sampling profiler that integrates with the excellent firefox profiler.

i used to use cargo-flamegraph, and flamegraph.pl before that, but the firefox profiler UI is so much more powerful (and unlike the perfetto UI, i actually understand how to use it effectively). samply itself is also both faster and easier to use, in large part because it resolves stacks and symbols on the fly, no postprocessing needed.

let’s get samply and perf (required by samply on linux), and while we’re at it, let’s set up mold for faster linking, since we’re gonna be iterating a lot.

shell.nix

with import <nixpkgs> {};
(mkShell.override { stdenv = clangStdenv; }) {
  name = "duktest-shell";
  LIBCLANG_PATH = lib.makeLibraryPath [ llvmPackages.clang-unwrapped.lib ];
  buildInputs = [ samply perf mold ];
  RUSTFLAGS = "-Clink-arg=-fuse-ld=mold";
}

we’ll need a couple of linux kernel settings too. the former will probably be familiar to anyone who has done profiling on linux, but the latter is also needed to avoid a “Failed to start profiling: mmap failed” error when running samply.

$ echo 1 | sudo tee /proc/sys/kernel/perf_event_paranoid
$ sudo sysctl kernel.perf_event_mlock_kb=2048

now let’s take a profile. for each profile i take, i’ll resolve everything and upload it, so you can see what i’m seeing.

$ cargo +nightly bench --no-run --features bench
$ samply record -- cargo +nightly bench --features bench

8.1. avoiding bytecode compilation (1709ns → 331ns)

focusing on the bencher.iter() closure, 88% of the time is spent in duk_eval_raw(), and 80pp of that is in duk_compile_raw(). we’re compiling NEXT_HANDLE_KEY++ over and over, every single time we create a handle!

https://share.firefox.dev/44UWvlj

let’s turn that into a function.

diff --git a/src/context.rs b/src/context.rs
index c8090f0..c192483 100644
--- a/src/context.rs
+++ b/src/context.rs
@@ -25,2 +25,5 @@ impl Context {
                 var NEXT_HANDLE_KEY = 0;
+                function next_handle_key() {
+                    return NEXT_HANDLE_KEY++;
+                }
             "#.as_ptr());
diff --git a/src/main.rs b/src/main.rs
index 4d13455..7a8ecb4 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -43,3 +43,4 @@ impl Handle<'_> {
             ctx.get_global_string(c"HANDLES".as_ptr());
-            ctx.eval_string(c"NEXT_HANDLE_KEY++".as_ptr());
+            ctx.get_global_string(c"next_handle_key".as_ptr());
+            ctx.call(0);
             let key = ctx.get_uint(-1);

$ cargo +nightly bench --features bench
test handle_new_drop ... bench:         331.94 ns/iter (+/- 3.03)

8.2. caching heap pointers (331ns → 245ns)

now we’re spending 29% of our time in duk_get_global_string(), and 22pp of that in duk_get_prop(). looking up properties is not fast!

https://share.firefox.dev/4gqvc9A

let’s cache the heap pointers for HANDLES and next_handle_key().

diff --git a/src/context.rs b/src/context.rs
index c192483..8d02f0d 100644
--- a/src/context.rs
+++ b/src/context.rs
@@ -7,2 +7,4 @@ pub struct Context {
     ctx: *mut duk_context,
+    pub(crate) handles: *mut c_void,
+    pub(crate) next_handle_key: *mut c_void,
 }
@@ -31,3 +33,9 @@ impl Context {

-            Self { ctx }
+            duk_get_global_string(ctx, c"HANDLES".as_ptr());
+            duk_get_global_string(ctx, c"next_handle_key".as_ptr());
+            let handles = duk_get_heapptr(ctx, -2);
+            let next_handle_key = duk_get_heapptr(ctx, -1);
+            duk_pop_2(ctx);
+
+            Self { ctx, handles, next_handle_key }
         }
diff --git a/src/main.rs b/src/main.rs
index ca3c650..07005f1 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -28,3 +28,3 @@ impl Drop for Handle<'_> {
             // `delete HANDLES[rust self.key]`
-            self.ctx.get_global_string(c"HANDLES".as_ptr());
+            self.ctx.push_heapptr(self.ctx.handles);
             self.ctx.del_prop_index(-1, self.key);
@@ -42,4 +42,4 @@ impl Handle<'_> {
             let index = ctx.normalize_index(index);
-            ctx.get_global_string(c"HANDLES".as_ptr());
-            ctx.get_global_string(c"next_handle_key".as_ptr());
+            ctx.push_heapptr(ctx.handles);
+            ctx.push_heapptr(ctx.next_handle_key);
             ctx.call(0);

$ cargo +nightly bench --features bench
test handle_new_drop ... bench:         245.00 ns/iter (+/- 8.03)

8.3. avoiding js function calls (245ns → 152ns)

now we’re spending 33% of our time in duk__handle_call_raw(), 33% of our time in duk_put_prop(), and 22% of our time in duk_del_prop().

https://share.firefox.dev/4vmq0qY

let’s move the steps for generating the next handle key out of javascript entirely. with a u64, we should be able to create 2³² new handles per second for 136 years, which is probably good enough. unfortunately duktape doesn’t support bigint yet, so we’ll only be able to create 2⁵³ handles without collisions, which comes out to 2²⁴ handles per second (59ns each) for 17 years. i don’t feel as comfortable with this, but we’ll replace it with something better soon anyway.

diff --git a/src/context.rs b/src/context.rs
index 8d02f0d..e921295 100644
--- a/src/context.rs
+++ b/src/context.rs
@@ -8,3 +8,3 @@ pub struct Context {
     pub(crate) handles: *mut c_void,
-    pub(crate) next_handle_key: *mut c_void,
+    pub(crate) next_handle_key: std::sync::atomic::AtomicU64,
 }
@@ -26,6 +26,2 @@ impl Context {
                 var HANDLES = {};
-                var NEXT_HANDLE_KEY = 0;
-                function next_handle_key() {
-                    return NEXT_HANDLE_KEY++;
-                }
             "#.as_ptr());
@@ -34,8 +30,6 @@ impl Context {
             duk_get_global_string(ctx, c"HANDLES".as_ptr());
-            duk_get_global_string(ctx, c"next_handle_key".as_ptr());
-            let handles = duk_get_heapptr(ctx, -2);
-            let next_handle_key = duk_get_heapptr(ctx, -1);
-            duk_pop_2(ctx);
+            let handles = duk_get_heapptr(ctx, -1);
+            duk_pop(ctx);

-            Self { ctx, handles, next_handle_key }
+            Self { ctx, handles, next_handle_key: 0.into() }
         }
diff --git a/src/main.rs b/src/main.rs
index 07005f1..3e321e8 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -20,3 +20,3 @@ struct Handle<'ctx> {
     ctx: &'ctx Context,
-    key: duk_uint_t,
+    key: u64,
 }
@@ -29,3 +29,4 @@ impl Drop for Handle<'_> {
             self.ctx.push_heapptr(self.ctx.handles);
-            self.ctx.del_prop_index(-1, self.key);
+            self.ctx.push_number(self.key as f64);
+            self.ctx.del_prop(-2);
             self.ctx.pop();
@@ -43,5 +44,4 @@ impl Handle<'_> {
             ctx.push_heapptr(ctx.handles);
-            ctx.push_heapptr(ctx.next_handle_key);
-            ctx.call(0);
-            let key = ctx.get_uint(-1);
+            let key = ctx.next_handle_key.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
+            ctx.push_number(key as f64);
             ctx.dup(index);

$ cargo +nightly bench --features bench
test handle_new_drop ... bench:         152.22 ns/iter (+/- 1.02)

8.4. avoiding stringifying integers (152ns → 103ns)

now we’re spending 44% of our time in duk_del_prop() and 39% of our time in duk_put_prop(), with 23pp plus 29pp of that in duk_to_string(). in other words, we’re spending over half of our time stringifying integers in decimal!

https://share.firefox.dev/4ym5iKO

let’s pack the bits of our u64 into a nine-byte string, of the form FF xx xx xx xx xx xx xx xx_h. why the leading FF_h?

duktape API strings are based on CESU-8, which is like UTF-8 but encodes values above U+FFFF differently and less efficiently (making it not a superset of UTF-8):

// U+10000 LINEAR B SYLLABLE B008 A
duk_eval_string(ctx, cr#""\uD800\uDC00""#.as_ptr());
let mut test_len = 0usize;
let test = duk_get_lstring(ctx, -1, &amp;mut test_len as *mut _);

// CESU-8: `[ed, a0, 80, ed, b0, 80]`
eprintln!("{:02x?}", std::slice::from_raw_parts(test as *const u8, test_len));
duk_pop(ctx);

// UTF-8: `[f0, 90, 80, 80]`
eprintln!("{:02x?}", "\u{10000}".as_bytes());

but they also allow the encoding of:

unpaired surrogates, which are necessary to represent javascript strings faithfully. duktape encodes these using the same technique as CESU-8, albeit technically out of spec. the newer, more efficient UTF-8-based counterpart to this approach is WTF-8, created by my (now) colleague Simon Sapin.
Symbol values, using byte sequences that are not otherwise valid. for example, Symbol.toPrimitive is \x81Symbol.toPrimitive\xff.
- and “hidden” Symbol values, which can only be created outside javascript, and need not ever be exposed to javascript. notably these have the form \xff followed by any sequence of bytes.

diff --git a/src/main.rs b/src/main.rs
index 3e321e8..61de949 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -29,3 +29,4 @@ impl Drop for Handle<'_> {
             self.ctx.push_heapptr(self.ctx.handles);
-            self.ctx.push_number(self.key as f64);
+            let key_symbol = Self::key_to_symbol(self.key);
+            self.ctx.push_lstring(key_symbol.as_ptr() as *const _, key_symbol.len());
             self.ctx.del_prop(-2);
@@ -37,2 +38,7 @@ impl Drop for Handle<'_> {
 impl Handle<'_> {
+    fn key_to_symbol(key: u64) -> [u8; 9] {
+        let key = key.to_ne_bytes();
+        [0xff, key[0], key[1], key[2], key[3], key[4], key[5], key[6], key[7]]
+    }
+
     /// create a handle for the object at `index` on the stack,
@@ -45,3 +51,4 @@ impl Handle<'_> {
             let key = ctx.next_handle_key.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
-            ctx.push_number(key as f64);
+            let key_symbol = Self::key_to_symbol(key);
+            ctx.push_lstring(key_symbol.as_ptr() as *const _, key_symbol.len());
             ctx.dup(index);

$ cargo +nightly bench --features bench
test handle_new_drop ... bench:         103.73 ns/iter (+/- 0.67)

8.5. array object fast path (103ns → 79ns)

now we’re spending 39% of our time in duk_push_lstring(), with 37pp of that in duk_heap_strtable_intern(), and we also spend 28% and 16% in duk_put_prop() and duk_del_prop() respectively. in general, javascript property keys are strings, and property accesses require both string interning and hash table lookup.

https://share.firefox.dev/3SWUsdR

to make most Array objects faster, duktape has a fast path for array index accesses on those objects, which stores values as an array internally. for that fast path to kick in though, we would need to meet a few requirements:

the object needs to be an Array object
the array needs to be dense, not sparse (no delete holes)
all array index properties need to have default attributes

let’s rework HANDLES to meet these requirements. if we’re gonna store handles as an array, we’ll want to reuse low indices whenever possible, to keep the array only as large as the peak number of concurrent handles.

another reason to reuse low indices is that by definition, an array index is less than 2³². my rule of thumb for avoiding collisions in integer id types is that u32 requires reuse to avoid collisions, u64 can be sequentially issued without reuse (up to 2³² per second for 136 years), and u128 can be randomly issued without reuse (hence UUIDv4).

when we destroy handles, we can ensure that future handles reuse their keys in constant time and constant extra space, by encoding a free list in their place, and keeping track of the head of that linked list:

initial state:
HANDLES = [object, object, object]
head → none
destroy handle 1:
HANDLES = [object, null, object]
head → handle 1 → none
destroy handle 0:
HANDLES = [1, null, object]
head → handle 0 → handle 1 → none
create handle (handle 0):
HANDLES = [object, null, object]
head → handle 1 → none
create handle (handle 1):
HANDLES = [object, object, object]
head → none
create handle (handle 2):
HANDLES = [object, object, object, object]
head → none

diff --git a/src/context.rs b/src/context.rs
index e921295..d1f8b65 100644
--- a/src/context.rs
+++ b/src/context.rs
@@ -1,2 +1,2 @@
-use std::{ffi::{CStr, c_char, c_void}, ptr::null_mut};
+use std::{cell::Cell, ffi::{CStr, c_char, c_void}, ptr::null_mut, rc::Rc};

@@ -8,3 +8,4 @@ pub struct Context {
     pub(crate) handles: *mut c_void,
-    pub(crate) next_handle_key: std::sync::atomic::AtomicU64,
+    pub(crate) next_handle_key: Cell<u32>,
+    pub(crate) free_handle_key: Rc<Cell<Option<u32>>>,
 }
@@ -25,3 +26,3 @@ impl Context {
             duk_eval_string(ctx, cr#"
-                var HANDLES = {};
+                var HANDLES = [];
             "#.as_ptr());
@@ -33,3 +34,3 @@ impl Context {

-            Self { ctx, handles, next_handle_key: 0.into() }
+            Self { ctx, handles, next_handle_key: 0.into(), free_handle_key: Rc::new(None.into()) }
         }
diff --git a/src/main.rs b/src/main.rs
index 61de949..15204c4 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -20,3 +20,3 @@ struct Handle<'ctx> {
     ctx: &'ctx Context,
-    key: u64,
+    key: u32,
 }
@@ -27,7 +27,10 @@ impl Drop for Handle<'_> {
         unsafe {
-            // `delete HANDLES[rust self.key]`
+            // `HANDLES[rust self.key] = tombstone`
             self.ctx.push_heapptr(self.ctx.handles);
-            let key_symbol = Self::key_to_symbol(self.key);
-            self.ctx.push_lstring(key_symbol.as_ptr() as *const _, key_symbol.len());
-            self.ctx.del_prop(-2);
+            if let Some(old_free_handle_key) = self.ctx.free_handle_key.replace(Some(self.key)) {
+                self.ctx.push_uint(old_free_handle_key);
+            } else {
+                self.ctx.push_null();
+            }
+            self.ctx.put_prop_index(-2, self.key);
             self.ctx.pop();
@@ -38,7 +41,2 @@ impl Drop for Handle<'_> {
 impl Handle<'_> {
-    fn key_to_symbol(key: u64) -> [u8; 9] {
-        let key = key.to_ne_bytes();
-        [0xff, key[0], key[1], key[2], key[3], key[4], key[5], key[6], key[7]]
-    }
-
     /// create a handle for the object at `index` on the stack,
@@ -47,10 +45,18 @@ impl Handle<'_> {
         unsafe {
-            // `HANDLES[NEXT_HANDLE_KEY++] = stack[rust index]`
+            // `HANDLES[key] = stack[rust index]`
             let index = ctx.normalize_index(index);
             ctx.push_heapptr(ctx.handles);
-            let key = ctx.next_handle_key.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
-            let key_symbol = Self::key_to_symbol(key);
-            ctx.push_lstring(key_symbol.as_ptr() as *const _, key_symbol.len());
+            let key = if let Some(key) = ctx.free_handle_key.get() {
+                ctx.get_prop_index(-1, key.into());
+                let old_free_handle_key = ctx.get_uint(-1);
+                ctx.free_handle_key.set(Some(old_free_handle_key));
+                ctx.pop();
+                key
+            } else {
+                let key = ctx.next_handle_key.get();
+                ctx.next_handle_key.set(key.checked_add(1).expect("too many handles"));
+                key
+            };
             ctx.dup(index);
-            ctx.put_prop(-3);
+            ctx.put_prop_index(-2, key);
             ctx.pop();

$ cargo +nightly bench --features bench
test handle_new_drop ... bench:          79.64 ns/iter (+/- 6.64)

https://share.firefox.dev/4pgwq9P

just the beginning

we now have handles that can be created and destroyed in under 80ns.

there’s so much more we can do with handles though! see JSVM, by my friend Alicia, for another duktape binding where the handles:

support canonicalisation (at least for java → javascript) – take another handle to the same object, you get the same handle
support chaining – drill down into a nested object without creating handles for the intermediate levels, and improve the error messages for type errors

embedding a javascript runtime in rust, part 3: object handles

2026-07-09T04:33:16.630Z

previously we looked at how to call javascript functions in rust, and found a relatively efficient way to export rust structs to javascript.

how do we access a javascript object from rust later, without keeping it on the value stack the whole time? we could store it in a js variable, and look it up later:

src/main.rs > unsafe {}

let ctx = duk_create_heap(None, None, None, null_mut(), None);
duk_push_c_function(ctx, Some(print), DUK_VARARGS);
duk_put_global_string(ctx, c"print".as_ptr());

duk_push_array(ctx);
duk_push_int(ctx, 0*0);
duk_put_prop_index(ctx, -2, 0);
duk_push_int(ctx, 1*1);
duk_put_prop_index(ctx, -2, 1);
duk_push_int(ctx, 2*2);
duk_put_prop_index(ctx, -2, 2);
duk_push_int(ctx, 3*3);
duk_put_prop_index(ctx, -2, 3);

// done with the array for now!
duk_put_global_string(ctx, c"squares".as_ptr());

assert_eq!(duk_get_top(ctx), 0);
// TODO: some other work
assert_eq!(duk_get_top(ctx), 0);

duk_get_global_string(ctx, c"print".as_ptr());
duk_get_global_string(ctx, c"squares".as_ptr());
duk_call(ctx, 1);
duk_pop(ctx);

println!();
duk_destroy_heap(ctx);

$ cargo run
0,1,4,9

this works, but property lookup can be slow. and for js objects that matter to rust, we really want to be able to interact with them as rust values, and tie their lifetimes to other values that depend on them.

6. js values must outlive their rust usage

((o) duktape has a garbage-collected heap, which makes allocations for strings, buffers, and (notably) objects. duktape also never moves the top-level allocations for those, so it would seem as though we could grab the heap pointer for the array, and push it back onto the stack later when we need it again. but in reality, we crash.

src/main.rs > unsafe {}

// done with the array for now!
let squares = duk_get_heapptr(ctx, -1);
duk_pop(ctx);

assert_eq!(duk_get_top(ctx), 0);
// TODO: some other work
assert_eq!(duk_get_top(ctx), 0);

duk_get_global_string(ctx, c"print".as_ptr());
duk_push_heapptr(ctx, squares);
duk_call(ctx, 1);
duk_pop(ctx);

println!();
duk_destroy_heap(ctx);

$ cargo run
Segmentation fault

we no longer store the object in a variable, so it’s vulnerable to garbage collection as soon as we duk_pop(). what we need is a rust value that:

can access a javascript value at any time, and
keeps that javascript value alive for as long as it lives.

that’s a handle! we can achieve those two things by:

saving the heap pointer in the handle with duk_get_heapptr(), then restoring it later with duk_push_heapptr() whenever we need to access it.
creating a strong reference to the object in javascript, by setting a property of a global variable, then deleting that property when the handle is dropped.

src/main.rs > unsafe {}

duk_eval_string(ctx, cr#"
    var HANDLES = {};
    var NEXT_HANDLE_KEY = 0;
"#.as_ptr());
duk_pop(ctx);

/// rooting reference to a javascript object.
#[derive(Debug)]
struct Handle {
    /// heap pointer of the object, to access it in [`Handle::push()`]
    ptr: *mut c_void,

    /// handle key, to release the handle in [`Drop`]
    ctx: *mut duk_context,
    key: duk_uint_t,
}

/// unroot the object, allowing it to be garbage collected.
impl Drop for Handle {
    fn drop(&mut self) {
        unsafe {
            // `delete HANDLES[rust self.key]`
            duk_get_global_string(self.ctx, c"HANDLES".as_ptr());
            duk_del_prop_index(self.ctx, -1, self.key);
            duk_pop(self.ctx);
        }
    }
}

impl Handle {
    /// create a handle for the object at `index` on the stack,
    /// rooting it to prevent garbage collection.
    fn new_from_stack(ctx: *mut duk_context, index: duk_idx_t) -> Handle {
        unsafe {
            // `HANDLES[NEXT_HANDLE_KEY++] = stack[rust index]`
            let index = duk_normalize_index(ctx, index);
            duk_get_global_string(ctx, c"HANDLES".as_ptr());
            duk_eval_string(ctx, c"NEXT_HANDLE_KEY++".as_ptr());
            let key = duk_get_uint(ctx, -1);
            duk_dup(ctx, index);
            duk_put_prop(ctx, -3);
            duk_pop(ctx);

            let ptr = duk_get_heapptr(ctx, index);
            Handle { ptr, ctx, key }
        }
    }

    /// push the object onto the value stack.
    fn push(&self) {
        unsafe {
            duk_push_heapptr(self.ctx, self.ptr);
        }
    }
}

these handles work a bit better, but the Drop impl causes a crash if it happens after duk_destroy_heap(), which is true for let bindings in the same scope.

src/main.rs > unsafe {}

// done with the array for now!
let squares = Handle::new_from_stack(ctx, -1);
duk_pop(ctx);

assert_eq!(duk_get_top(ctx), 0);
// TODO: some other work
assert_eq!(duk_get_top(ctx), 0);

duk_get_global_string(ctx, c"print".as_ptr());
squares.push();
duk_call(ctx, 1);
duk_pop(ctx);

println!();
duk_destroy_heap(ctx);

$ cargo run
0,1,4,9
Segmentation fault

7. the js heap must outlive all handles

we need to ensure that the duktape heap and context aren’t destroyed before we run the Drop impl of any Handle, because dropping a handle requires running instructions in the context. but ctx is a raw pointer (*mut duk_context), which has no lifetime.

to give the context a lifetime, we need to use references. but raw pointers are Copy, which means we can trivially sneak them out of any reference, so we’ll need to create a non-Copy wrapper struct…

src/context.rs

use std::{ffi::{CStr, c_char, c_void}, ptr::null_mut};

use crate::sys::*;

#[derive(Debug)]
pub struct Context {
    ctx: *mut duk_context,
}

impl Drop for Context {
    fn drop(&mut self) {
        unsafe {
            duk_destroy_heap(self.ctx);
        }
    }
}

impl Context {
    pub fn create() -> Self {
        unsafe {
            let ctx = duk_create_heap(None, None, None, null_mut(), Some(fatal));
            Self { ctx }
        }
    }
}

macro_rules! fn_wrapper {
    ($call_name:ident, $method_name:ident(ctx: *mut duk_context $(, $arg_name:ident: $arg_ty:ty)* $(,)?) $(-> $ret_ty:ty)?) => {
        #[allow(unused)]
        pub unsafe fn $method_name(&self $(, $arg_name: $arg_ty)*) $(-> $ret_ty)? {
            unsafe {
                $call_name(self.ctx $(, $arg_name)*)
            }
        }
    };
}

impl Context {
    // wrappers for functions not supported by bindgen
    fn_wrapper!(duk_eval_string, eval_string(ctx: *mut duk_context, src: *const c_char));

    // wrappers mechanically derived from bindgen output for `duktape.h`
    fn_wrapper!(duk_push_undefined, push_undefined(ctx: *mut duk_context));
    fn_wrapper!(duk_push_null, push_null(ctx: *mut duk_context));
    fn_wrapper!(duk_push_boolean, push_boolean(ctx: *mut duk_context, val: duk_bool_t));
    fn_wrapper!(duk_push_true, push_true(ctx: *mut duk_context));
    fn_wrapper!(duk_push_false, push_false(ctx: *mut duk_context));
    fn_wrapper!(duk_push_number, push_number(ctx: *mut duk_context, val: duk_double_t));
    fn_wrapper!(duk_push_nan, push_nan(ctx: *mut duk_context));
    fn_wrapper!(duk_push_int, push_int(ctx: *mut duk_context, val: duk_int_t));
    fn_wrapper!(duk_push_uint, push_uint(ctx: *mut duk_context, val: duk_uint_t));
    fn_wrapper!(duk_push_string, push_string(ctx: *mut duk_context, str_: *const ::std::os::raw::c_char, ) -> *const ::std::os::raw::c_char);
    fn_wrapper!(duk_push_lstring, push_lstring(ctx: *mut duk_context, str_: *const ::std::os::raw::c_char, len: duk_size_t, ) -> *const ::std::os::raw::c_char);
    fn_wrapper!(duk_push_pointer, push_pointer(ctx: *mut duk_context, p: *mut ::std::os::raw::c_void));
    // ...
}

extern "C" fn fatal(_: *mut c_void, message: *const c_char) {
    unsafe {
        panic!("{:?}", CStr::from_ptr(message));
    }
}

…so we can borrow it in Handle. unfortunately (for my brevity) this also means we’ll have to rewrite all of the duktape calls from duk_foo(ctx, ...) to ctx.foo(...).

src/main.rs > unsafe {}

let ctx = Context::create();
ctx.push_c_function(Some(print), DUK_VARARGS);
ctx.put_global_string(c"print".as_ptr());

ctx.push_array();
ctx.push_int(0*0);
ctx.put_prop_index(-2, 0);
ctx.push_int(1*1);
ctx.put_prop_index(-2, 1);
ctx.push_int(2*2);
ctx.put_prop_index(-2, 2);
ctx.push_int(3*3);
ctx.put_prop_index(-2, 3);

ctx.eval_string(cr#"
    var HANDLES = {};
    var NEXT_HANDLE_KEY = 0;
"#.as_ptr());
ctx.pop();

/// rooting reference to a javascript object.
#[derive(Debug)]
struct Handle<'ctx> {
    /// heap pointer of the object, to access it in [`Handle::push()`]
    ptr: *mut c_void,

    /// handle key, to release the handle in [`Drop`]
    ctx: &'ctx Context,
    key: duk_uint_t,
}

/// unroot the object, allowing it to be garbage collected.
impl Drop for Handle<'_> {
    fn drop(&mut self) {
        unsafe {
            // `delete HANDLES[rust self.key]`
            self.ctx.get_global_string(c"HANDLES".as_ptr());
            self.ctx.del_prop_index(-1, self.key);
            self.ctx.pop();
        }
    }
}

impl Handle<'_> {
    /// create a handle for the object at `index` on the stack,
    /// rooting it to prevent garbage collection.
    fn new_from_stack(ctx: &Context, index: duk_idx_t) -> Handle<'_> {
        unsafe {
            // `HANDLES[NEXT_HANDLE_KEY++] = stack[rust index]`
            let index = ctx.normalize_index(index);
            ctx.get_global_string(c"HANDLES".as_ptr());
            ctx.eval_string(c"NEXT_HANDLE_KEY++".as_ptr());
            let key = ctx.get_uint(-1);
            ctx.dup(index);
            ctx.put_prop(-3);
            ctx.pop();

            let ptr = ctx.get_heapptr(index);
            Handle { ptr, ctx, key }
        }
    }

    /// push the object onto the value stack.
    fn push(&self) {
        unsafe {
            self.ctx.push_heapptr(self.ptr);
        }
    }
}

// done with the array for now!
let squares = Handle::new_from_stack(&ctx, -1);
ctx.pop();

assert_eq!(ctx.get_top(), 0);
// TODO: some other work
assert_eq!(ctx.get_top(), 0);

ctx.get_global_string(c"print".as_ptr());
squares.push();
ctx.call(1);
ctx.pop();

println!();
// impl Drop for Handle will run first
// then impl Drop for Context will call duk_destroy_heap()

with the Handle borrowing Context, rust ensures that the Context is dropped after the Handle when they both go out of scope. voilà!

$ cargo run
0,1,4,9

stay tuned!

the Handle impl shown above is very inefficient, taking over 1500ns to create and destroy a single handle. next time, we’ll optimise it, down to less than 80ns.

embedding a javascript runtime in rust, part 2: mixing languages

2026-07-08T02:52:40.952Z

previously we created some rust bindings for ((o) duktape, a javascript runtime that turned out to be pretty easy to embed.

we want to be able to pass structs from rust to javascript, so that scripts can act on the same information as the rest of the program.

say we have a struct that represents the author of a blog post.

src/main.rs > fn main()

struct Author {
    name: String,
    link_url: String,
    avatar_url: String,
    email_address: String,
    phone_number: String,
    pronouns: String,
    bio: String,
}

let author = Author {
    name: "shuppy".to_owned(),
    link_url: "https://shuppy.org".to_owned(),
    avatar_url: "https://shuppy.org/avatar.png".to_owned(),
    email_address: "test@shuppy.org".to_owned(),
    phone_number: "+61 491 570 006".to_owned(),
    pronouns: "she/her".to_owned(),
    bio: "
        Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor
        incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud
        exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure
        dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
        Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt
        mollit anim id est laborum.
    ".repeat(100),
};

unsafe {
    todo!();
}

4. calling js functions in rust

we want to let the user write a linkify_author() function to customise the markdown we generate for links to the author. first we create a heap and context, then we run the user’s custom code, then we call the user’s function.

src/main.rs > fn main() > todo!()

let ctx = duk_create_heap(None, None, None, null_mut(), None);
duk_push_c_function(ctx, Some(print), DUK_VARARGS);
duk_put_global_string(ctx, c"print".as_ptr());

// run the user’s custom code
duk_eval_string(ctx, cr#"
    function linkify_author(author) {
        return "[" + author.name + "](" + author.link_url + ")";
    }
"#.as_ptr());
duk_pop(ctx);

// TODO: call the user’s function

// `result = pop()`
println!("{:?}", CStr::from_ptr(duk_get_string(ctx, -1)));
duk_pop(ctx);

duk_destroy_heap(ctx);

duktape’s interpreter is a stack machine. we interact with that machine by feeding it instructions that push values onto, pop values off, read from, or reorder the stack.

to call linkify_author(), we push the function, then the arguments (the author), then call that function with one argument, leaving the return value on the stack. but the author is a rust struct, so how do we push that?

src/main.rs > fn main() > todo!() > // TODO: call the user’s function

// `push(linkify_author(rust author))`
duk_get_global_string(ctx, c"linkify_author".as_ptr());
// TODO: duk_push_something(rust author);
duk_call(ctx, 1);

5. exporting rust structs to js

we need to build an object in javascript representing the Author value we have in rust.

let’s reorder our instructions a bit, to untangle building the object from calling the function. this is more readable, but probably less efficient.

src/main.rs > fn main() > todo!() > // TODO: call the user’s function

// TODO: duk_push_something(rust author);

// `push(linkify_author(rust author))`
duk_get_global_string(ctx, c"linkify_author".as_ptr());
duk_pull(ctx, -2);
duk_call(ctx, 1);

the obvious approach would be to build a javascript object that has all of the same fields as the rust Author, converted to javascript types.

src/main.rs > fn main() > todo!() > // TODO: duk_push_something(rust author);

// `author = {};`
// `for (key in author) author[key] = convert rust author[key];`
// `push(author)`
duk_push_object(ctx);
duk_push_lstring(ctx, author.name.as_ptr() as *const _, author.name.len());
duk_put_prop_string(ctx, -2, c"name".as_ptr());
duk_push_lstring(ctx, author.link_url.as_ptr() as *const _, author.link_url.len());
duk_put_prop_string(ctx, -2, c"link_url".as_ptr());
duk_push_lstring(ctx, author.avatar_url.as_ptr() as *const _, author.avatar_url.len());
duk_put_prop_string(ctx, -2, c"avatar_url".as_ptr());
duk_push_lstring(ctx, author.email_address.as_ptr() as *const _, author.email_address.len());
duk_put_prop_string(ctx, -2, c"email_address".as_ptr());
duk_push_lstring(ctx, author.phone_number.as_ptr() as *const _, author.phone_number.len());
duk_put_prop_string(ctx, -2, c"phone_number".as_ptr());
duk_push_lstring(ctx, author.pronouns.as_ptr() as *const _, author.pronouns.len());
duk_put_prop_string(ctx, -2, c"pronouns".as_ptr());
duk_push_lstring(ctx, author.bio.as_ptr() as *const _, author.bio.len());
duk_put_prop_string(ctx, -2, c"bio".as_ptr());

this works!

$ cargo run
"[shuppy](https://shuppy.org)"

but if the javascript code doesn’t end up reading all of the fields, then we’ve wasted time copying a bunch of strings for nothing.

what if we instead build an object that wraps an “inner” rust pointer, with getters for each field that only copy strings into javascript when needed?

src/main.rs > fn main() > todo!() > // TODO: duk_push_something(rust author);

// `author = {inner: rust author};`
// `for (key in author) Object.defineProperty(author, key, {get});`
// `push(author)`
duk_push_object(ctx);
duk_push_pointer(ctx, &author as *const _ /* 🗞️ bad */ as *mut _);
duk_put_prop_string(ctx, -2, c"inner".as_ptr());
macro_rules! define_author_getter {
    ($ctx:ident, $key:literal, |author| author.$field:ident) => {{
        duk_push_string($ctx, $key.as_ptr());
        duk_push_c_function($ctx, Some(get), 1);
        duk_def_prop($ctx, -3, DUK_DEFPROP_HAVE_GETTER);
        extern "C" fn get(ctx: *mut duk_context) -> duk_ret_t {
            unsafe {
                duk_push_this(ctx);
                duk_get_prop_string(ctx, -1, c"inner".as_ptr());
                let author = duk_get_pointer(ctx, -1) as *const Author;
                duk_pop_2(ctx);
                let result = &(*author).$field;
                duk_push_lstring(ctx, result.as_ptr() as *const _, result.len());
            }
            1
        }
    }};
}
define_author_getter!(ctx, c"name", |author| author.name);
define_author_getter!(ctx, c"link_url", |author| author.link_url);
define_author_getter!(ctx, c"avatar_url", |author| author.avatar_url);
define_author_getter!(ctx, c"email_address", |author| author.email_address);
define_author_getter!(ctx, c"phone_number", |author| author.phone_number);
define_author_getter!(ctx, c"pronouns", |author| author.pronouns);
define_author_getter!(ctx, c"bio", |author| author.bio);

we’re defining getters for each field on the instance, and if we have more than one instance, that’s quite a few instructions per instance. let’s define them on a prototype, which many instances can cheaply reuse.

src/main.rs > fn main() > todo!() > // TODO: duk_push_something(rust author);

duk_eval_string(ctx, cr#"
    function Author(inner) {
        this.inner = inner;
    }
"#.as_ptr());
duk_pop(ctx);

// `for (key in author) Object.defineProperty(Author.prototype, key, {get})`
duk_eval_string(ctx, c"Author.prototype".as_ptr());
macro_rules! define_author_getter {
    ($ctx:ident, $key:literal, |author| author.$field:ident) => {{
        duk_push_string($ctx, $key.as_ptr());
        duk_push_c_function($ctx, Some(get), 1);
        duk_def_prop($ctx, -3, DUK_DEFPROP_HAVE_GETTER);
        extern "C" fn get(ctx: *mut duk_context) -> duk_ret_t {
            unsafe {
                duk_push_this(ctx);
                duk_get_prop_string(ctx, -1, c"inner".as_ptr());
                let author = duk_get_pointer(ctx, -1) as *const Author;
                duk_pop_2(ctx);
                let result = &(*author).$field;
                duk_push_lstring(ctx, result.as_ptr() as *const _, result.len());
            }
            1
        }
    }};
}
define_author_getter!(ctx, c"name", |author| author.name);
define_author_getter!(ctx, c"link_url", |author| author.link_url);
define_author_getter!(ctx, c"avatar_url", |author| author.avatar_url);
define_author_getter!(ctx, c"email_address", |author| author.email_address);
define_author_getter!(ctx, c"phone_number", |author| author.phone_number);
define_author_getter!(ctx, c"pronouns", |author| author.pronouns);
define_author_getter!(ctx, c"bio", |author| author.bio);
duk_pop(ctx);

// `push(new Author(rust author))`
duk_get_global_string(ctx, c"Author".as_ptr());
duk_push_pointer(ctx, &author as *const _ /* 🗞️ bad */ as *mut _);
duk_new(ctx, 1);

indeed this is the fastest approach! if we subtract the ~360ns of work that is common to all three approaches, they take ~1167ns, ~595ns, and ~251ns respectively.

$ cargo +nightly bench --features bench
test empty_object     ... bench:         360.78 ns/iter (+/- 24.98)
test obvious_approach ... bench:       1,527.54 ns/iter (+/- 11.06)
test with_getters     ... bench:         955.30 ns/iter (+/- 10.53)
test with_prototype   ... bench:         611.16 ns/iter (+/- 6.16)

stay tuned!

next time, we’ll look at building some abstractions over these bindings, plus some tools to help us manage the interpreter’s value stack correctly.

full src/main.rs

mod sys;

use std::{ffi::CStr, io::{Write, stdout}, ptr::null_mut};

use crate::sys::*;

fn main() {
    struct Author {
        name: String,
        link_url: String,
        avatar_url: String,
        email_address: String,
        phone_number: String,
        pronouns: String,
        bio: String,
    }

    let author = Author {
        name: "shuppy".to_owned(),
        link_url: "https://shuppy.org".to_owned(),
        avatar_url: "https://shuppy.org/avatar.png".to_owned(),
        email_address: "test@shuppy.org".to_owned(),
        phone_number: "+61 491 570 006".to_owned(),
        pronouns: "she/her".to_owned(),
        bio: "
            Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor
            incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud
            exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure
            dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
            Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt
            mollit anim id est laborum.
        ".repeat(100),
    };

    unsafe {
        let ctx = duk_create_heap(None, None, None, null_mut(), None);
        duk_push_c_function(ctx, Some(print), DUK_VARARGS);
        duk_put_global_string(ctx, c"print".as_ptr());

        // run the user’s custom code
        duk_eval_string(ctx, cr#"
            function linkify_author(author) {
                return "[" + author.name + "](" + author.link_url + ")";
            }
        "#.as_ptr());
        duk_pop(ctx);

        duk_eval_string(ctx, cr#"
            function Author(inner) {
                this.inner = inner;
            }
        "#.as_ptr());
        duk_pop(ctx);

        // `for (key in author) Object.defineProperty(Author.prototype, key, {get})`
        duk_eval_string(ctx, c"Author.prototype".as_ptr());
        macro_rules! define_author_getter {
            ($ctx:ident, $key:literal, |author| author.$field:ident) => {{
                duk_push_string($ctx, $key.as_ptr());
                duk_push_c_function($ctx, Some(get), 1);
                duk_def_prop($ctx, -3, DUK_DEFPROP_HAVE_GETTER);
                extern "C" fn get(ctx: *mut duk_context) -> duk_ret_t {
                    unsafe {
                        duk_push_this(ctx);
                        duk_get_prop_string(ctx, -1, c"inner".as_ptr());
                        let author = duk_get_pointer(ctx, -1) as *const Author;
                        duk_pop_2(ctx);
                        let result = &(*author).$field;
                        duk_push_lstring(ctx, result.as_ptr() as *const _, result.len());
                    }
                    1
                }
            }};
        }
        define_author_getter!(ctx, c"name", |author| author.name);
        define_author_getter!(ctx, c"link_url", |author| author.link_url);
        define_author_getter!(ctx, c"avatar_url", |author| author.avatar_url);
        define_author_getter!(ctx, c"email_address", |author| author.email_address);
        define_author_getter!(ctx, c"phone_number", |author| author.phone_number);
        define_author_getter!(ctx, c"pronouns", |author| author.pronouns);
        define_author_getter!(ctx, c"bio", |author| author.bio);
        duk_pop(ctx);

        // `push(new Author(rust author))`
        duk_get_global_string(ctx, c"Author".as_ptr());
        duk_push_pointer(ctx, &author as *const _ /* 🗞️ bad */ as *mut _);
        duk_new(ctx, 1);

        // `push(linkify_author(rust author))`
        duk_get_global_string(ctx, c"linkify_author".as_ptr());
        duk_pull(ctx, -2);
        duk_call(ctx, 1);

        // `result = pop()`
        println!("{:?}", CStr::from_ptr(duk_get_string(ctx, -1)));
        duk_pop(ctx);

        duk_destroy_heap(ctx);
    }
}

extern "C" fn print(ctx: *mut duk_context) -> duk_ret_t {
    unsafe {
        duk_push_string(ctx, c" ".as_ptr());
        duk_insert(ctx, 0);
        duk_join(ctx, duk_get_top(ctx) - 1);
        let text = duk_safe_to_string(ctx, -1);
        let text = CStr::from_ptr(text);
        let _ = stdout().write_all(text.to_bytes());
    }
    0
}

extra code for cargo bench

#![cfg_attr(feature = "bench", feature(test))]
#[cfg(feature = "bench")]
extern crate test;

#[cfg(feature = "bench")]
#[bench]
fn empty_object(bencher: &mut test::bench::Bencher) {
    struct Author {
        name: String,
        link_url: String,
        avatar_url: String,
        email_address: String,
        phone_number: String,
        pronouns: String,
        bio: String,
    }

    let author = Author {
        name: "shuppy".to_owned(),
        link_url: "https://shuppy.org".to_owned(),
        avatar_url: "https://shuppy.org/avatar.png".to_owned(),
        email_address: "test@shuppy.org".to_owned(),
        phone_number: "+61 491 570 006".to_owned(),
        pronouns: "she/her".to_owned(),
        bio: "
            Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor
            incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud
            exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure
            dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
            Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt
            mollit anim id est laborum.
        ".repeat(100),
    };

    unsafe {
        let ctx = duk_create_heap(None, None, None, null_mut(), None);
        duk_push_c_function(ctx, Some(print), DUK_VARARGS);
        duk_put_global_string(ctx, c"print".as_ptr());

        // run the user’s custom code
        duk_eval_string(ctx, cr#"
            function linkify_author(author) {
                return "[" + author.name + "](" + author.link_url + ")";
            }
        "#.as_ptr());
        duk_pop(ctx);

        bencher.iter(|| {
            // `push({})`
            duk_push_object(ctx);

            // `push(linkify_author(rust author))`
            duk_get_global_string(ctx, c"linkify_author".as_ptr());
            duk_pull(ctx, -2);
            duk_call(ctx, 1);

            // `result = pop()`
            let result = CStr::from_ptr(duk_get_string(ctx, -1));
            duk_pop(ctx);
            result
        });

        duk_destroy_heap(ctx);
    }
}

#[cfg(feature = "bench")]
#[bench]
fn obvious_approach(bencher: &mut test::bench::Bencher) {
    struct Author {
        name: String,
        link_url: String,
        avatar_url: String,
        email_address: String,
        phone_number: String,
        pronouns: String,
        bio: String,
    }

    let author = Author {
        name: "shuppy".to_owned(),
        link_url: "https://shuppy.org".to_owned(),
        avatar_url: "https://shuppy.org/avatar.png".to_owned(),
        email_address: "test@shuppy.org".to_owned(),
        phone_number: "+61 491 570 006".to_owned(),
        pronouns: "she/her".to_owned(),
        bio: "
            Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor
            incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud
            exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure
            dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
            Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt
            mollit anim id est laborum.
        ".repeat(100),
    };

    unsafe {
        let ctx = duk_create_heap(None, None, None, null_mut(), None);
        duk_push_c_function(ctx, Some(print), DUK_VARARGS);
        duk_put_global_string(ctx, c"print".as_ptr());

        // run the user’s custom code
        duk_eval_string(ctx, cr#"
            function linkify_author(author) {
                return "[" + author.name + "](" + author.link_url + ")";
            }
        "#.as_ptr());
        duk_pop(ctx);

        bencher.iter(|| {
            // `author = {};`
            // `for (key in author) author[key] = convert rust author[key];`
            // `push(author)`
            duk_push_object(ctx);
            duk_push_lstring(ctx, author.name.as_ptr() as *const _, author.name.len());
            duk_put_prop_string(ctx, -2, c"name".as_ptr());
            duk_push_lstring(ctx, author.link_url.as_ptr() as *const _, author.link_url.len());
            duk_put_prop_string(ctx, -2, c"link_url".as_ptr());
            duk_push_lstring(ctx, author.avatar_url.as_ptr() as *const _, author.avatar_url.len());
            duk_put_prop_string(ctx, -2, c"avatar_url".as_ptr());
            duk_push_lstring(ctx, author.email_address.as_ptr() as *const _, author.email_address.len());
            duk_put_prop_string(ctx, -2, c"email_address".as_ptr());
            duk_push_lstring(ctx, author.phone_number.as_ptr() as *const _, author.phone_number.len());
            duk_put_prop_string(ctx, -2, c"phone_number".as_ptr());
            duk_push_lstring(ctx, author.pronouns.as_ptr() as *const _, author.pronouns.len());
            duk_put_prop_string(ctx, -2, c"pronouns".as_ptr());
            duk_push_lstring(ctx, author.bio.as_ptr() as *const _, author.bio.len());
            duk_put_prop_string(ctx, -2, c"bio".as_ptr());

            // `push(linkify_author(rust author))`
            duk_get_global_string(ctx, c"linkify_author".as_ptr());
            duk_pull(ctx, -2);
            duk_call(ctx, 1);

            // `result = pop()`
            let result = CStr::from_ptr(duk_get_string(ctx, -1));
            duk_pop(ctx);
            result
        });

        duk_destroy_heap(ctx);
    }
}

#[cfg(feature = "bench")]
#[bench]
fn with_getters(bencher: &mut test::bench::Bencher) {
    struct Author {
        name: String,
        link_url: String,
        avatar_url: String,
        email_address: String,
        phone_number: String,
        pronouns: String,
        bio: String,
    }

    let author = Author {
        name: "shuppy".to_owned(),
        link_url: "https://shuppy.org".to_owned(),
        avatar_url: "https://shuppy.org/avatar.png".to_owned(),
        email_address: "test@shuppy.org".to_owned(),
        phone_number: "+61 491 570 006".to_owned(),
        pronouns: "she/her".to_owned(),
        bio: "
            Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor
            incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud
            exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure
            dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
            Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt
            mollit anim id est laborum.
        ".repeat(100),
    };

    unsafe {
        let ctx = duk_create_heap(None, None, None, null_mut(), None);
        duk_push_c_function(ctx, Some(print), DUK_VARARGS);
        duk_put_global_string(ctx, c"print".as_ptr());

        // run the user’s custom code
        duk_eval_string(ctx, cr#"
            function linkify_author(author) {
                return "[" + author.name + "](" + author.link_url + ")";
            }
        "#.as_ptr());
        duk_pop(ctx);

        bencher.iter(|| {
            // `author = {inner: rust author};`
            // `for (key in author) Object.defineProperty(author, key, {get});`
            // `push(author)`
            duk_push_object(ctx);
            duk_push_pointer(ctx, &author as *const _ /* 🗞️ bad */ as *mut _);
            duk_put_prop_string(ctx, -2, c"inner".as_ptr());
            macro_rules! define_author_getter {
                ($ctx:ident, $key:literal, |author| author.$field:ident) => {{
                    duk_push_string($ctx, $key.as_ptr());
                    duk_push_c_function($ctx, Some(get), 1);
                    duk_def_prop($ctx, -3, DUK_DEFPROP_HAVE_GETTER);
                    extern "C" fn get(ctx: *mut duk_context) -> duk_ret_t {
                        unsafe {
                            duk_push_this(ctx);
                            duk_get_prop_string(ctx, -1, c"inner".as_ptr());
                            let author = duk_get_pointer(ctx, -1) as *const Author;
                            duk_pop_2(ctx);
                            let result = &(*author).$field;
                            duk_push_lstring(ctx, result.as_ptr() as *const _, result.len());
                        }
                        1
                    }
                }};
            }
            define_author_getter!(ctx, c"name", |author| author.name);
            define_author_getter!(ctx, c"link_url", |author| author.link_url);
            define_author_getter!(ctx, c"avatar_url", |author| author.avatar_url);
            define_author_getter!(ctx, c"email_address", |author| author.email_address);
            define_author_getter!(ctx, c"phone_number", |author| author.phone_number);
            define_author_getter!(ctx, c"pronouns", |author| author.pronouns);
            define_author_getter!(ctx, c"bio", |author| author.bio);

            // `push(linkify_author(rust author))`
            duk_get_global_string(ctx, c"linkify_author".as_ptr());
            duk_pull(ctx, -2);
            duk_call(ctx, 1);

            // `result = pop()`
            let result = CStr::from_ptr(duk_get_string(ctx, -1));
            duk_pop(ctx);
            result
        });

        duk_destroy_heap(ctx);
    }
}

#[cfg(feature = "bench")]
#[bench]
fn with_prototype(bencher: &mut test::bench::Bencher) {
    struct Author {
        name: String,
        link_url: String,
        avatar_url: String,
        email_address: String,
        phone_number: String,
        pronouns: String,
        bio: String,
    }

    let author = Author {
        name: "shuppy".to_owned(),
        link_url: "https://shuppy.org".to_owned(),
        avatar_url: "https://shuppy.org/avatar.png".to_owned(),
        email_address: "test@shuppy.org".to_owned(),
        phone_number: "+61 491 570 006".to_owned(),
        pronouns: "she/her".to_owned(),
        bio: "
            Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor
            incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud
            exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure
            dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
            Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt
            mollit anim id est laborum.
        ".repeat(100),
    };

    unsafe {
        let ctx = duk_create_heap(None, None, None, null_mut(), None);
        duk_push_c_function(ctx, Some(print), DUK_VARARGS);
        duk_put_global_string(ctx, c"print".as_ptr());

        // run the user’s custom code
        duk_eval_string(ctx, cr#"
            function linkify_author(author) {
                return "[" + author.name + "](" + author.link_url + ")";
            }
        "#.as_ptr());
        duk_pop(ctx);

        duk_eval_string(ctx, cr#"
            function Author(inner) {
                this.inner = inner;
            }
        "#.as_ptr());
        duk_pop(ctx);

        // `for (key in author) Object.defineProperty(Author.prototype, key, {get})`
        duk_eval_string(ctx, c"Author.prototype".as_ptr());
        macro_rules! define_author_getter {
            ($ctx:ident, $key:literal, |author| author.$field:ident) => {{
                duk_push_string($ctx, $key.as_ptr());
                duk_push_c_function($ctx, Some(get), 1);
                duk_def_prop($ctx, -3, DUK_DEFPROP_HAVE_GETTER);
                extern "C" fn get(ctx: *mut duk_context) -> duk_ret_t {
                    unsafe {
                        duk_push_this(ctx);
                        duk_get_prop_string(ctx, -1, c"inner".as_ptr());
                        let author = duk_get_pointer(ctx, -1) as *const Author;
                        duk_pop_2(ctx);
                        let result = &(*author).$field;
                        duk_push_lstring(ctx, result.as_ptr() as *const _, result.len());
                    }
                    1
                }
            }};
        }
        define_author_getter!(ctx, c"name", |author| author.name);
        define_author_getter!(ctx, c"link_url", |author| author.link_url);
        define_author_getter!(ctx, c"avatar_url", |author| author.avatar_url);
        define_author_getter!(ctx, c"email_address", |author| author.email_address);
        define_author_getter!(ctx, c"bio", |author| author.bio);
        duk_pop(ctx);

        bencher.iter(|| {
            // `push(new Author(rust author))`
            duk_get_global_string(ctx, c"Author".as_ptr());
            duk_push_pointer(ctx, &author as *const _ /* 🗞️ bad */ as *mut _);
            duk_new(ctx, 1);

            // `push(linkify_author(rust author))`
            duk_get_global_string(ctx, c"linkify_author".as_ptr());
            duk_pull(ctx, -2);
            duk_call(ctx, 1);

            // `result = pop()`
            let result = CStr::from_ptr(duk_get_string(ctx, -1));
            duk_pop(ctx);
            result
        });

        duk_destroy_heap(ctx);
    }
}

embedding a javascript runtime in rust, part 1: duktape bindings

2026-07-07T16:26:32.950Z

say you write a program in rust, with all of the benefits that entails (performance, type safety, etc), but you want to make parts of it programmable by the user without recompiling. you want to embed some kind of interpreted language, like lua or javascript, but surely that would be a pain? it’s not as bad as you might think!

which language? which runtime?

lua is famously embeddable, but i honestly don’t enjoy writing lua. that might be because i haven’t done it much. javascript is also embeddable, and after twenty years of writing javascript, i find it… tolerable. so let’s go with javascript.

there are many, many javascript runtimes out there. V8 and spidermonkey are cutting-edge options, very fast, but also very complex.

spidermonkey is what firefox and servo use, and it even has rust bindings, so you might think this is what i would pick. after all, servo is my job, and i’ve used html5ever and cssparser in hobby projects, and they’re great!

but i’ve never embedded an interpreter before, and the five months i had working on something adjacent to spidermonkey in servo tells me that a newbie like me should cut my teeth on something simpler.

enter: duktape

((o) duktape is a javascript runtime that’s about as easy to embed as it gets: one .c file, one .h file, and a straightforward C API. but there are some big drawbacks:

it only really supports ES5. this is (or was) a common target for compilers like babel, but if you’re writing it by hand, that means you can use things like bind() on Function and map() on Array, but not things like const or arrow functions.
it has no JIT, so it’s much, much slower than V8 and spidermonkey.

xkcd standards but it’s crates.io

i searched for “duktape” on crates.io and found 25 packages, most of which appeared to be unmaintained or immature. one even admits:

Aren't there about a dozen of these crates?
Yes. They all seem to address slightly different needs so I've added to the pile. Sorry.

so we’re shit outta luck right? not necessarily! we can roll our own duktape bindings without an unreasonable amount of effort. let’s do that here, in this post!

the code below is compatible with duktape 2.7.0.

1. low-level bindings

let’s create a rust program first, then grab a copy of duktape. while the duktape releases include the actual sources (not all in a single .c and .h) and build scripts we can use to create a customised flavour of duktape, we can just use the default flavour in src this time:

$ cargo new duktest
$ cd duktest
$ curl -O https://duktape.org/duktape-2.7.0.tar.xz
$ tar xf duktape-2.7.0.tar.xz
$ ls duktape-2.7.0/src
duk_config.h  duk_source_meta.json  duktape.c  duktape.h

we’re gonna use bindgen to generate most of our low-level bindings.

Cargo.toml

[build-dependencies]
bindgen = "0.72.1"

bindgen will use libclang to parse duktape.h. with nix, that looks like this:

shell.nix

with import <nixpkgs> {};
(mkShell.override { stdenv = clangStdenv; }) {
  name = "duktest-shell";
  LIBCLANG_PATH = lib.makeLibraryPath [ llvmPackages.clang-unwrapped.lib ];
}

the build script is almost entirely based on this example in the bindgen docs, but one important thing is to tell bindgen to only generate bindings for duktape.h, not a bunch of other standard library headers that will just cause us problems:

build.rs

// `error[E0428]: the name `FP_NAN` is defined multiple times`
// `error[E0428]: the name `FP_INFINITE` is defined multiple times`
// `error[E0428]: the name `FP_ZERO` is defined multiple times`
// `error[E0428]: the name `FP_SUBNORMAL` is defined multiple times`
// `error[E0428]: the name `FP_NORMAL` is defined multiple times`
// `warning: `extern` block uses type `u128`, which is not FFI-safe`
.allowlist_file(".*/duktape[.]h")

full build.rs

use std::env;
use std::path::PathBuf;

fn main() {
    let libdir_path = PathBuf::from("duktape-2.7.0/src")
        .canonicalize()
        .expect("cannot canonicalize path");

    let h_path = libdir_path.join("duktape.h");
    let c_path = libdir_path.join("duktape.c");
    let obj_path = libdir_path.join("duktape.o");
    let lib_path = libdir_path.join("libduktape.a");
    let (h_path, c_path, obj_path, lib_path) = (
        h_path.to_str().expect("unsupported path"),
        c_path.to_str().expect("unsupported path"),
        obj_path.to_str().expect("unsupported path"),
        lib_path.to_str().expect("unsupported path"),
    );

    // Tell cargo to look for shared libraries in the specified directory
    println!("cargo:rustc-link-search={}", libdir_path.to_str().unwrap());

    // Tell cargo to tell rustc to link our `duktape` library. Cargo will
    // automatically know it must look for a `libduktape.a` file.
    println!("cargo:rustc-link-lib=duktape");

    // Run `clang` to compile the `duktape.c` file into a `duktape.o` object file.
    if !std::process::Command::new("clang")
        .args(["-c", "-o", obj_path, c_path])
        .output()
        .expect("failed to spawn `clang`")
        .status
        .success()
    {
        panic!("failed to compile object file");
    }

    // Run `ar` to generate the `libhello.a` file from the `hello.o` file.
    if !std::process::Command::new("ar")
        .args(["rcs", lib_path, obj_path])
        .output()
        .expect("failed to spawn `ar`")
        .status
        .success()
    {
        panic!("failed to emit library file");
    }

    let bindings = bindgen::Builder::default()
        // `error[E0428]: the name `FP_NAN` is defined multiple times`
        // `error[E0428]: the name `FP_INFINITE` is defined multiple times`
        // `error[E0428]: the name `FP_ZERO` is defined multiple times`
        // `error[E0428]: the name `FP_SUBNORMAL` is defined multiple times`
        // `error[E0428]: the name `FP_NORMAL` is defined multiple times`
        // `warning: `extern` block uses type `u128`, which is not FFI-safe`
        .allowlist_file(".*/duktape[.]h")
        .header(h_path)
        // Tell cargo to invalidate the built crate whenever any of the
        // included header files changed.
        .parse_callbacks(Box::new(bindgen::CargoCallbacks::new()))
        .generate()
        .expect("failed to generate bindings");

    // Write the bindings to the $OUT_DIR/bindings.rs file.
    let out_path = PathBuf::from(env::var("OUT_DIR").unwrap());
    bindings
        .write_to_file(out_path.join("bindings.rs"))
        .expect("failed to write bindings!");
}

now we pull in those bindings from $OUT_DIR/bindings.rs into a module:

src/sys.rs

#![allow(unused)]
#![allow(nonstandard_style)]
include!(concat!(env!("OUT_DIR"), "/bindings.rs"));

src/main.rs

mod sys;

2. missing pieces

bindgen is great, but there are some things it just can’t do. for these things, we’ll have to write the bindings for by hand, translating from C to rust.

a big one is function-like macros. duktape uses a modest amount of these, thankfully not too many as of version 2.7.0, but they also reserve the right to change any API to a macro (or back to a real function) in any release.

so far, i’ve had to translate duk_eval, duk_eval_noresult, duk_peval, duk_peval_noresult, duk_eval_string, duk_peval_string, duk_eval_lstring, duk_peval_lstring, and duk_safe_to_string. here are a few of those:

duktape-2.7.0/src/duktape.h

#define duk_eval_string(ctx,src)  \
    ((void) duk_eval_raw((ctx), (src), 0, 0 /*args*/ | DUK_COMPILE_EVAL | DUK_COMPILE_NOSOURCE | DUK_COMPILE_STRLEN | DUK_COMPILE_NOFILENAME))

#define duk_peval_lstring(ctx,buf,len)  \
    (duk_eval_raw((ctx), buf, len, 0 /*args*/ | DUK_COMPILE_EVAL | DUK_COMPILE_NOSOURCE | DUK_COMPILE_SAFE | DUK_COMPILE_NOFILENAME))

#define duk_safe_to_string(ctx,idx) \
    duk_safe_to_lstring((ctx), (idx), NULL)

src/sys.rs

use std::{ffi::c_char, ptr::null_mut};

pub unsafe fn duk_eval_string(ctx: *mut duk_context, src: *const c_char) {
    unsafe {
        duk_eval_raw(
            ctx,
            src,
            0,
            0 /*args*/ | DUK_COMPILE_EVAL | DUK_COMPILE_NOSOURCE | DUK_COMPILE_STRLEN | DUK_COMPILE_NOFILENAME,
        );
    }
}

pub unsafe fn duk_peval_lstring(ctx: *mut duk_context, buf: *const c_char, len: duk_size_t) -> duk_int_t {
    unsafe {
        duk_eval_raw(ctx, buf, len, 0 /*args*/ | DUK_COMPILE_EVAL | DUK_COMPILE_NOSOURCE | DUK_COMPILE_SAFE | DUK_COMPILE_NOFILENAME)
    }
}

pub unsafe fn duk_safe_to_string(ctx: *mut duk_context, idx: duk_idx_t) -> *const c_char {
    unsafe {
        duk_safe_to_lstring(ctx, idx, null_mut())
    }
}

bindgen does support simple preprocessor constants, but for reasons unclear to me, not all of them make it through. the only one i’ve run into so far is DUK_VARARGS:

duktape-2.7.0/src/duktape.h

/* Indicates that a native function does not have a fixed number of args,
 * and the argument stack should not be capped/extended at all.
 */
#define DUK_VARARGS                       ((duk_int_t) (-1))

src/sys.rs

pub const DUK_VARARGS: duk_int_t = -1;

3. hello world

with this, we have enough to start running some javascript!

src/main.rs

mod sys;

use std::{ffi::CStr, io::{Write, stdout}, ptr::null_mut};

use crate::sys::*;

fn main() {
    unsafe {
        let ctx = duk_create_heap(None, None, None, null_mut(), None);
        duk_push_c_function(ctx, Some(nop), DUK_VARARGS);
        duk_put_global_string(ctx, c"nop".as_ptr());
        duk_push_c_function(ctx, Some(print), DUK_VARARGS);
        duk_put_global_string(ctx, c"print".as_ptr());

        duk_eval_string(ctx, c"nop()".as_ptr());
        duk_eval_string(ctx, cr#"print("hello js world\n")"#.as_ptr());

        duk_pop(ctx);
        duk_destroy_heap(ctx);

        println!("goodbye rust world");
    }
}

extern "C" fn nop(_: *mut duk_context) -> duk_ret_t {
    0
}

extern "C" fn print(ctx: *mut duk_context) -> duk_ret_t {
    unsafe {
        duk_push_string(ctx, c" ".as_ptr());
        duk_insert(ctx, 0);
        duk_join(ctx, duk_get_top(ctx) - 1);
        let text = duk_safe_to_string(ctx, -1);
        let text = CStr::from_ptr(text);
        let _ = stdout().write_all(text.to_bytes());
    }
    0
}

$ cargo run
hello js world
goodbye rust world

stay tuned!

next time, we’ll look at building some abstractions over these bindings, plus some tools to help us manage the interpreter’s value stack correctly.

it’s only nix if it comes from the utrecht region of the netherlands. otherwise it’s just sparkling input hashing

2025-08-22T12:11:17.079Z

how do we add caching to a static site generator? quoth last episode:

we need a way to know if a post has changed, or any of its attachments have changed, since it was last rendered or its metadata was last cached, and we also need to know what tag pages will need to be rerendered.

after a bunch of experimentation, i think nix is pretty much what we want, but nix doesn’t really work on windows, so it can’t be nix, so let’s build our own nix :3

disclaimer again

none of this has landed on the main branch yet, for good reason. it’s probably buggy as shit, so don’t run it on your real blog yet.

juicy numbers

in the autost#54 patch as of 218670e77f08e:

this blog (4803 threads)

this blog is my cohost archive plus a couple hundred newer posts. that’s 4803 posts (9816 if you count posts referenced by replies), and ~4.7 GB of attachments.

	no cache	cold cache	warm cache
rendering the site	828 ms	1731 ms	385 ms (−53.5%)
rendering after edit			438 ms
querying a tag	236 ms	1045 ms	90 ms (−61.8%)

note: percentages are relative to “no cache”

details: rendering the site

$ export RUST_LOG=autost=warn
$ time autost render --skip-attachments
9.36s user 1.30s system 1287% cpu 0.828 total
$ time autost render --skip-attachments --use-cache
10.26s user 3.97s system 821% cpu 1.731 total
$ time autost render --skip-attachments --use-cache
1.02s user 1.53s system 661% cpu 0.385 total

details: rendering again after editing tags in one of the posts

$ unset RUST_LOG
$ time autost render --skip-attachments --use-cache
INFO build{function=ReadFile id=b0143e6e233ed...}: autost::cache: building
INFO build{function=RenderMarkdown id=e0c542ce051e2...}: autost::cache: building
INFO build{function=FilteredPost id=5ad850559c540...}: autost::cache: building
INFO build{function=Thread id=799f2c033be6a...}: autost::cache: building
INFO build{function=RenderedThread id=844c6195470bd...}: autost::cache: building
INFO build{function=Thread id=9dda7da0c21ae...}: autost::cache: building
INFO build{function=RenderedThread id=eac9c7f17c6f8...}: autost::cache: building
INFO autost::cache: writing cache pack 5ad
INFO autost::cache: writing cache pack 9dd
INFO autost::cache: writing cache pack 799
INFO autost::cache: writing cache pack e0c
INFO autost::cache: writing cache pack eac
INFO autost::cache: writing cache pack 844
INFO autost::cache: writing cache pack b01
1.27s user 1.72s system 681% cpu 0.438 total

details: querying a tag

$ unset RUST_LOG
$ time autost cache test --list-threads-in-tag usb3sun --use-cache
37 threads in tag "usb3sun":
- "2023-01-06T18:08:35.092Z", "posts/787278.html", "SPARCstations have a unique serial-like interface "
- "2023-06-23T17:15:54.361Z", "posts/1742287.html", "usb3sun is an adapter that lets you connect usb ke"
- "2023-06-24T17:50:23.538Z", "posts/1650431.html", "SPARCstations have a unique serial-like interface "
[...]
INFO autost::cache: writing cache pack 372
0.49s user 0.93s system 410% cpu 0.347 total

$ rm -R cache
$ export RUST_LOG=autost=warn
$ time autost cache test --list-threads-in-tag usb3sun
1.77s user 0.27s system 864% cpu 0.236 total
$ time autost cache test --list-threads-in-tag usb3sun --use-cache
2.39s user 2.77s system 493% cpu 1.045 total
$ time autost cache test --list-threads-in-tag usb3sun --use-cache
0.07s user 0.30s system 412% cpu 0.090 total

big archive (146752 threads)

ok but what if i merged the cohost archives of everyone i followed into one very big archive? that’s 146752 posts (309140 if you count posts referenced by replies), and ~64 GB of attachments.

	no cache	cold cache	warm cache
rendering the site	24.90 s	35.75 s	11.18 s (−55.1%)
querying a tag	5.551 s	11.67 s	1.547 s (−72.1%)

note: percentages are relative to “no cache”

details: rendering the site

$ export RUST_LOG=autost=warn
$ time autost render --skip-attachments
258.96s user 49.31s system 1237% cpu 24.902 total
$ time autost render --skip-attachments --use-cache
309.85s user 72.00s system 1067% cpu 35.755 total
$ time autost render --skip-attachments --use-cache
33.57s user 44.87s system 701% cpu 11.188 total

details: querying a tag

$ unset RUST_LOG
$ time autost cache test --list-threads-in-tag usb3sun
37 threads in tag "usb3sun":
- "2023-01-06T18:08:35.092Z", "posts/787278.html", "SPARCstations have a unique serial-like interface "
- "2023-06-23T17:15:54.361Z", "posts/1742287.html", "usb3sun is an adapter that lets you connect usb ke"
- "2023-06-24T17:50:23.538Z", "posts/1650431.html", "SPARCstations have a unique serial-like interface "
[...]
2025-08-28T09:39:27.095295Z  INFO autost::cache: writing cache pack 857
14.77s user 18.37s system 381% cpu 8.692 total

$ rm -R cache
$ export RUST_LOG=autost=warn
$ time autost cache test --list-threads-in-tag usb3sun
34.12s user 5.86s system 720% cpu 5.551 total
$ time autost cache test --list-threads-in-tag usb3sun --use-cache
57.26s user 32.47s system 768% cpu 11.673 total
$ time autost cache test --list-threads-in-tag usb3sun --use-cache
1.47s user 5.72s system 464% cpu 1.547 total

anyway! i repeat, how do we add caching to a static site generator?

bad answer: timestamps

make is an incremental build system, where “incremental” means it only rebuilds things that have changed. it does this with file timestamps and a dependency graph: a file needs to be rebuilt if any of its dependencies are newer than it.

this is inadequate, because you can easily change the timestamp without changing the contents or change the contents without changing the timestamp. and good luck if you want to simultaneously cache more than one version of a build product.

this is also unnecessarily frugal, except for attachments, as we’ll see shortly.

better answer: hashes

nowadays reading files is pretty fast, as long as they’re stored on an ssd. let’s write a little microbenchmark to see just how fast it can be. all of these results were taken on my framework 13 with an AMD 7840U and a charger connected.

details: running the microbenchmark

$ autost cache benchmark <posts|posts-recursive|attachments> sum-paths-len <10..=100>
$ autost cache benchmark <posts|posts-recursive|attachments> sum-read-len <10..=100>
$ autost cache benchmark <posts|posts-recursive|attachments> blake3 <10..=100>
$ autost cache benchmark <posts|posts-recursive|attachments> blake3-mmap-rayon <10..=100>

this blog (4803 threads)

it looks like reading and hashing all of the posts doesn’t even double the time taken to walk the tree, but doing all of the attachments would incur an over 30x time penalty.

	time to walk	and read	and blake3
4803 posts	5.62 ms	9.69 ms	9.94 ms (1.76x)
9816 posts (recursive)	11.32 ms	18.24 ms	19.21 ms (1.69x)
5520 attachments	14.48 ms	458.0 ms	508.9 ms (35.1x)

note: multipliers are relative to “time to walk”

big archive (146752 threads)

again it’s less than a 2x penalty if we read and hash the posts, but an over 30x penalty for attachments.

	time to walk	and read	and blake3
146752 posts	197.6 ms	298.2 ms	313.5 ms (1.58x)
309140 posts (recursive)	388.9 ms	612.6 ms	630.3 ms (1.62x)
126626 attachments	326.4 ms	9417 ms	11027 ms (33.7x)

note: multipliers are relative to “time to walk”

another way of looking at it is, if we’re gonna have to read a bunch of files, we might as well hash them so we can easily check if they’ve changed. again this doesn’t make sense for attachments, whose contents are never actually read by autost render. i’m not sure i have a good solution for attachments yet.

best answer: nix?

rendering your site involves many steps:

making a list of posts
rendering the posts
- reading the post sources
- for posts in markdown, rendering the markdown
- parsing the post html into a dom tree
- extracting metadata
  - and rendering the referenced posts (if any)
  - and scanning for references to attachments
- applying transformations (e.g. making all images lazy loaded)
- serialising the dom tree back to html
- sanitising the html to “safe” tags and attributes
writing the output files
- copying the static files (e.g. style.css, script.js)
- hard linking the referenced attachments
- writing a page for each thread, containing that thread
- writing a page for each tag, containing all of its threads
- writing an atom feed for each tag, containing all of its threads
- writing a page for each built-in collection (e.g. index.html)

we don’t just want to cache the final build outputs (html pages and atom feeds), because the intermediate build steps are also useful for things like querying metadata. so how do we build a cache for all of the different kinds of build steps without descending into cache invalidation and ad-hoc serialisation hell?

many of these steps are pretty much functions (in the mathematical sense) that take some input and transform it to some output. in fact, all of them are, although some of them are easier to describe that way than others.

nix is a build system and package manager that uses this observation to build the largest ever single repository of software with efficient binary caching. how can we apply its ideas here?

imagine the process of loading a thread, just enough to know its metadata like its tags and what attachments it references:

for the last post in the thread
- read the post sources
- if the post is in markdown, render the markdown
- parse the html and extract metadata
repeat for each of the posts it references

now let’s say we described each of these steps as a function. note that for the caching to work correctly, it’s very important that aside from readFile(), all of the other functions are pure. they can only use their input arguments, which is why makeThread() can’t take one post and load all of the other posts it references from disk.

readFile(path, hash) → content reads path with expected hash hash, and returns the content of that file with that hash (more on the hash later)
renderMarkdown(markdown) → html renders the given markdown, and returns the rendered html
loadPost(html) → post (html, metadata) parses the given html into a dom tree, extracts the metadata, and returns a post
makeThread(posts…) → thread (posts…) takes the given posts, combines them into a thread, and returns that thread

each of those functions just Builds The Thing. to make them cacheable and allow us to run them with maximum parallelism, we want to be able to make a “build plan” that completely describes how to Build The Thing without doing all of the work of actually Building The Thing.

let’s describe the build planning as a second set of functions:

readFilePlan(path) → “readFile(path, hash)”
renderMarkdownPlan(path) → “renderMarkdown(readFile(path, hash))”
loadPostPlan(path) →
- | “loadPost(renderMarkdown(readFile(path, hash))” if markdown
- | “loadPost(readFile(path, hash))” if html
makeThreadPlan(path) → “makeThread(loadPost(…), loadPost(…), …)”

one problem we run into with makeThreadPlan() is that we need the metadata of one of the posts to know what other loadPost() calls to include in the result. this is unfortunate, because it means we can’t entirely avoid Building The Thing when doing build planning.

now here’s the magic!

let’s say we’re building the thread for 10000216.md, which replies to 10000215.md. makeThreadPlan(10000216.md) returns the build plan below. it was relatively easy to compute, and it completely describes how to load the thread from the files that make up the thread.

“makeThread(loadPost(renderMarkdown(readFile(10000216.md, 44062ae08bb67…))), loadPost(renderMarkdown(readFile(10000215.md, 8adba2770e58c…))))”

remember the “(more on hash later)” from earlier? if the build plan for a thread includes the hash of each of its posts, then we can hash the build plan and get an ID that changes if and only if any of the posts have changed.

and with that kind of unique ID for each thing we build, caching is easy! since the contents of the build plan for a given ID can’t possibly change (by definition), and the contents of the output really shouldn’t change (if we wrote our build logic correctly), we never need to delete or update anything, except maybe to free up disk space.

to me, this is the essence of nix, and i had great fun capturing that essence in our funny little static site generator. if you wanna see how this ersatz nix works, check out src/cache.rs and src/cache/drv.rs ^w^

that’s the post. but if you’re hungry for even more detail, read on…

performant store

processing thousands or even hundreds of thousands of posts is hard work. some things that helped with cache storage bottlenecks:

staring at hundreds of flamegraphs
switching from sqlite to plain old files
switching from json to bincode
switching from tokio async to rayon sync
- dedicated rayon thread pools for writing files with 4x the threads, since they will spend most of their time in syscalls waiting for i/o
- explicitly creating a thread pool for normal work, since rayon otherwise just uses the thread pool of the innermost fork-join scope
caching build products in memory, then writing that to disk
- storing native rust types in the memory cache, so we can take build product serialisation off the critical path
- offloading writes to dedicated thread pools, so we can take build product file writing off the critical path
moving from one file per build product, to 4096 “packs” of cache data
- splitting the memory cache into 4096 “packs” as well
- splitting the dirty bit into 4096 dirty bits as well
reducing overheads of syncing memory caches with the disk store
- switching from DashMap to HashMap, because it’s cheaper when we have thousands of them, and still performs well enough when sharded
- lazy deserialisation of cache items, loading as Vec<u8> until needed

tag indexes

autost cache test --list-threads-in-tag builds a “tag index” to more efficiently look up posts having the given tag, but it was hard to do this in a way that wasn’t slower than just computing the metadata from scratch. two things made it possible.

the build plans themselves were expensive, because for each evaluation we had to deserialise a big tree of inputs from ThreadDrv to FilteredPostDrv to RenderMarkdownDrv and finally down to ReadFileDrv. plus load their outputs into the memory cache for no reason. we fixed this by hiding everything other than ReadFileDrv from the build plan, inferring them within the TagIndexDrv builder. idk about this… it feels like cheating or subverting how nix is supposed to work?

the obvious representation of the tag index is a HashMap or BTreeMap, but this has to be deserialised in full on first use without any parallelism, which is a waste unless our query involves the whole dataset. instead we can build a tiny sqlite database and serialise it to Vec<u8>, then serialise that as the build output. since sqlite can obviously query a table without building a whole map in memory, this is much faster.

static site generators can have little a database, as a treat

2025-08-16T03:44:10.012Z

i’ve started working on autost again, with the hope i can tackle some of the problems i wrote about in april. so far, i’ve been cautiously experimenting with adding a database, which again, seems to be unavoidable if we want autost to be anything more than a Cohost Archive You Can Keep Posting To. but this can solve other problems too!

disclaimer

none of this has landed on the main branch yet, for good reason. it’s probably buggy as shit, so don’t run it on your real blog yet.

post ids

to publish a new post, we need to choose a new post id, and our current approach is pretty minimalist (derogatory):

let id = 10000000
try to create posts/{id}.md
if we can’t because it already exists:
1. let id = id + 1
2. go to step 2

this is obviously inefficient, because it gets slower and slower the more posts you’ve made, and it doesn’t even work correctly if your last post was a .html post.

we could cache the last post id in a text file, and that sounds easy at first, but storing data in text files gets complicated fast. now you have to worry about things like

how to avoid data loss if a power outage happens while writing
how to version and migrate the file format when you need to extend it
how to make related changes to multiple files together or not at all

and all of these are solved problems if we use an actual database. here’s a simple sqlite database that solves our post id problems:

CREATE TABLE "post" (
    "post_id" INTEGER PRIMARY KEY AUTOINCREMENT  -- e.g. 10000250
    , "path" TEXT NOT NULL                       -- e.g. '10000250.md'
    , "rendered_path" TEXT NULL UNIQUE           -- e.g. '10000250.html'
);

since the "rendered_path" is UNIQUE, we can’t accidentally create a "path" = '10000250.md' and a "path" = '10000250.html' that both have "rendered_path" = '10000250.html'. and choosing the next post id so we can publish a new post is easy:

BEGIN;
-- no "post_id", so we generate it with AUTOINCREMENT.
-- empty "path", because we don’t know it yet.
INSERT INTO "post" ("path") VALUES ('');
-- now we check the “last insert id”, which is done outside sql.
-- let’s say it’s 10000250.
UPDATE "post" SET "path" = '10000250.md' WHERE "post_id" = 10000250;
COMMIT;

check out autost#53 for the complete patch.

caching

right now, answering questions like “what posts are tagged #birds?” takes about as long as rendering your whole site, because in both cases we have to read all of your posts, parse the html, and extract the metadata. and when you publish or edit a post, we have to render your whole site from scratch.

this is generally how static site generators work, and Computer Touchers like that because having your whole blog be a bunch of text files is (a) elegant and (b) works well with version control. but the average Computer Toucher also has one (1) blog post, titled “New Year, New Blog, or: How I Learned To Stop Worrying and Switch From Twelvety to Sext.js”, so we should take their opinions with a grain of salt.

static site generators do have other benefits though. they’ve got good data portability, and static sites are easy to host on any web server, even web servers that are free because Big Coding is using them as a loss leader.

so maybe we want something in between, where your post files are still the primary source of truth, but we cache the metadata that we extract from them in the database, or we cache the fact that the post has already been rendered.

that will leave us with one of the two hard problems in computer science: now we need a way to know if a post has changed, or any of its attachments have changed, since it was last rendered or its metadata was last cached, and we also need to know what tag pages will need to be rerendered. ~~hmm. is this just nix?~~

what is an interesting post? the greatest question in the history of autost, locked after 12239 commits of organic growth

2025-04-19T07:39:11.582Z

autost was written in a month-long daze after cohost announced their shutdown. it started out as a way for me to archive my own chosts on shuppy.org. since the plan all along was for me to continue chosting from there, that came next. then it gained the ability to import posts from other blogs, archive the chosts of everyone you follow, and archive chosts liked by you and your friends.

buried deep within the core of autost is some gnarly logic that decides which posts should be rendered, which posts should be unlisted or publicly visible (“interesting” posts), and which tags should have their own tag pages (“interesting” tags).

the problems i wanted to solve initially were:

i want to archive my own chosts and continue chosting on the same site
i want to curate which of my archived chosts are publicly visible
we need a way to store the posts that are being replied to

but the way i solved them was a bunch of hardcoded rules:

posts you make with autost are publicly visible
- in practice, this means posts authored by your [self_author] href
your own archived chosts in “interesting” tags are public too
- in practice, this means posts authored by your other_self_authors
- it’s “your own” if you authored the last post in the thread
- the “last post” does not include transparent shares
- “interesting” tags are those linked in the top nav section
specific archived chosts can be made public or unlisted
posts in subdirectories are ignored except for rendering replies

there are so many problems with this model:

you can’t put any posts in subdirectories
you can’t opt out of the rules for archived chosts. autost cohost-archive hacks around this by setting the [self_author] href of each archive to a cohost url
you can’t organise posts in any way other than tags. autost cohost-archive hacks around this by making a separate site for each project you follow
there’s no clear way to know why a post exists, or what you want to do with it, other than this big pile of emergent behaviour
since every site has to abide by the same rules, any changes to the rules could break existing sites or change how they behave

we need to move away from these hardcoded rules. i think it’s time to sit down and actually define what we want autost to be capable of, and design a data model around that. this will require changes to existing sites, but we can build automated migrations or migration tools to upgrade those sites.

when you render your site, we should ask questions like “do we render this post?”, “is this post publicly visible?”, and “which tags have tag pages?”, but let whoever created the post (autost server, autost cohost2autost, autost import, or you) decide the answers.

we’ll probably want to store those answers in a database. it’s all files right now, which is like a database, except harder to extend and harder to safely mix human changes and machine changes. having a database may make it easier to extend autost to do other things with archived chosts, like organise the chosts of everyone you follow into a single browsable archive, or generate a page of your liked chosts.

we may even want to move posts that only exist for rendering replies out of the posts directory and into a separate store, like the attachment store. this will require changes to the <link rel=references> in existing posts though, so it would need to be done with extra care.

autost 1.4.0: akkoma importer, lazy loading, and a windows fix

2025-03-02T13:00:50.213Z

this update (full changelog) includes several minor improvements that landed after the big cohost shutdown scramble (rip eggbug). most notably:

all <img> elements are now lazy loaded, massively reducing the size of large threads pages
you can now reblog posts from akkoma instances, including those on the website league

autost server now lets you click [+] on any tag to post in it
autost server is no longer broken on windows (@ar1a, #39)

so… real talk. a handful of you actually use (or are considering using) autost for more than just archiving your chosts, and i think that’s wonderful. but maybe at this point you’re wondering if the project’s days are numbered.

development has been pretty slow the past couple of months, because i fell in love with a fox, and i needed a break after the last crunch. but i don’t consider autost finished, and i intend to keep working on it, just at a healthier pace.

higher priorities include being able to upload images from the post composer and compose posts from another device. but longer term, i wonder if there’s a niche for autost to be an easier kind of blog engine.

most blogging solutions are either saas, complex self-hosted (wordpress), or require you to be a computer-toucher and have frontend knowledge (static site generators). what if you could just run an app on your computer and get an editor for your blog? what if you could compose posts, follow and reblog your friends, and deploy your blog to neocities or some other host, all without touching the command line?

if that interests you, let me know at autost@shuppy.org or on the league :)

polycule-driven development

2025-03-02T12:36:03.196Z

shuppy (@delan) — [archived] 2023-09-04T16:19:21.424Z

polycule-driven development

(@ariashark + @bark)

#polyamory #webdev #github #charming

archive your chosts and the chosts you care about

2024-12-29T08:32:56.792Z

only a couple days left until cohost goes down! why not archive the chosts you care about, so you can look back on them without relying on wayback machine?

cohost-dl will do your chosts plus your liked chosts, while autost will do both of those plus chosts by people you follow! here’s how you can use autost for that:

install autost
log in to cohost, press F12, then grab your “connect.sid” cookie (see screenshot or consult someone with stickers on their laptop)
if you own more than one page, switch to the one you want to archive
get archiving!
repeat from step 3 until you’ve archived all of your pages

please email me on autost@shuppy.org if you have any questions! good luck ^w^

Cookies > cohost.org > connect.sid > copy the value. chrome devtools: Application tab > Cookies > cohost.org > connect.sid > copy the value. don’t have safari sorry" loading="lazy">

autost 1.0, now rebloggable with h-entry

2024-10-11T11:50:24.625Z

>>> download <<<

[readme] [changes in 1.0.0]

autost is a cohost-compatible archive and blog engine, which i use right here on shuppy.org. for another autost blog, check out LotteMakesStuff dot PINK :D

your posts are now rebloggable, thanks to the microformats2 h-entry format! for the most consistent experience, be sure to set your [self_author] display_handle setting to a domain name.

you can now reblog posts with autost import. for example, to reblog Natalie’s post Reblogging posts with h-entry:

$ autost import https://nex-3.com/blog/reblogging-posts-with-h-entry/
2024-10-03T15:30:33.221087Z  INFO run_migrations: autost::migrations: hard linking attachments out of site/attachments
2024-10-03T15:30:33.411823Z  INFO autost::command::import: writing RelativePath { inner: "posts/imported/1.html", kind: Other }
2024-10-03T15:30:33.422458Z  INFO autost::command::import: click here to reply: http://[::1]:8420/posts/compose?reply_to=imported/1.html

you can now create attachments from local files with autost attach. for example:

$ autost attach ~/Downloads/atom{1,2}.png
2024-10-11T11:47:39.031943Z  INFO run_migrations: autost::migrations: hard linking attachments out of site/attachments
2024-10-11T11:47:39.137328Z  INFO autost::command::attach: created attachment: <attachments/d9c42006-07ca-4027-8abb-dfa670a7e637/atom1.png>
2024-10-11T11:47:39.137477Z  INFO autost::command::attach: created attachment: <attachments/887df41f-a1ce-4e66-8a9f-09ef1f6fedca/atom2.png>

warning: autost attach does not strip any exif data yet, so be careful not to leak your gps location in any photos you upload! use exiftool or a photo editor to check.

the atom output is also way better now! most importantly, threads with multiple posts are no longer an unreadable mess:

before

→

after

note that your subscribers will need to convince their clients to redownload your posts, or unsubscribe and resubscribe, to see these effects on existing posts.

subscribe here for updates, or email me on autost@shuppy.org if you have any questions. enjoy :3

untitled post by shuppy.org

2024-10-03T15:31:32.511Z

Natalie (nex-3.com) — [archived] 2024-09-27T07:38:00Z

Reblogging posts with h-entry
Natalie wrote:

Once I add the ability to embed arbitrary blog posts from other blogs on here it's over. I'm gonna be reblogging like a wild animal. Y'all are gonna have your eyes blown clean outta your heads.

Thrilled to announce that I now have this up and running, at least in its most basic aspect. The embed above is automatically generated and pulled down directly from the source post. Nothing in this is specific to my blog; I can also do it with someone else's. By way of example, please enjoy this post from my beautiful wife:

Liz wrote:

there is a very specific feeling of relief upon realizing I don’t need to hurry to finish a library book before it’s due, because I definitely will want to buy a copy for future reference and cross-checking.

(the book in question is Gossip Men: J. Edgar Hoover, Joe McCarthy, Roy Cohn, and the Politics of Insinuation by Christopher M. Elias)

Injecting embeds

Here's what the embed looks like in my blog source right now:
{% genericPost "https://nex-3.com/blog/once-i-add-the/",
    time: "2024-09-20T07:06:00Z",
    tags: "#meta",
    author: "Natalie",
    authorUrl: "/",
    authorAvatar: "/assets/avatar.webp" %}
  <p>
    Once I add the ability to embed arbitrary blog posts from other blogs on
    here it's over. I'm gonna be reblogging like a wild animal. Y'all are gonna
    have your eyes blown clean outta your heads.
  </p>
{% endgenericPost %}
I have a template for the embed, some CSS to style it, and a little custom Liquid tag to bring it all together. But the real magic is in how I generate the genericPost in the first place. Here's what the original source looks like before I run my embed injector on it:
 
https://nex-3.com/blog/once-i-add-the/
That's it! Just a URL surrounded by empty lines. The injector pulls down the webpage, extracts critical information about the blog post, and replaces the URL with a call to genericPost. My Letterboxd and Cohost embeds work the same way, with their own custom templates and metadata that let me match the style of the original websites.

Structured post data with h-entry

But I did say that this wasn't specific to my blog. With Letterboxd and Cohost, I've just hard-coded their HTML structure. I can't rely on that if I want to get information from any old blog, though. They all use different HTML structures!

So instead, I'm making use of the h-entry microformat. This is a tiny little specification that defines a way to mark up an existing post to indicate metadata in the existing HTML structure. At its simplest, it's just a few class names annotated with the HTML. Here's the simplified HTML for the post above:
<article class="h-entry">
  <p class="attribution">
    <a href="/blog/once-i-add-the/" rel="canonical" class="u-url">
      Posted
      <time datetime="2024-09-20T07:06:00Z" class="dt-published">
        20 September 2024
      </time>
      by
      <span class="p-author h-card">
        <span class="p-name">Natalie</span>
        <data class="u-url" value="/"></data>
        <data class="p-logo" value="/assets/avatar.webp"></data>
      </span>
    </a>
  </p>

  <div class="e-content">
    <p>
      Once I add the ability to embed arbitrary blog posts from other
      blogs on here it's over. I'm gonna be reblogging like a wild animal.
      Y'all are gonna have your eyes blown clean outta your heads.
    </p>
  </div>

  <ol class="tags"><li><a href="/tag/meta" class="p-category">meta</a></li></ol>
</article>
Here are the special class names I'm using:

h-entry wraps the entire thing, indicating that it's a post.

u-url goes on a link and indicates the post's canonical URL.

dt-published goes on a <time> element and indicates when the post was made.

p-author can just be the author's name, but I've made it into a whole h-card with the following metadata:

p-name is my name.

u-url is my personal URL, which is to say this website.

p-logo is the URL of my avatar, so people reblogging me have something to show by my posts.

p-name isn't used here because this post is untitled, but if it had a title that would be the class for it.

e-content is the HTML content of the post itself.

p-category is the name of a tag associated with the post.

Of these, only h-entry, u-url, and e-content are really critical. There are a handful of other features defined in the spec, including a way to indicate which post you're replying to, that you should check out if you're interested. But these are the most critical.

Make your posts rebloggable!

You should do this too! If you can edit your blog's post layout, it's extremely easy. Just add those classes to the appropriate places, and you're off to go. If you don't already have an HTML element for some piece of information, you can add invisible <data> tags like I did above to provide the info to consumers without changing the way your post looks to readers.

Even if your blog doesn't allow you to edit the layout, if you can add HTML to the posts you can do this by hand. There's no need to use the specific <article> or <p> tags I do above... just divs will work. You can even make your rebloggable content different than the original, which I'm planning to do to avoid having my embeds look too weird if they get reblogged.

If you end up adding h-entry support, or even better if you end up making use of mine, drop me a comment and let me know!
#meta #web

i’ve started adding h-entry support to autost! still need to rewrite relative urls and cache embedded images, but i can already import and share/reply to posts:

$ autost import https://nex-3.com/blog/reblogging-posts-with-h-entry/
2024-10-03T15:30:33.221087Z  INFO run_migrations: autost::migrations: hard linking attachments out of site/attachments
2024-10-03T15:30:33.411823Z  INFO autost::command::import: writing RelativePath { inner: "posts/imported/1.html", kind: Other }
2024-10-03T15:30:33.422458Z  INFO autost::command::import: click here to reply: http://[::1]:8420/posts/compose?reply_to=imported/1.html

autost: archive your chosts, write new posts, reply to posts

2024-10-01T07:31:13.085Z

>>> download <<<

[readme] [changes in 0.3.0]

autost is a cohost-compatible archive and blog engine. it’s pretty much a static site generator that can import chosts, but i plan to eventually add a rss/atom feed reader and chronological timeline.

you no longer need to know how to use git or cargo to use it! and now you can write posts and reply to posts in the composer!

cohost is shutting down soon, so go here for updates or email me on autost@shuppy.org if you have any questions. enjoy :3

untitled post by @delan

2024-09-29T23:45:20.039Z

shuppy (@delan) — [archived] 2024-09-27T04:03:45.836Z

autost: a cohost-compatible archive and blog engine

…and hopefully feed reader?

want to archive your chosts on your website but have too many for the cohost web component? want something like cohost-dl except you can keep posting?

autost is my take on that. you can see what it looks like on shuppy.org:

it’s pretty much a static site generator that can import chosts. some features:

perfect html rendering of your chosts and css crimes
(css things like bounce and spin work too, but not yet 100%)

include and exclude specific tags or chosts in your archive

tag your archived chosts without editing them on cohost

automatically rename and add tags based on the existing tags

make new posts with mostly-cohost-compatible markdown
(there’s even a post composer with live preview! but it’s very much a wip)

i’ve only had a couple weeks to work on this, so it’s not the most polished, and for now you need a bit of command line knowledge to get it going.

but my goal here is to make a blog engine with the same posting and reading experience as cohost, where you can follow people with rss/atom feeds, see their posts on a chronological timeline, and share their posts on yours.

does that vibe with anyone? should i keep working on this? let me know!

#The Cohost Global Feed #cohost utilities #Cohost Shutdown #data export #autost

shuppy (@delan) — [archived] 2024-09-29T12:12:41.252Z

autost 0.2.0: it has a compose button now

some big usability improvements in this one, plus a couple of breaking changes!

breaking changes: cohost2autost no longer takes the ./posts and site/attachments arguments, and render no longer takes the site argument. these paths were always ./posts, site/attachments, and site (#8).

when viewing your site with the server, there’s a compose button now (#7).

in the server, you also get a link to your site in the terminal now, and you no longer get a NotFound error if your base_url setting is not /.

you no longer have to type RUST_LOG=info to see what’s happening (#5).

attachment filenames containing % and ? are now handled correctly (#4).

questions and contributions are welcome! if you run into any problems, or want to help out, let me know on github or email me on autost@shuppy.org.

oh and go here for updates once cohost noposts :)

#The Cohost Global Feed #cohost utilities #Cohost Shutdown #data export #autost

rebug for the morning crew! please let me know if autost works or doesn't work for you, and go here for updates ^w^

autost 0.2.0: it has a compose button now

2024-09-29T12:12:41.252Z

shuppy (@delan) — [archived] 2024-09-27T04:03:45.836Z

autost: a cohost-compatible archive and blog engine

…and hopefully feed reader?

want to archive your chosts on your website but have too many for the cohost web component? want something like cohost-dl except you can keep posting?

autost is my take on that. you can see what it looks like on shuppy.org:

it’s pretty much a static site generator that can import chosts. some features:

perfect html rendering of your chosts and css crimes
(css things like bounce and spin work too, but not yet 100%)

include and exclude specific tags or chosts in your archive

tag your archived chosts without editing them on cohost

automatically rename and add tags based on the existing tags

make new posts with mostly-cohost-compatible markdown
(there’s even a post composer with live preview! but it’s very much a wip)

i’ve only had a couple weeks to work on this, so it’s not the most polished, and for now you need a bit of command line knowledge to get it going.

but my goal here is to make a blog engine with the same posting and reading experience as cohost, where you can follow people with rss/atom feeds, see their posts on a chronological timeline, and share their posts on yours.

does that vibe with anyone? should i keep working on this? let me know!

#The Cohost Global Feed #cohost utilities #Cohost Shutdown #data export #autost

shuppy (@delan) — [archived] 2024-09-27T15:18:30.415Z

some big usability improvements in this one, plus a couple of breaking changes!

breaking changes: cohost2autost no longer takes the ./posts and site/attachments arguments, and render no longer takes the site argument. these paths were always ./posts, site/attachments, and site (#8).

when viewing your site with the server, there’s a compose button now (#7).

in the server, you also get a link to your site in the terminal now, and you no longer get a NotFound error if your base_url setting is not /.

you no longer have to type RUST_LOG=info to see what’s happening (#5).

attachment filenames containing % and ? are now handled correctly (#4).

questions and contributions are welcome! if you run into any problems, or want to help out, let me know on github or email me on autost@shuppy.org.

oh and go here for updates once cohost noposts :)

autost: a cohost-compatible archive and blog engine

2024-09-27T04:03:45.836Z

…and hopefully feed reader?

want to archive your chosts on your website but have too many for the cohost web component? want something like cohost-dl except you can keep posting?

autost is my take on that. you can see what it looks like on shuppy.org:

it’s pretty much a static site generator that can import chosts. some features:

perfect html rendering of your chosts and css crimes
(css things like bounce and spin work too, but not yet 100%)
include and exclude specific tags or chosts in your archive
tag your archived chosts without editing them on cohost
automatically rename and add tags based on the existing tags
make new posts with mostly-cohost-compatible markdown
(there’s even a post composer with live preview! but it’s very much a wip)

i’ve only had a couple weeks to work on this, so it’s not the most polished, and for now you need a bit of command line knowledge to get it going.

but my goal here is to make a blog engine with the same posting and reading experience as cohost, where you can follow people with rss/atom feeds, see their posts on a chronological timeline, and share their posts on yours.

does that vibe with anyone? should i keep working on this? let me know!

active ingredients: path manipulation 70% w/w

2024-09-25T16:02:42.280Z

shuppy (@delan) — [archived] 2024-09-16T16:11:32.249Z

made a website so i can continue chosting

>>> shuppy.org <<<
or add me to your feed reader!
>>> atom feed <<<
(need a feed reader? here are a bunch)
(it works well in FreshRSS and thunderbird)

oh and i’ve updated my contact details!! say hi and send me stuff :3

i had an old website, but for a variety of reasons it really didn’t feel like a good successor to my cohost page. i feel like what i really want is to Continue Chosting next month, just on my own website? (well no, what i really want is cohost, but failing that…)

so i made this! and many of my old chosts are there now too! i’ll need a bit more time to figure out how to actually make new posts (lol) and have them play nice with my old chosts, but hey, now you know where to find me once i do o/

#Cohost Shutdown #shuppy.org #autost

i gotta pay down this tech debt asafp or this thing is gonna burst into treats.

also, clicking “publish” in the composer redirects you to the post now, but i figured this would be funnier than posting a screenshot of a redirect.

farewell programmer art, hello cohost colors

2024-09-25T14:44:31.092Z

shuppy (@delan) — [archived] 2024-09-16T16:11:32.249Z

made a website so i can continue chosting

>>> shuppy.org <<<
or add me to your feed reader!
>>> atom feed <<<
(need a feed reader? here are a bunch)
(it works well in FreshRSS and thunderbird)

oh and i’ve updated my contact details!! say hi and send me stuff :3

i had an old website, but for a variety of reasons it really didn’t feel like a good successor to my cohost page. i feel like what i really want is to Continue Chosting next month, just on my own website? (well no, what i really want is cohost, but failing that…)

so i made this! and many of my old chosts are there now too! i’ll need a bit more time to figure out how to actually make new posts (lol) and have them play nice with my old chosts, but hey, now you know where to find me once i do o/

#Cohost Shutdown #shuppy.org #autost

shuppy.org now works in fluent-reader

2024-09-24T14:49:50.330Z

eggbug asked…

I use (fluent-reader)[https://github.com/yang991178/fluent-reader]

thanks, i’ve fixed that now!

fluent-reader seems to need a <base> tag in each <content> to tell it how to resolve relative urls. it doesn’t support xml:base, which i’ve now reported to the devs, but it turns out my feeds didn’t have that either. my bad!

yes!! this is what we need!!

2024-09-23T09:52:01.631Z

Natalie (@nex3) — [archived] 2024-09-23T06:19:47.460Z

"Reblogging" as a Verb on the Modern Web
One of my main goals with my new blog is to preserve the concept of "reposting" that's both a successful and, I think, desirable aspect of the "social network" model. It's a great way to spread visibility of other people's work and engage in conversation across different blogs. It's a big part of why I'm putting so much work into the concept of "embeds", as for Cohost and Letterboxd.

These embeds are quite automated: I just drop a link in my blog and as long as it's one of the supported sites, a little post-processing tool automatically fetches the information and adds it as template params. All in all, I'm pretty happy with this flow, but there's one critical problem: it requires me to write a separate scraper for every website.

The great strength of independent websites for social interaction is that everyone's can be as different as they'd like, but that also means that there's no standard way to understand or interact with them. Even something as simple as "what is the text of the post at this URL" is difficult to answer in general. We'd all hope that everyone would use semantic HTML so perfectly that you could just read the contents of the outermost <article> tag on the page, but the real world is never so pristine. We need a more explicit way to indicate the critical prose and metadata for something that's considered a "post".

I've been rolling this problem around in my head for the past week, and I have what is at least the germ of an idea. The core observation is that RSS and Atom feeds already have all the metadata that's strictly necessary for something like this, but they suffer from being time-limited—their use-case is focused on syndication, so a website can only be expected to have its most recent posts available in such a nice format.

So imagine if we co-opted this format for use in something more persistent. Something like:
<link rel="somens:reblog" href="post.xml">
made with @nex3's syntax highlighter

where the XML file looks like
<?xml version="1.0" encoding="utf-8"?>
<reblog xmlns="https://nex-3.com/somens" xmlns:atom="http://www.w3.org/2005/Atom">
  <atom:link href="https://nex-3.com/blog/once-i-add-the/post.xml" rel="self"/>
  <atom:link href="https://nex-3.com/"/>
  <atom:entry>
    <atom:link href="https://nex-3.com/blog/once-i-add-the/" rel="alternate"/>
    <atom:id>https://nex-3.com/blog/once-i-add-the/</atom:id>
    <atom:published>2024-09-20T07:06:00Z</atom:published>
    <atom:updated>2024-09-20T07:06:00Z</atom:updated>
    <atom:author>
      <atom:name>Natalie</atom:name>
      <atom:uri>https://nex-3.com/</atom:uri>
    </atom:author>
    <atom:category>meta</atom:category>
    <atom:content type="html">&lt;p&gt;Once I add the ability to embed arbitrary blog posts from other blogs on here it&#39;s over. I&#39;m gonna be reblogging like a wild animal. Y&#39;all are gonna have your eyes blown clean outta your heads.&lt;/p&gt;</atom:content>
  </atom:entry>
</reblog>
Do you see my vision? This is mostly just stealing structure from Atom, but with the explicit guarantee that any logical "entries" on the page in question will also exist in the "reblog" XML. Maybe there should be some sort of way to explicitly link the two as well, idk. But in the general case, a page with one post would link to a single-post XML file which could then be used as the source for a reblog. Wouldn't that be neat?

Edit: Something like this already exists! Hooray!
#rss #atom #web

i’ve been working on something similar that will rely on this (see #shuppy.org, autost). my posts are in an html-fragment-based format, which looks something like this:

<link rel="references" href="7801740/7742758.html">
<link rel="references" href="7801740/7800234.html">
<meta name="title" content="the post hole works">
<meta name="published" content="2024-09-22T14:24:07.963Z">
<link rel="author" href="https://cohost.org/delan" name="shuppy (@delan)">
<meta name="tags" content="shuppy.org">

[post content in markdown or html here]

made with @nex3's syntax highlighter

this is not the best interchange format for others to reblog, though, because it relies on you rendering markdown the same way as me, and you would need to fetch the posts i’m replying to separately. your idea would really help with that!

one suggestion: it would be nice if the format you came up with was also usable in the “whole blog” feed, where each <atom:entry> is a post, including the posts it replies to. i started thinking about atom-compatible ways of doing that in this chost.

the post hole works

2024-09-22T14:24:07.963Z

shuppy (@delan) — [archived] 2024-09-16T16:11:32.249Z

made a website so i can continue chosting

>>> shuppy.org <<<
or add me to your feed reader!
>>> atom feed <<<
(need a feed reader? here are a bunch)
(it works well in FreshRSS and thunderbird)

oh and i’ve updated my contact details!! say hi and send me stuff :3

i had an old website, but for a variety of reasons it really didn’t feel like a good successor to my cohost page. i feel like what i really want is to Continue Chosting next month, just on my own website? (well no, what i really want is cohost, but failing that…)

so i made this! and many of my old chosts are there now too! i’ll need a bit more time to figure out how to actually make new posts (lol) and have them play nice with my old chosts, but hey, now you know where to find me once i do o/

#Cohost Shutdown #shuppy.org #autost

shuppy (@delan) — [archived] 2024-09-22T07:13:00.633Z

now working on the post hole

#shuppy.org #autost

>>> https://shuppy.org/posts/10000000.html <<<

hello from autost

2024-09-22T14:15:55.103Z

is this thing on?

now working on the post hole

2024-09-22T07:13:00.633Z

shuppy (@delan) — [archived] 2024-09-16T16:11:32.249Z

made a website so i can continue chosting

>>> shuppy.org <<<
or add me to your feed reader!
>>> atom feed <<<
(need a feed reader? here are a bunch)
(it works well in FreshRSS and thunderbird)

oh and i’ve updated my contact details!! say hi and send me stuff :3

i had an old website, but for a variety of reasons it really didn’t feel like a good successor to my cohost page. i feel like what i really want is to Continue Chosting next month, just on my own website? (well no, what i really want is cohost, but failing that…)

so i made this! and many of my old chosts are there now too! i’ll need a bit more time to figure out how to actually make new posts (lol) and have them play nice with my old chosts, but hey, now you know where to find me once i do o/

#Cohost Shutdown #shuppy.org #autost

tag synonyms and tag implications

2024-09-19T09:14:00.737Z

shuppy (@delan) — [archived] 2024-09-16T16:11:32.249Z

made a website so i can continue chosting

>>> shuppy.org <<<
or add me to your feed reader!
>>> atom feed <<<
(need a feed reader? here are a bunch)
(it works well in FreshRSS and thunderbird)

oh and i’ve updated my contact details!! say hi and send me stuff :3

i had an old website, but for a variety of reasons it really didn’t feel like a good successor to my cohost page. i feel like what i really want is to Continue Chosting next month, just on my own website? (well no, what i really want is cohost, but failing that…)

so i made this! and many of my old chosts are there now too! i’ll need a bit more time to figure out how to actually make new posts (lol) and have them play nice with my old chosts, but hey, now you know where to find me once i do o/

#Cohost Shutdown #shuppy.org #autost

to better curate my chosts even after cohost shuts down, i can not only tag chosts without editing them, but also rename tags and add related tags automatically. for example:

#photography now includes all of my birds and wildlife shots
#watching now includes all of my tv and film chosting
#reverse engineering now includes all of my ghidra chosts

(the blog software is here, if you wanna play around with it)

untitled post by @delan

2024-09-18T16:32:29.265Z

eggbug asked…

Just thought I would let you know that your rss page uses relative urls for the photos, which prevents my rss reader from being able to display them. I am not sure if this is an issue with my own rss reader or if this is something that affects all rss readers.

thanks for the heads up! i've been testing with freshrss so far, if you can tell me what reader you use that would be really helpful

i can now add tags to my chosts without editing them >:3

2024-09-18T12:10:39.594Z

shuppy (@delan) — [archived] 2024-09-16T16:11:32.249Z

made a website so i can continue chosting

>>> shuppy.org <<<
or add me to your feed reader!
>>> atom feed <<<
(need a feed reader? here are a bunch)
(it works well in FreshRSS and thunderbird)

oh and i’ve updated my contact details!! say hi and send me stuff :3

i had an old website, but for a variety of reasons it really didn’t feel like a good successor to my cohost page. i feel like what i really want is to Continue Chosting next month, just on my own website? (well no, what i really want is cohost, but failing that…)

so i made this! and many of my old chosts are there now too! i’ll need a bit more time to figure out how to actually make new posts (lol) and have them play nice with my old chosts, but hey, now you know where to find me once i do o/

#Cohost Shutdown #shuppy.org #autost

next stop: tag synonyms and tag implications

untitled post by @delan

2024-09-18T00:58:43.604Z

jckarter (@jckarter) — [archived] 2024-09-17T19:56:12.434Z

what if rss or atom had a way to say that a post was in reply to another post from another feed. and feed readers let you display posts in threaded order

shuppy (@delan) — [archived] 2024-09-18T00:44:49.749Z

what if atom had a way to embed a copy of the post you're sharing or replying to, not just link to it? i would say just add a second <content>, but that’s forbidden :(

edit: hmm how about nesting <entry>, or putting an <entry> inside the <link> (instead of self-closing <link/>). per § 6.4.2, we may be able to nest <entry> as long as we have a foreign element in between, like <atom:entry><cohost:reply><atom:entry>

made a website so i can continue chosting

2024-09-16T16:11:32.249Z

>>> shuppy.org <<<

or add me to your feed reader!

>>> atom feed <<<

(need a feed reader? here are a bunch)
(it works well in FreshRSS and thunderbird)

oh and i’ve updated my contact details!! say hi and send me stuff :3

i had an old website, but for a variety of reasons it really didn’t feel like a good successor to my cohost page. i feel like what i really want is to Continue Chosting next month, just on my own website? (well no, what i really want is cohost, but failing that…)

so i made this! and many of my old chosts are there now too! i’ll need a bit more time to figure out how to actually make new posts (lol) and have them play nice with my old chosts, but hey, now you know where to find me once i do o/