shuppy (shuppy.org) —

embedding a javascript runtime in rust, part 1: duktape bindings

say you write a program in rust, with all of the benefits that entails (performance, type safety, etc), but you want to make parts of it programmable by the user without recompiling. you want to embed some kind of interpreted language, like lua or javascript, but surely that would be a pain? it’s not as bad as you might think!

which language? which runtime?

lua is famously embeddable, but i honestly don’t enjoy writing lua. that might be because i haven’t done it much. javascript is also embeddable, and after twenty years of writing javascript, i find it… tolerable. so let’s go with javascript.

there are many, many javascript runtimes out there. V8 and spidermonkey are cutting-edge options, very fast, but also very complex.

spidermonkey is what firefox and servo use, and it even has rust bindings, so you might think this is what i would pick. after all, servo is my job, and i’ve used html5ever and cssparser in hobby projects, and they’re great!

but i’ve never embedded an interpreter before, and the five months i had working on something adjacent to spidermonkey in servo tells me that a newbie like me should cut my teeth on something simpler.

enter: duktape

((o) duktape is a javascript runtime that’s about as easy to embed as it gets: one .c file, one .h file, and a straightforward C API. but there are some big drawbacks:

xkcd standards but it’s crates.io

i searched for “duktape” on crates.io and found 25 packages, most of which appeared to be unmaintained or immature. one even admits:

Aren't there about a dozen of these crates?
Yes. They all seem to address slightly different needs so I've added to the pile. Sorry.

so we’re shit outta luck right? not necessarily! we can roll our own duktape bindings without an unreasonable amount of effort. let’s do that here, in this post!

the code below is compatible with duktape 2.7.0.

1. low-level bindings

let’s create a rust program first, then grab a copy of duktape. while the duktape releases include the actual sources (not all in a single .c and .h) and build scripts we can use to create a customised flavour of duktape, we can just use the default flavour in src this time:

$ cargo new duktest
$ cd duktest
$ curl -O https://duktape.org/duktape-2.7.0.tar.xz
$ tar xf duktape-2.7.0.tar.xz
$ ls duktape-2.7.0/src
duk_config.h  duk_source_meta.json  duktape.c  duktape.h

we’re gonna use bindgen to generate most of our low-level bindings.

Cargo.toml
 
[build-dependencies]
bindgen = "0.72.1"

bindgen will use libclang to parse duktape.h. with nix, that looks like this:

shell.nix
 
with import <nixpkgs> {};
(mkShell.override { stdenv = clangStdenv; }) {
  name = "duktest-shell";
  LIBCLANG_PATH = lib.makeLibraryPath [ llvmPackages.clang-unwrapped.lib ];
}

the build script is almost entirely based on this example in the bindgen docs, but one important thing is to tell bindgen to only generate bindings for duktape.h, not a bunch of other standard library headers that will just cause us problems:

build.rs
 
// `error[E0428]: the name `FP_NAN` is defined multiple times`
// `error[E0428]: the name `FP_INFINITE` is defined multiple times`
// `error[E0428]: the name `FP_ZERO` is defined multiple times`
// `error[E0428]: the name `FP_SUBNORMAL` is defined multiple times`
// `error[E0428]: the name `FP_NORMAL` is defined multiple times`
// `warning: `extern` block uses type `u128`, which is not FFI-safe`
.allowlist_file(".*/duktape[.]h")
full build.rs
 
use std::env;
use std::path::PathBuf;

fn main() {
    let libdir_path = PathBuf::from("duktape-2.7.0/src")
        .canonicalize()
        .expect("cannot canonicalize path");

    let h_path = libdir_path.join("duktape.h");
    let c_path = libdir_path.join("duktape.c");
    let obj_path = libdir_path.join("duktape.o");
    let lib_path = libdir_path.join("libduktape.a");
    let (h_path, c_path, obj_path, lib_path) = (
        h_path.to_str().expect("unsupported path"),
        c_path.to_str().expect("unsupported path"),
        obj_path.to_str().expect("unsupported path"),
        lib_path.to_str().expect("unsupported path"),
    );

    // Tell cargo to look for shared libraries in the specified directory
    println!("cargo:rustc-link-search={}", libdir_path.to_str().unwrap());

    // Tell cargo to tell rustc to link our `duktape` library. Cargo will
    // automatically know it must look for a `libduktape.a` file.
    println!("cargo:rustc-link-lib=duktape");

    // Run `clang` to compile the `duktape.c` file into a `duktape.o` object file.
    if !std::process::Command::new("clang")
        .args(["-c", "-o", obj_path, c_path])
        .output()
        .expect("failed to spawn `clang`")
        .status
        .success()
    {
        panic!("failed to compile object file");
    }

    // Run `ar` to generate the `libhello.a` file from the `hello.o` file.
    if !std::process::Command::new("ar")
        .args(["rcs", lib_path, obj_path])
        .output()
        .expect("failed to spawn `ar`")
        .status
        .success()
    {
        panic!("failed to emit library file");
    }

    let bindings = bindgen::Builder::default()
        // `error[E0428]: the name `FP_NAN` is defined multiple times`
        // `error[E0428]: the name `FP_INFINITE` is defined multiple times`
        // `error[E0428]: the name `FP_ZERO` is defined multiple times`
        // `error[E0428]: the name `FP_SUBNORMAL` is defined multiple times`
        // `error[E0428]: the name `FP_NORMAL` is defined multiple times`
        // `warning: `extern` block uses type `u128`, which is not FFI-safe`
        .allowlist_file(".*/duktape[.]h")
        .header(h_path)
        // Tell cargo to invalidate the built crate whenever any of the
        // included header files changed.
        .parse_callbacks(Box::new(bindgen::CargoCallbacks::new()))
        .generate()
        .expect("failed to generate bindings");

    // Write the bindings to the $OUT_DIR/bindings.rs file.
    let out_path = PathBuf::from(env::var("OUT_DIR").unwrap());
    bindings
        .write_to_file(out_path.join("bindings.rs"))
        .expect("failed to write bindings!");
}

now we pull in those bindings from $OUT_DIR/bindings.rs into a module:

src/sys.rs
 
#![allow(unused)]
#![allow(nonstandard_style)]
include!(concat!(env!("OUT_DIR"), "/bindings.rs"));
src/main.rs
 
mod sys;

2. missing pieces

bindgen is great, but there are some things it just can’t do. for these things, we’ll have to write the bindings for by hand, translating from C to rust.

a big one is function-like macros. duktape uses a modest amount of these, thankfully not too many as of version 2.7.0, but they also reserve the right to change any API to a macro (or back to a real function) in any release.

so far, i’ve had to translate duk_eval, duk_eval_noresult, duk_peval, duk_peval_noresult, duk_eval_string, duk_peval_string, duk_eval_lstring, duk_peval_lstring, and duk_safe_to_string. here are a few of those:

duktape-2.7.0/src/duktape.h
 
#define duk_eval_string(ctx,src)  \
    ((void) duk_eval_raw((ctx), (src), 0, 0 /*args*/ | DUK_COMPILE_EVAL | DUK_COMPILE_NOSOURCE | DUK_COMPILE_STRLEN | DUK_COMPILE_NOFILENAME))

#define duk_peval_lstring(ctx,buf,len)  \
    (duk_eval_raw((ctx), buf, len, 0 /*args*/ | DUK_COMPILE_EVAL | DUK_COMPILE_NOSOURCE | DUK_COMPILE_SAFE | DUK_COMPILE_NOFILENAME))

#define duk_safe_to_string(ctx,idx) \
    duk_safe_to_lstring((ctx), (idx), NULL)
src/sys.rs
 
use std::{ffi::c_char, ptr::null_mut};

pub unsafe fn duk_eval_string(ctx: *mut duk_context, src: *const c_char) {
    unsafe {
        duk_eval_raw(
            ctx,
            src,
            0,
            0 /*args*/ | DUK_COMPILE_EVAL | DUK_COMPILE_NOSOURCE | DUK_COMPILE_STRLEN | DUK_COMPILE_NOFILENAME,
        );
    }
}

pub unsafe fn duk_peval_lstring(ctx: *mut duk_context, buf: *const c_char, len: duk_size_t) -> duk_int_t {
    unsafe {
        duk_eval_raw(ctx, buf, len, 0 /*args*/ | DUK_COMPILE_EVAL | DUK_COMPILE_NOSOURCE | DUK_COMPILE_SAFE | DUK_COMPILE_NOFILENAME)
    }
}

pub unsafe fn duk_safe_to_string(ctx: *mut duk_context, idx: duk_idx_t) -> *const c_char {
    unsafe {
        duk_safe_to_lstring(ctx, idx, null_mut())
    }
}

bindgen does support simple preprocessor constants, but for reasons unclear to me, not all of them make it through. the only one i’ve run into so far is DUK_VARARGS:

duktape-2.7.0/src/duktape.h
 
/* Indicates that a native function does not have a fixed number of args,
 * and the argument stack should not be capped/extended at all.
 */
#define DUK_VARARGS                       ((duk_int_t) (-1))
src/sys.rs
 
pub const DUK_VARARGS: duk_int_t = -1;

3. hello world

with this, we have enough to start running some javascript!

src/main.rs
 
mod sys;

use std::{ffi::CStr, io::{Write, stdout}, ptr::null_mut};

use crate::sys::*;

fn main() {
    unsafe {
        let ctx = duk_create_heap(None, None, None, null_mut(), None);
        duk_push_c_function(ctx, Some(nop), DUK_VARARGS);
        duk_put_global_string(ctx, c"nop".as_ptr());
        duk_push_c_function(ctx, Some(print), DUK_VARARGS);
        duk_put_global_string(ctx, c"print".as_ptr());

        duk_eval_string(ctx, c"nop()".as_ptr());
        duk_eval_string(ctx, cr#"print("hello js world\n")"#.as_ptr());

        duk_pop(ctx);
        duk_destroy_heap(ctx);

        println!("goodbye rust world");
    }
}

extern "C" fn nop(_: *mut duk_context) -> duk_ret_t {
    0
}

extern "C" fn print(ctx: *mut duk_context) -> duk_ret_t {
    unsafe {
        duk_push_string(ctx, c" ".as_ptr());
        duk_insert(ctx, 0);
        duk_join(ctx, duk_get_top(ctx) - 1);
        let text = duk_safe_to_string(ctx, -1);
        let text = CStr::from_ptr(text);
        let _ = stdout().write_all(text.to_bytes());
    }
    0
}

$ cargo run
hello js world
goodbye rust world

stay tuned!

next time, we’ll look at building some abstractions over these bindings, plus some tools to help us manage the interpreter’s value stack correctly.