elixir_unikernel

elixir_unikernel turns a mix release into a bootable machine image that runs on a memory-safe kernel written in Rust. There is no Linux userland in the image: no shell, no init system, no dynamic loader. The kernel starts a 300 line static C program, /init, which configures the network interface and execs beam.smp. From then on the Erlang VM is the only process.

Booting from ROM..
[kernel] running /init as the init process
[init] elixir_unikernel init, uptime 0.450 s
[init] net: eth0 10.0.2.15/24 gw 10.0.2.2
[init] exec /rel/erts-17.1/bin/beam.smp (iex mode)
Erlang/OTP 29 [erts-17.1] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [jit:ns]
13:01:08.809 [info] uniapp starting (otp 29, elixir 1.20.4, uptime 3.80s)
Interactive Elixir (1.20.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)>

What you get

• A 9.4 MB initramfs.cpio.gz containing Erlang/OTP 29.1.1 with a fully static beam.smp, Elixir 1.20.4, and your release. OpenSSL 3.5 is linked into the VM, so :crypto, :ssl and :public_key work out of the box.
• A 5.8 MB Asterinas kernel: Linux-ABI compatible, written in Rust, with virtio-net and a TCP/IP stack.
• A make based build that pins every upstream by git tag and a smoke test that boots the image and checks TCP, UDP, TLS 1.3 (client and server) and DNS over virtio-net.
• The same image also boots on a stock Linux kernel, which is useful for telling kernel problems from application problems.

What it is not

It is not a unikernel in the strict sense. The kernel and the BEAM are two binaries with a syscall boundary between them. What is removed is everything else: the image is a kernel plus one process. If you want a single linked binary, see Why these kernels for the Hermit track.

It also is not a general purpose OS. There is no persistent storage, no distribution/epmd, and no way to spawn external programs because there are none. See Limits.

Where to go next

• Prerequisites and Build and first boot get you to an IEx prompt in about 15 minutes on a cold cache.
• Shipping your own application replaces the sample app.
• Architecture explains how the pieces fit.

Prerequisites

Everything is compiled inside Docker containers; the host only needs a few tools. make check verifies all of them and prints install hints.

macOS (Apple Silicon)

Docker Desktop is not required. Colima works well:

brew install colima docker docker-buildx qemu coreutils
mkdir -p ~/.docker/cli-plugins
ln -sf "$(brew --prefix)/opt/docker-buildx/bin/docker-buildx" ~/.docker/cli-plugins/
colima start --cpu 8 --memory 12 --disk 80 --vm-type vz --vz-rosetta
make check

Give the VM at least 8 CPUs and 12 GB: the Asterinas kernel build and the OTP build both like parallelism. The containers run natively on arm64; only the final ERTS binaries are cross-compiled for x86-64, so there is no emulation on the build path (see Build pipeline).

QEMU on macOS has no KVM. The image still boots, through TCG (software emulation of x86-64). Expect 3 to 5 s to the IEx prompt and about 12 minutes for the full make smoke.

Linux (x86-64)

sudo apt install docker.io docker-buildx-plugin qemu-system-x86 make python3 openssl
sudo usermod -aG docker,kvm "$USER"   # re-login
make check

With a writable /dev/kvm, make selects KVM automatically and boots are fast.

Disk and network

The first build downloads the OTP, Elixir and Asterinas sources and the Asterinas development container (about 4 GB). Budget 15 GB of Docker disk. Build artefacts live under build/ and are not committed.

Build and first boot

git clone https://github.com/bmalum/elixir_unikernel.git
cd elixir_unikernel
make check          # host prerequisites
make initramfs      # ~6 min cold, ~1 min when OTP/Elixir layers are cached
make run-m1         # build the Asterinas kernel (~4 min cold) and boot into IEx

make initramfs builds four Docker stages: an x86-64 Alpine sysroot, a cross-compiled OTP tree, a native OTP (so Elixir and Mix run at full speed), and the release. The output is build/initramfs.cpio.gz.

make run-m1 compiles Asterinas inside its upstream development container, applies the two patches from builder/asterinas-patches/, and starts QEMU:

Booting from ROM..OSTD initialized. Preparing components.
...
[kernel] unpacking initramfs.cpio.gz to rootfs ...
[kernel] running /init as the init process
[init] elixir_unikernel init, uptime 0.450 s
[init] net: eth0 10.0.2.15/24 gw 10.0.2.2
[init] exec /rel/erts-17.1/bin/beam.smp (iex mode)
Erlang/OTP 29 [erts-17.1] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [jit:ns]
Interactive Elixir (1.20.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)>

Type an expression:

iex(1)> :inet.getifaddrs()
iex(2)> :ssl.connect(~c"www.erlang.org", 443, [verify: :verify_peer, cacertfile: ~c"/etc/ssl/cacert.pem"])

Leave QEMU with Ctrl-a followed by x.

The [init] lines about Not a tty on Asterinas are expected: that kernel does not implement the interface-configuration ioctls and configures 10.0.2.15/24 itself. /init logs and continues. See Networking.

Application mode

make run-m1-app

boots the same image with uniapp.mode=app: no shell, the release’s application starts under its supervision tree, and the sample app prints a few self-test lines:

LISTEN udp 4001
LISTEN tcp 4000
LISTEN tls 4443
PROBE dns ok
PROBE tls ok
PROBE done

While it runs, talk to it from another terminal:

printf 'hello\n' | nc 127.0.0.1 14000                                   # TCP echo
printf 'ping' | nc -u -w1 127.0.0.1 14001                               # UDP echo
(echo tls-hello; sleep 1) | openssl s_client -connect 127.0.0.1:14443 -tls1_3 -quiet

QEMU forwards host ports 14000, 14001 and 14443 to the guest’s 4000, 4001 and 4443.

The same image on Linux

make run-m0

boots build/initramfs.cpio.gz on a stock Linux 6.1 kernel (the Firecracker CI build, which has virtio and the serial console compiled in). Nothing in the image is Linux specific; this target exists so that a failure can be blamed on the right side of the syscall boundary.

Running the tests

make smoke-m0     # Linux, ~3 min under TCG
make smoke-m1     # Asterinas, ~3 min
make smoke        # release + both kernels + both smoke tests + sizes, ~12 min cold

scripts/smoke.sh boots the image twice per kernel and checks the GOALS.md criteria:

Application mode

• /init exec’d beam.smp, the application started, no shell appeared
• the TCP, UDP and TLS echo servers report LISTEN
• from the host, through the QEMU port forwards: a TCP echo round trip, a UDP echo round trip, and a TLS 1.3 session with openssl s_client that echoes a line
• inside the guest: PROBE dns ok (a lookup through OTP’s pure-Erlang resolver inet_res) and PROBE tls ok (:ssl.connect with verify_peer against the bundled CA store to TLS_HOST, default www.erlang.org)
• no crash dump, kernel panic or erl_child_setup error in the console log

IEx mode

• the Interactive Elixir (1.20...) banner and an iex(1)> prompt
• two expressions are typed on the serial console and their output is checked

Console logs are kept in build/logs/<label>-<mode>.log (raw) and .log.clean (ANSI stripped). The host-side client results are in <label>-clients.log.

Options

The retry logic exists because QEMU TCG on a loaded laptop occasionally starts slowly; a retry is reported with an info: line and does not fail the test. Before the timerfd fix in Asterinas patches it also masked a real kernel bug, so if you see retries on every run, look at the console log.

Continuous integration

.github/workflows/smoke.yml runs the same three steps on GitHub Actions: build the release (cached with type=gha), then M0 and M1 in parallel jobs under TCG. Logs are uploaded as artefacts.

Booting and the console

The image boots in one of three modes, chosen on the kernel command line.

The boot sequence is the same in every mode:

1. The kernel unpacks the initramfs into its root file system and runs /init as PID 1.
2. /init mounts proc, devtmpfs and sysfs where the kernel supports them, reads /proc/cmdline, configures lo and the first ethernet interface, writes /etc/inetrc, /etc/resolv.conf and /etc/hosts, exports the environment ERTS expects (BINDIR, ROOTDIR, RELEASE_*, LANG=C.UTF-8, TERM=dumb), and execs beam.smp.
3. ERTS loads the release boot script (releases/<vsn>/start.boot) and sys.config, starts the applications, and in iex mode starts user_drv with Elixir’s shell.

There is no process other than the BEAM. ERTS does spawn its erl_child_setup helper at start (it is needed for Ports); it is the only other binary in the image and it idles.

The serial console

All output goes to the first serial port (console=ttyS0), which QEMU connects to your terminal with -nographic. Input works the same way, so IEx is fully usable: history, tab completion and line editing work because prim_tty sees a tty.

TERM is set to dumb because the image contains no terminfo database. Colours and the fancy prompt are therefore off; set uniapp.term=xterm is not supported yet.

Ctrl-C does not open the BREAK menu (ERTS runs with +Bd). Ctrl-a x exits QEMU. Ctrl-a c switches to the QEMU monitor, where info registers or system_powerdown are available.

Logs

The release’s Logger writes to standard output, which is the console. In app mode that is the only output channel. If you need logs elsewhere, ship them over the network from your application; there is no syslog and no disk.

The log level can be set at boot with the UNIAPP_LOG_LEVEL environment variable, which the sample config/runtime.exs reads; /init does not set it, so add a command-line switch if you need one (see The init program).

Exit and reboot

System.stop/0 or :init.stop/0 halts the VM. Because beam.smp is PID 1, the kernel then has no init process: Linux panics with “Attempted to kill init”, Asterinas prints a message and halts. Either way QEMU stays running until you quit it (the Makefile passes -no-reboot). A clean power-off from inside the guest is not implemented yet.

Crash dumps are disabled (ERL_CRASH_DUMP=/dev/null): there is nowhere to write them and a crash on a read-only initramfs would otherwise stall for the dump. The crash reason is still printed to the console.

Networking

The guest has one Ethernet interface. By default it is configured with DHCP; static addresses come from the kernel command line.

Under QEMU user-mode networking (-netdev user) DHCP yields 10.0.2.15, gateway 10.0.2.2 and the DNS forwarder 10.0.2.3; the make run-* targets rely on that. If DHCP fails, /init falls back to exactly those values.

How the interface is configured

On Linux /init runs a small DHCPv4 client over an AF_PACKET socket (DISCOVER, OFFER, REQUEST, ACK; five attempts with backoff), then applies the lease with the classic ioctls SIOCSIFADDR, SIOCSIFNETMASK, SIOCSIFFLAGS and SIOCADDRT. The interface name is discovered from /sys/class/net, falling back to eth0.

Asterinas has no AF_PACKET sockets, so there the kernel does DHCP: ip=dhcp on the command line (the Linux ip= parameter) starts smoltcp’s DHCPv4 client on eth0 and publishes the lease in /proc/net/dhcp as eth0 10.0.2.15/24 10.0.2.2 dns 10.0.2.3 (or eth0 pending). /init waits up to 30 s for that line and takes the nameserver from it; the address and route are already installed. Without ip=dhcp Asterinas keeps its compiled-in 10.0.2.15/24 via 10.0.2.2, which /init may override with uniapp.ip=: the set-ioctls work since patch 0003. The log line tells you which path ran: (kernel dhcp), (dhcp), (static) or (default).

Name resolution

OTP’s default lookup method on Unix is native, which spawns the port program inet_gethost. That binary is not in the image, so /init writes /etc/inetrc:

{lookup, [file, dns]}.
{host, {127,0,0,1}, ["localhost"]}.
{edns, 0}.

and points ERL_INETRC at it. Lookups then go through inet_res, OTP’s resolver written in Erlang, which reads nameservers from /etc/resolv.conf. /init writes that file from uniapp.dns. Keep both files: inet_db re-reads resolv.conf periodically and a missing file clears the nameserver list.

:inet.gethostbyname/1, :gen_tcp.connect(~c"host", ...), :httpc, Req, Finch and friends all work through this path.

TLS

OpenSSL 3.5 is statically linked into beam.smp, so :crypto, :ssl and :public_key are available without any files. For verifying servers, the image ships Alpine’s CA bundle at /etc/ssl/cacert.pem:

:ssl.connect(~c"www.erlang.org", 443,
  verify: :verify_peer,
  cacertfile: ~c"/etc/ssl/cacert.pem",
  versions: [:"tlsv1.3"],
  server_name_indication: ~c"www.erlang.org")

For a TLS server, generate a certificate at boot (the sample app uses :public_key.pkix_test_data/1 with digest: :sha256; SHA-1 signed test certificates are rejected by modern clients) or ship a certificate and key in priv/ of your release.

Reaching the guest from the host

With QEMU user networking the guest is not routable from the host; use port forwards. The Makefile forwards host 14000, 14001 and 14443 to guest 4000 (TCP), 4001 (UDP) and 4443 (TCP). Add more in QEMU_BASE, or switch to a tap device for a real bridged network.

Known gaps on Asterinas

• Binding to 0.0.0.0 attaches the socket to the NIC only, so traffic to 127.0.0.1 does not reach a socket bound to the wildcard address. Bind explicitly to 127.0.0.1 for loopback-only servers. This comes from the local patch that makes wildcard binds work at all; see Asterinas patches.
• :inet.getifaddrs/0 works on Asterinas main but returned {:error, :eaddrnotavail} on v0.18.1.
• IPv6 is untested.

Shipping your own application

The image embeds whatever mix release produces from app/. Replace the sample with your project and rebuild.

What to keep from the sample

Three small pieces in app/ are worth copying into your project:

mix.exs release options:

releases: [
  myapp: [
    include_executables_for: [],      # no bin/myapp shell script, /init starts beam.smp
    include_erts: erts(),             # cross-compiled ERTS from the builder, see below
    strip_beams: true,
    steps: [:assemble, &Myapp.ReleaseSteps.prune/1]
  ]
]

defp erts do
  case System.get_env("UNIAPP_ERTS") do
    nil -> true
    root -> root |> Path.join("erts-*") |> Path.wildcard() |> List.first()
  end
end

UNIAPP_ERTS is set by the Dockerfile to /opt/otp-x86, the cross-compiled OTP tree. Mix then takes both ERTS and the OTP applications from there, which is what makes the release x86-64 even though Mix itself runs natively.

lib/.../release_steps.ex (prune/1) deletes everything the shell-free boot does not need: the release’s bin/, all ERTS binaries except beam.smp and erl_child_setup, sources, includes and docs. scripts/assemble-rootfs.sh refuses to build an image that contains any other ELF file, so keep this step.

lib/.../cmdline.ex reads key=value pairs from the kernel command line (/init copies it into KERNEL_CMDLINE). Use it for runtime configuration instead of environment variables.

Dependencies

Pure Elixir and Erlang dependencies work unchanged. Add the Hex steps to the release stage of builder/Dockerfile:

COPY app/mix.exs app/mix.lock ./
RUN mix local.hex --force && mix local.rebar --force && mix deps.get --only prod
COPY app/config config
COPY app/lib lib
COPY app/priv priv
RUN mix release --overwrite --path /out/release

Dependencies with NIFs do not work: a NIF is a shared object loaded with dlopen, and the image has no dynamic loader. Prefer the pure-Elixir alternative (pbkdf2_elixir instead of bcrypt_elixir, postgrex is fine, exqlite is not) or link the NIF statically into beam.smp the way OTP’s own crypto and asn1 are. The latter means building the NIF as a .a archive and adding it to --enable-static-nifs in the cross OTP stage; it is real work but has been done for OTP’s NIFs already.

Configuration

config/runtime.exs is evaluated at boot inside the image. Keep reboot_system_after_config at its default (false): the release directory is read-only and a reboot would need to write a new sys.config.

Environment variables are what /init sets (RELEASE_*, LANG, TERM, KERNEL_CMDLINE). Anything else must come from the command line via Cmdline.get/2 or be baked into config/config.exs.

Starting without a shell

In app mode the release boots with -noshell and the applications listed in mix.exs start under application_controller. If an application exits, init stops the node, which ends the VM (there is no supervisor above beam.smp). The sample’s echo server therefore retries a failed listen instead of crashing, so that a kernel limitation never takes the image down. Do the same for anything that touches the OS at start.

Build and boot

make initramfs      # ~1 min once OTP and Elixir are cached
make run-m1-app

The release name and version are read by /init from RELEASE_NAME and RELEASE_VSN compile-time defines; if you rename the app from uniapp, update the two #defines at the top of init/init.c (or pass them as -DRELEASE_NAME=... in the Dockerfile).

A Phoenix application

Phoenix is ordinary OTP code, and the image already has everything a Phoenix release needs: static ERTS, :crypto/:ssl, a working inet over virtio-net and a release boot script. Four things differ from a normal deployment.

1. Generate and vendor

mix phx.new app --no-ecto --no-mailer --install

Drop the result into app/, then re-add the three pieces from Shipping your own application: the erts() helper and release options in mix.exs, ReleaseSteps.prune/1, and Cmdline.

Check mix.lock for NIF-based dependencies. The Phoenix defaults (bandit/cowboy, plug, jason, phoenix_pubsub, telemetry, websock_adapter) are pure BEAM. bcrypt_elixir and argon2_elixir are not; use pbkdf2_elixir. SQLite via exqlite is not; Postgres via postgrex is.

2. Assets

There is no Node or esbuild in the image, so assets are built in the Docker release stage. The Elixir esbuild and tailwind packages download an x86-64 binary and run it; that only works in an x86-64 container. Two options:

• Build assets on the host (or in CI) and commit or COPY priv/static into the builder, removing assets.deploy from the release step.
• Add a small x86-64 stage that runs mix assets.deploy (slow on arm64 hosts, it runs under emulation).

The digested files in priv/static are served by Plug.Static straight from the initramfs.

3. Configuration

Replace the environment-variable plumbing in config/runtime.exs:

import Config

if config_env() == :prod do
  port = String.to_integer(Myapp.Cmdline.get("phx.port", "4000"))
  host = Myapp.Cmdline.get("phx.host", "localhost")

  config :myapp, MyappWeb.Endpoint,
    server: true,                       # no bin/server script sets PHX_SERVER
    url: [host: host, port: port],
    http: [ip: {0, 0, 0, 0}, port: port],
    check_origin: false,                # or ["//" <> host]
    secret_key_base: Myapp.Cmdline.get("phx.secret") || raise("phx.secret= missing")
end

A secret on the kernel command line is visible to anyone who can read the VM’s configuration, which is the same threat model as an environment variable in a container. For HTTPS, add https: [port: 4443, certfile: ..., keyfile: ...] with the files in priv/, or generate a self-signed pair at boot as the sample’s TLS echo server does.

4. Forward the port

Add hostfwd=tcp::14080-:4000 to QEMU_BASE in the Makefile, then:

make initramfs run-m1-app
open http://127.0.0.1:14080

What to expect

• Image size grows by roughly 4 to 6 MB (gzipped) for Phoenix and its dependencies, well inside the budget.
• Memory: plan on 256 MB guest RAM; a LiveView app with a few connections sits around 90 to 130 MB RSS.
• Ecto works as a client to a database outside the VM. There is no database inside the image.
• mix phx.server, code reloading, mix tasks and System.cmd/3 do not exist at runtime.
• Expect to find one or two socket options that Asterinas does not implement yet (ENOPROTOOPT in the log). Bandit and Cowboy tolerate most of them; report the ones that break and test on make run-m0-app to confirm it is kernel-side.

Releasing a bundle

make dist

produces dist/elixir_unikernel-<version>-x86_64/ and a .tar.gz of it:

The version comes from the VERSION file at the repository root. Bump it, update CHANGELOG.md, tag the commit, and attach the tarball to the release.

Running the bundle elsewhere

The bundle needs only qemu-system-x86_64. On a Linux host with KVM:

tar xzf elixir_unikernel-0.1.0-x86_64.tar.gz
cd elixir_unikernel-0.1.0-x86_64
sha256sum -c SHA256SUMS
./run.sh app

run.sh uses QEMU user networking with the same port forwards as the Makefile. For anything beyond a demo, replace the -netdev user line with a tap or bridge device and pass the real addresses as uniapp.ip=, uniapp.gw= and uniapp.dns= arguments.

Other VMMs

The kernel is a multiboot ELF, so it needs a loader that speaks multiboot: QEMU does, Firecracker and cloud-hypervisor do not. Asterinas can also produce a Linux bzImage (--grub-boot-protocol linux in scripts/build-asterinas.sh), which Firecracker accepts; that variant did not produce console output under QEMU TCG in our tests and is not part of the smoke test yet. The initramfs itself is a plain gzip newc cpio and works with any of them.

Reproducibility

make smoke from an empty build/ rebuilds everything from the pinned tags and runs both smoke tests. The Docker stages are deterministic in content (same sources, same flags) but not byte-identical: Rust and C builds embed paths and timestamps, so two asterinas.elf files from the same commit differ. VERSIONS is the record of what went in.

Running on EC2

The image boots as an EC2 AMI on Nitro x86-64 instances: the same GPT disk that make smoke-disk boots under OVMF, uploaded as an EBS snapshot and registered as a UEFI, ENA-enabled machine image. Both kernels work. With Asterinas the guest contains no Linux at all: GRUB, the Asterinas kernel with the ENA driver from patch 0005, /init, beam.smp and the release.

[kernel] ena: found 1d0f:ec20 rev 0 at PciDeviceLocation { bus: 0, device: 5, function: 0 }
[kernel] ena: 0a-ff-cf-04-ad-1f ready, 127 Rx buffers of 4096 bytes, 128 Tx slots, MSI-X vector 1
[kernel] running /init as the init process
[init] elixir_unikernel init, uptime 1.641 s
[init] net: eth0 172.31.12.217/20 gw 172.31.0.1 dns 172.31.0.2 (kernel dhcp)
[init] exec /rel/erts-17.1/bin/beam.smp (app mode)
10:33:45.369 [info] uniapp starting (otp 29, elixir 1.20.4, uptime 4.36s)

That is the EC2 Serial Console of a t3.small. Nothing in the image is EC2-specific except the ENA driver; the address comes from the VPC’s DHCP.

Prerequisites

• An AWS account and credentials in the environment (AWS_PROFILE, AWS_REGION). The scripts never read a region from the repository.
• aws CLI v2, python3 with boto3, openssl, curl.
• IAM permissions for EC2 (ebs:StartSnapshot, ebs:PutSnapshotBlock, ebs:CompleteSnapshot, ec2:RegisterImage, ec2:RunInstances, security groups, ec2:GetConsoleOutput) and, for the serial console, ec2-instance-connect:SendSerialConsoleSSHPublicKey.
• For the serial console, once per account: aws ec2 enable-serial-console-access.

Everything the scripts create is tagged Project=elixir_unikernel and Version=<VERSION>; the smoke tests terminate their instance and delete their security group in an EXIT trap, including on Ctrl-C.

Build and publish

make ami                       # build/disk-asterinas.raw: 1 GiB GPT, 94 MB EFI partition
make ami-publish               # -> ami-0a9c33776357b6a40 (prints the id)
make ami KERNEL=linux          # the same disk around Linux 6.1 with ENA + NVMe
make ami-publish KERNEL=linux

scripts/ami-publish.py uses the EBS direct APIs: it starts a snapshot of a 1 GiB volume, uploads only the non-zero 512 KiB blocks (about 45 of 2048), completes it and registers the image with BootMode=uefi, EnaSupport=true, SriovNetSupport=simple, root device /dev/xvda. No S3 bucket and no vmimport role are needed, and the whole step takes about 12 seconds. The AMI name is elixir_unikernel-<VERSION>-<kernel> (plus -iex for DISK_MODE=iex); an existing image of that name is reused, AMI_FLAGS=--force replaces it.

The command line is baked into the image (DISK_CMDLINE_asterinas in the Makefile): console=ttyS0 earlycon loglevel=error ip=dhcp uniapp.tls_host=... uniapp.mode=app. Keep loglevel=error: info logs every syscall and overruns the 64 KB console buffer within seconds. DISK_MODE=iex builds an image that drops into IEx on the serial console.

Launch and verify

make smoke-ec2                 # publish (or reuse), launch t3.small, assert, terminate
scripts/smoke-ec2.sh <ami-id> [label]

The script creates a security group that allows 4000/tcp and 4443/tcp from the caller’s /24 (CLIENT_CIDR overrides it; NAT pools rotate the source address between connections), launches one t3.small (INSTANCE_TYPE) in the default VPC, polls GetConsoleOutput for uniapp starting (limit BOOT_TIMEOUT, default 120 s), echoes through TCP 4000 and TLS 1.3 4443 from the internet and checks the PROBE dns ok and PROBE tls ok lines. With a data snapshot (make smoke-ec2 builds one from scripts/mkdata.sh and passes it as DATA_SNAPSHOT) the launch also gets a 1 GiB EBS data volume on /dev/sdf, the IAM role elixir_unikernel-smoke (created on first use; CloudWatch Logs permissions only) and user data that turns on uniapp.cloudwatch=1, uniapp.data=auto and a VM exit 75 s after boot. The script then checks ten more things: IMDS identity, user-data overrides, NTP sync, /data mounted, boot_count 1, the CloudWatch shipper, the guest-initiated reboot, boot_count 2 on the second boot, log events in /elixir_unikernel/smoke/<instance-id>, the BootCount EMF document, and then times aws ec2 reboot-instances (next boot visible within 180 s) and aws ec2 stop-instances (stopped within 150 s). Both kernels pass all 23 (Asterinas ami-0b7ba8bda566dc68c, Linux ami-058539254c3af3853).

KEEP=1 leaves the instance running. The console is saved to build/logs/ec2-<label>.log.

A run on the Asterinas AMI:

instance i-01ac1400270175f66 (t3.small, ami-0a9c33776357b6a40)
public ip 3.73.65.60, running after 17 s
  PASS uniapp starting on the console after 45s (limit 120s)
  PASS init configured the NIC: net: eth0 172.31.12.217/20 gw 172.31.0.1 dns 172.31.0.2 (kernel dhcp)
  PASS address came from DHCP
  PASS internet -> instance TCP 4000 echo
  PASS internet -> instance TLS 1.3 4443 echo
  PASS PROBE dns ok
  PASS PROBE tls ok
  PASS no crash
SMOKE asterinas (ami-0a9c33776357b6a40 on t3.small): 8 passed, 0 failed

“45 s” is measured from the instance entering running until the line is visible through GetConsoleOutput, which lags by tens of seconds. The guest itself reports BEAM up 4.4 s after the kernel started and the DHCP lease at 1.6 s; the firmware and GRUB account for most of the rest.

The serial console

scripts/ec2-console.sh <instance-id>            # GetConsoleOutput, read-only
scripts/ec2-console.sh <instance-id> --attach   # interactive, via EC2 Instance Connect
scripts/smoke-ec2-iex.sh <ami-id>               # launches a DISK_MODE=iex image and types 1 + 2

--attach pushes a one-off SSH key with SendSerialConsoleSSHPublicKey and connects to serial-console.ec2-instance-connect.<region>.aws. There is no login prompt: the serial line is the Erlang VM’s stdio, so an uniapp.mode=iex image shows iex(1)> and an app image shows the log. One session per instance; ~. detaches, and EC2 needs about 30 s before it accepts the next one.

Cleaning up

make ami-clean                 # deregister this version's AMIs and delete their snapshots
aws ec2 describe-instances --filters Name=tag:Project,Values=elixir_unikernel \
  --query 'Reservations[].Instances[].[InstanceId,State.Name]' --output text

A t3.small costs about 18 USD per month if left running; the snapshot of a 1 GiB volume is cents.

Health checks and throughput

The sample app serves GET /healthz on port 8080 (uniapp.health_port=): 503 {"status":"starting"} until the echo servers listen, then 200 with uptime, memory, process count, instance id, boot count and the last measured echo throughput; GET /livez is 200 as soon as the socket is open. For an ALB target group use path /healthz, port 8080, matcher 200; for an auto-scaling group set the health check type to ELB. The implementation (app/lib/uniapp/health.ex, 160 lines of :gen_tcp) is the template for your own application’s endpoint.

Throughput, t3.small, Asterinas with the ENA driver after patch 0009:

scripts/bench-ec2.sh <ami> [kernel args] launches two instances, has the second push 8 MB through the first’s TCP echo and prints the sender’s BENCH tcp_echo_peer line; both are terminated afterwards.

Operating it

• Configuration comes from EC2 user data, not from rebuilding the AMI: plain key=value lines (uniapp.mode=iex, uniapp.log_group=/prod/web, anything the command line accepts). /init fetches it over IMDSv2 and the raw text is at /run/user-data for the application.
• Crashes restart the instance: when the VM exits, /init reboots the machine and the AMI boots again (about 45 s). Set uniapp.on_exit=poweroff with --instance-initiated-shutdown-behavior terminate if the ASG should replace the instance instead.
• stop-instances, reboot-instances, terminate-instances press the ACPI power button. The kernel turns it into SIGPWR, /init stops the VM with SIGTERM (ERTS shuts the applications down in order), waits up to 20 s and powers off through ACPI S5. Stops complete in about 15 s, reboots in about 45 s, instead of EC2’s four-minute hard reset.
• State lives on EBS volumes: uniapp.data=auto|/dev/X|LABEL=name mounts one at /data, uniapp.mounts=LABEL=cache:/cache,/dev/nvme3n1:/logs mounts more (ext2, read-write; UNIAPP_DATA, UNIAPP_MOUNTS in the environment). scripts/mkdata.sh out.raw 64 mylabel makes an empty image, scripts/ami-publish.py --snapshot-only turns it into a snapshot to launch from. ext2 has no journal: /init runs sync(2) every uniapp.sync_s seconds (default 30) and on exit, so an unclean stop loses at most that window; call fsync for anything that must not. Verified: a file written without fsync survived a hard kill 30 s later under QEMU and a VM exit on EC2. Growing a volume means creating a larger one and copying (no online resize, no resize2fs in the image). erl_crash.dump lands on /data.
• Logs and metrics go to CloudWatch from inside the BEAM (uniapp.cloudwatch=1, instance role with logs:* on the group): every Logger event is batched into PutLogEvents, metrics are EMF documents in the same stream. Nothing else runs on the instance, so this is the only telemetry path; use it as the model for your own application.
• Time is set from Amazon Time Sync at boot and hourly.

What does not work yet

• ENA: no LLQ, no TSO; up to 8 queue pairs (one per vCPU by default), tested on 2-vCPU instances only. See the ENA driver notes.
• Asterinas’s NVMe driver needs patch 0007 to talk to EBS (the controller reports 32 queue entries and rejects larger queues); it has one I/O queue and no interrupts tuning, fine for a boot counter and crash dumps, not for a database.
• IPv6: the stack handles IPv6 sockets (loopback verified on both kernels, uniapp.probe_ipv6=1), but there is no SLAAC or DHCPv6 client, so an instance in a dual-stack subnet gets no global address; static configuration only. Hibernation (--hibernation-options) is not supported: the kernel has no S4 and the image no swap.
• Graviton (arm64), Xen-based instance types and Marketplace publishing are out of scope.

Troubleshooting

Start by deciding which side of the syscall boundary is at fault: boot the same image on Linux with make run-m0-app. If it works there and not on Asterinas, it is a kernel gap; if it fails on both, it is the image or the application.

Nothing after Booting from ROM..

Asterinas: the guest has less than 144 MB (M1_MEM). The kernel is loaded at physical 128 MB and needs room above that. Also check that earlycon is on the command line; without it early output is lost.

Linux: with less than 128 MB the kernel OOM-kills erl_child_setup and panics (“System is deadlocked on memory”); the console shows it.

[init] execv ... No such file or directory

The ERTS version compiled into /init (-DERTS_VSN=) does not match the release. The Dockerfile derives it from /opt/otp-x86/ERTS_VSN; if you changed OTP versions, rebuild from the release stage (make release).

Environment variable BINDIR is not set

/init was not used (for example the kernel ran beam.smp directly). ERTS needs BINDIR to find erl_child_setup.

Can not execute .../inet_gethost : enoent

The resolver fell back to native lookup. /etc/inetrc is missing or ERL_INETRC is not exported. Both are written by /init; check the console for a write /etc/inetrc error.

DNS resolves nothing (nxdomain for everything)

/etc/resolv.conf is missing. OTP’s inet_db re-reads it periodically and clears the nameserver list when the file does not exist, even if inetrc named a server. /init writes it from uniapp.dns=.

{:error, :eaddrnotavail} from listen or open

On Asterinas v0.18.1 and unpatched main, bind() to 0.0.0.0 fails. Make sure the kernel was built by scripts/build-asterinas.sh (it applies 0001-bind-unspecified-to-default-iface.patch); the build log prints applied 0001-....

The VM is alive but IEx never prints a prompt, or the app stalls

Before patch 0002-timerfd-settime-invalidate-readiness, Asterinas kept reporting an already-disarmed timerfd as readable, and ERTS’s poll thread spun at 100 % on epoll_wait. The symptom was a stall at random points, always with one vCPU. Check the build log for applied 0002-.... The C reproducer is described in Asterinas patches.

make smoke-* says host port 14443 is in use

A previous QEMU is still running, or something else listens on the forwarded port. pkill qemu-system-x86_64, or set HOST_PORT_TLS=24443 (and the TCP and UDP variants).

TCP False in the client log while the guest says LISTEN tcp

Something on the host already listens on HOST_PORT_TCP (a local iex -S mix, for example), so the client connected to it instead of the guest. The port guard catches TCP and TLS; UDP has no LISTEN state and is not checked.

Segmentation fault while compiling OTP, collect2 or cc1 crashes

You are running x86-64 containers under Rosetta (Apple Silicon with --platform linux/amd64). The current Dockerfile does not do that: it builds natively and cross-compiles only ERTS with clang. If you re-enabled an emulated path, expect random compiler crashes.

beam.smp segfaults immediately, even on Linux

A beam.smp linked with -static that still has a PT_DYNAMIC segment (readelf -l). OTP’s erts/configure hardcodes LIBZSTD="-Wl,-Bstatic -lzstd -Wl,-Bdynamic"; the trailing -Bdynamic makes lld link libc dynamically into the “static” binary. The Dockerfile patches it to -lzstd before configure.

Where to look

• build/logs/*.log.clean: console transcripts from the last smoke run.
• build/asterinas-build.log, build/release-build.log: build output.
• QEMU monitor (Ctrl-a c, then info registers -a) for a stuck guest.
• uniapp.mode=erl uniapp.eval="...": run an Erlang expression at boot without the Elixir CLI, for example uniapp.eval="io:format('~p~n',[inet:getifaddrs()])" (no double quotes inside the value).

Kernel command line

/init reads /proc/cmdline and copies it to the KERNEL_CMDLINE environment variable for the application. Values may be quoted with double quotes if they contain spaces; a value cannot itself contain a double quote.

Keys from EC2 user data are read before the kernel’s own line, so uniapp.mode=iex in user data turns an app image into an IEx one without rebuilding the AMI.

Kernel-side keys that matter:

Sample app environment (set in the release stage or by /init):

Make targets and variables

make help prints this list from the Makefile itself.

Targets

Variables

Files

Image contents

build/rootfs/ after make release:

/init                                static, 40 KB
/rel/erts-17.1/bin/beam.smp          static, 9.9 MB, JIT, OpenSSL 3.5 + zlib + zstd + crypto/asn1 NIFs linked in
/rel/erts-17.1/bin/erl_child_setup   static, 59 KB
/rel/lib/asn1-5.5.2/ebin
/rel/lib/compiler-10.0.6/ebin
/rel/lib/crypto-5.10/ebin
/rel/lib/elixir-1.20.4/ebin
/rel/lib/iex-1.20.4/ebin
/rel/lib/kernel-11.0.4/ebin
/rel/lib/logger-1.20.4/ebin
/rel/lib/public_key-1.21.7/ebin
/rel/lib/sasl-4.4/ebin
/rel/lib/ssl-11.7.7/ebin
/rel/lib/stdlib-8.1/ebin
/rel/lib/uniapp-0.1.0/ebin
/rel/releases/0.1.0/{start,start_clean}.{boot,script}, sys.config, vm.args, uniapp.rel, consolidated/
/rel/releases/RELEASES, start_erl.data
/etc/ssl/cacert.pem                  Alpine CA bundle, 230 KB
/dev /proc /sys /tmp                 empty mount points

Written by /init at boot: /etc/inetrc, /etc/resolv.conf, /etc/hosts.

Not present, by construction: sh, erl, erlexec, epmd, inet_gethost, heart, erl_call, escript, run_erl, to_erl, any .so, any dynamic loader, any Erlang source, include or documentation file. scripts/assemble-rootfs.sh fails the build if any ELF other than the three listed is found.

Sizes

OTP applications excluded from the build

Configured out with --without-* in the Dockerfile and therefore absent from both the cross and the native OTP: wx, odbc, debugger, observer, et, megaco, jinterface, cdv, dialyzer, diameter, eldap, erl_docgen, ftp, mnesia, os_mon, reltool, snmp, tftp, common_test, edoc, ssh, runtime_tools, syntax_tools. tools, eunit, inets, xmerl, parsetools and erl_interface are built (Elixir needs some of them at compile time) but only enter the image if your release lists them.

Limits and measurements

All numbers from 2026-10-07, QEMU 11.1 with TCG on an Apple M-series host (x86-64 emulated), 2 vCPUs. KVM numbers will be considerably better.

Memory

beam.smp RSS with the sample app, -mode interactive (default): 62 MB, 234 modules loaded. With -mode embedded: 116 MB, 742 modules. erlang:memory(total) is 38 MB versus 63 MB.

Boot time

Under TCG every instruction is emulated; most of the application-start time is the JIT compiling modules on first load. The GOALS.md target of 2 s from kernel entry to prompt has not been measured with KVM yet.

Size

Shipped image 15.2 MB (9.4 MB initramfs + 5.8 MB kernel) against a 40 MB budget. Adding Phoenix costs 4 to 6 MB compressed.

Not supported

• Erlang distribution and epmd: nothing in the image, no -name/-sname.
• Port.open/2 with external programs, System.cmd/3, os:cmd/1: there are no programs. erl_child_setup is present, so the mechanism works; there is just nothing to execute.
• NIFs loaded at runtime: no dlopen. OTP’s crypto and asn1 are linked statically.
• Persistent storage: the initramfs is read-only; /tmp is RAM. ext2 data volumes can be mounted (uniapp.data=, uniapp.mounts=); no journal (periodic sync, fsync for critical writes), no online resize, one NVMe I/O queue of 32 entries on EBS.
• Hot code upgrades: no writable release directory (and -mode interactive loads from the read-only tree anyway).
• Terminfo: TERM=dumb, no colours in IEx.
• IPv6: sockets and loopback work; no SLAAC/DHCPv6, so no global address without static configuration. DHCP: IPv4 only, one lease per boot (renewals are handled by the kernel client on Asterinas; /init on Linux does not renew).
• EC2: x86-64 Nitro instances with ENA only (tested on t3.small). The ENA driver has checksum offload and up to 8 queue pairs with RSS but no LLQ or TSO; about 100 MB/s NIC to NIC with one pair. One NVMe I/O queue. No IPv6.
• Power: the ACPI power button and S5 work (EC2 stop/reboot/terminate); other ACPI events (sleep button, lid, thermal) are ignored.
• Asterinas only: interface configuration from the command line (the address is compiled into the kernel), binding a socket to 0.0.0.0 and reaching it over loopback, hardware other than virtio.

Architecture

 QEMU / KVM (x86-64)
 ┌───────────────────────────────────────────────────────────────┐
 │ Asterinas kernel (Rust, Linux ABI)          5.8 MB ELF        │
 │   virtio-net ─ smoltcp TCP/IP   virtio-rng   ramfs (initramfs) │
 ├──────────────── syscalls (Linux ABI) ─────────────────────────┤
 │ /init  (static C, PID 1, 40 KB)                               │
 │   mount, ifconfig, inetrc, env, exec ─┐                       │
 │                                       ▼                       │
 │ beam.smp (static musl, OTP 29.1.1, JIT, OpenSSL 3.5 inside)   │
 │   kernel · stdlib · sasl · crypto · ssl · public_key · asn1   │
 │   elixir · logger · iex · your release                        │
 │   erl_child_setup (idle helper)                               │
 └───────────────────────────────────────────────────────────────┘

Three design decisions shape everything else.

Run the real BEAM

The runtime is an unmodified Erlang/OTP 29.1.1 built from the upstream tag. No patches to ERTS, no alternative VM. Everything that works in OTP works here, with the limits that follow from having no other programs and no dynamic loader (see Limits). The price is that the kernel must speak the Linux ABI well enough for ERTS, which is a few hundred syscalls including fork/execve (for erl_child_setup), epoll, timerfd, futex, socketpair and the socket API.

Replace the kernel, keep the ABI

Asterinas implements the Linux ABI in Rust with a small unsafe core. Because the ABI is the same, the identical initramfs boots on a stock Linux kernel (make run-m0), which gives a reference for every bug: if it works on Linux it is the kernel. Two Asterinas patches were needed, both small and both candidates for upstream (Asterinas patches).

Delete the userland instead of shrinking it

A typical Erlang deployment starts with a shell script (bin/myapp) that runs erlexec, which computes an argument vector and execs beam.smp. /init does what those two do, in C, with the arguments fixed at build time, and nothing else. There is no shell to run the script, no erl to find the emulator, no epmd, no inet_gethost (replaced by configuring OTP’s Erlang-level resolver). scripts/assemble-rootfs.sh enforces this: any ELF other than /init, beam.smp and erl_child_setup fails the build.

Static everything

beam.smp is linked with -static against musl, with OpenSSL, zlib, zstd and OTP’s own NIFs (crypto, asn1) linked in via --enable-static-nifs. Nothing is loaded at runtime except .beam files. The same applies to erl_child_setup and /init. This is what makes “no dynamic loader in the image” possible, and it removes an entire class of deployment problems.

Boot flow

1. The VMM loads the kernel ELF and the initramfs, passes the command line.
2. The kernel initialises devices, unpacks the cpio into its root file system, and executes /init.
3. /init configures the environment (The init program) and execs beam.smp with the argument vector a release’s bin/myapp start would have produced.
4. ERTS reads releases/<vsn>/start.boot, loads the listed applications (lazily in interactive mode), applies sys.config and config/runtime.exs, and starts the supervision trees. In iex mode -user elixir starts the shell on the serial console.
5. From here it is an ordinary node, minus the things in Limits.

What is deliberately not here

• An init system, service manager or process supervisor above the BEAM. OTP’s supervision trees are the supervisor. If the node stops, the VM is done; restart it from outside.
• A writable root. Configuration comes in through the kernel command line.
• A shell or any debugging tool other than IEx itself (and uniapp.mode=erl, uniapp.eval= for the cases where IEx is what is broken).

The init program

init/init.c is PID 1. It is about 300 lines of C, statically linked against musl, compiled for x86-64 with clang in the release Docker stage. It replaces the release’s shell script, erlexec, and an init system.

Steps

1. mount proc on /proc, devtmpfs on /dev, sysfs on /sys. Failures are logged and ignored (Asterinas has no devtmpfs; it populates /dev itself).
2. Read /proc/cmdline. param(key) returns the value of key=value, honouring double quotes.
3. Network: bring lo up with 127.0.0.1, find the first non-lo interface (SIOCGIFCONF, then /sys/class/net, then common names), set address, netmask and flags (SIOCSIFADDR, SIOCSIFNETMASK, SIOCSIFFLAGS), add the default route (SIOCADDRT). Each ioctl failure is logged with its errno and ignored, because Asterinas configures the interface itself.
4. Write /etc/inetrc ({lookup,[file,dns]}, so OTP never looks for inet_gethost), /etc/resolv.conf (the nameserver; inet_db reads this file and clears nameservers if it is missing) and /etc/hosts.
5. EC2 (uniapp.imds=1): fetch an IMDSv2 token, instance id, region, availability zone and type; fetch user data, save it to /run/user-data and prepend its key=value lines to the command line so that param() finds them first.
6. Time (uniapp.ntp): one SNTP exchange against Amazon Time Sync (169.254.169.123) or the given server, then clock_settime. On Nitro the firmware clock has been 0.4 to 2.7 s off at boot.
7. Data volumes (uniapp.data, uniapp.mounts): mount ext2 volumes read-write, by device node or by label (read from the superblock), retrying for 5 s while the block devices appear. ext2 is what the Asterinas kernel mounts; scripts/mkdata.sh makes a suitable image (4 KiB blocks, no dir_index). The supervisor calls sync(2) every uniapp.sync_s seconds (30) and unmounts everything before power off.
8. Export the environment: ROOTDIR, BINDIR (ERTS uses it to locate erl_child_setup; a missing BINDIR is fatal), EMU, PROGNAME, HOME, LANG=C.UTF-8, TERM=dumb, RELEASE_ROOT, RELEASE_NAME, RELEASE_VSN, RELEASE_MODE, RELEASE_NODE, RELEASE_SYS_CONFIG, ERL_CRASH_DUMP (/data/erl_crash.dump when a data volume is mounted, else /dev/null), KERNEL_CMDLINE, UNIAPP_DATA, EC2_*, AWS_REGION, EC2_IMDS_TOKEN.
9. Build the argument vector, fork and execv beam.smp in the child. /init stays PID 1 as a supervisor: it sleeps in poll(2) on the input devices (woken by SIGCHLD, a power button event, or a 1 s tick), reaps orphans, resyncs the clock every hour, syncs the volumes, and when the VM exits it logs the status, unmounts /data and performs uniapp.on_exit (default reboot(RB_AUTOBOOT)). The kernel’s restart path (patch 0006) then resets the machine; on EC2 the same AMI boots again, which is how a crashed node heals under an auto-scaling group. A power button press (SIGPWR from the Asterinas kernel, or KEY_POWER on /dev/input/event* on Linux) sends SIGTERM to the VM, waits up to 20 s, and powers off regardless of uniapp.on_exit; that is what EC2 stop and reboot requests become.

The argument vector

erlexec normally produces this; the layout matters because beam.smp and init parse different sections.

beam.smp
  -Bd                      emulator flags (erl's +X flags are -X here)
  [uniapp.emu flags]
  --                       end of emulator flags
  -root /rel -bindir /rel/erts-17.1/bin -progname erl
  --
  -home /
  --                       end of system flags, start of init flags
  -boot /rel/releases/0.1.0/start
  -boot_var RELEASE_LIB /rel/lib
  -mode interactive|embedded
  -config /rel/releases/0.1.0/sys
  -noshell                 (not in erl mode)
  [-eval EXPR]
  -user elixir -extra --no-halt +iex          iex mode
  -s elixir start_cli -extra --no-halt         app mode

+Bd disables the BREAK handler so Ctrl-C on the serial console does not open the menu. -user elixir is what bin/iex passes; -s elixir start_cli is what bin/elixir passes. Both were read from Elixir’s bin/elixir script at the pinned tag.

Compile-time constants

RELEASE_ROOT (/rel), RELEASE_NAME (uniapp), RELEASE_VSN (0.1.0) have defaults in the source; ERTS_VSN must be passed with -D and the Dockerfile reads it from the cross OTP tree so it always matches the shipped beam.smp.

Why C and not Rust or Erlang

It is 300 lines that do five ioctls and an exec. A static C binary is 40 KB, builds in a second with the cross clang that is already in the image, and has no runtime of its own to debug. Rust would be equally fine and larger; Erlang cannot run before beam.smp is running.

Build pipeline

Everything runs in builder/Dockerfile, natively on the Docker host’s architecture. Only the final ERTS binaries are cross-compiled for x86-64. This matters on Apple Silicon: an earlier version of the pipeline ran x86-64 containers under Rosetta, took 40 minutes, and crashed randomly in cc1 and collect2. The native pipeline takes about 6 minutes cold.

base       Alpine + clang + lld + an x86-64 Alpine sysroot (apk --arch x86_64 --root /sysroot)
   │          musl, gcc/g++ runtime, static OpenSSL 3.5, zlib, zstd, ncurses
   ├─ otp-src    clone OTP at $OTP_TAG, patch erts/configure (LIBZSTD -Bdynamic)
   │     ├─ otp-cross   otp_build configure --xcomp-conf=... ; boot -a ; release -a /opt/otp-x86
   │     │              relink beam.smp + erl_child_setup with -static ; run under qemu-user
   │     └─ otp-native  ./configure ; make ; install to /opt/otp        (for Elixir and Mix)
   ├─ elixir     clone at $ELIXIR_TAG, make, install (native OTP)
   ├─ release    mix release with UNIAPP_ERTS=/opt/otp-x86 ; cross-compile /init ; CA bundle
   └─ out        scratch image with /release, /init, /etc/ssl/cacert.pem, ERTS_VSN

make release exports the out stage to build/out and scripts/assemble-rootfs.sh lays out build/rootfs.

Cross-compiling OTP

OTP supports cross builds through otp_build configure --xcomp-conf=FILE; builder/xcomp/erl-xcomp-x86_64-alpine-linux-musl.conf sets CC="clang --target=x86_64-alpine-linux-musl --sysroot=/sysroot", lld as the linker, LIBS="-lcrypto -lssl -lz -lzstd" for the static NIFs, and the feature answers configure cannot test when cross compiling (erl_xcomp_*). otp_build boot -a builds a native bootstrap compiler, cross-compiles ERTS and the C parts of the libraries, and compiles all .erl files with the bootstrap. otp_build release -a installs the result.

.beam files are architecture independent, which is why the native OTP can run Elixir and Mix, and Mix can still assemble an x86-64 release: mix.exs points include_erts at the cross tree and Mix copies both ERTS and the OTP applications from there.

Three non-obvious fixes

LIBZSTD. erts/configure hardcodes LIBZSTD="-Wl,-Bstatic -lzstd -Wl,-Bdynamic". With -static, the trailing -Bdynamic makes lld link libc dynamically anyway; the result has a PT_DYNAMIC segment and segfaults before main. (bfd refuses with “attempted static link of dynamic object”.) The Dockerfile replaces it with -lzstd.

-static placement. Passing LDFLAGS=-static to configure breaks its dynamic-driver (-shared) conftests. It is passed to make instead, and only for the emulator target, then the two shipped binaries are relinked and copied over the installed PIE versions.

Static NIFs. --enable-static-nifs=yes links asn1rt_nif.a and crypto.a into beam.smp; their dependencies (OpenSSL etc.) must be in LIBS. The .so variants are still built and are deleted by the release’s prune step.

Verification inside the build

• file/readelf assert that beam.smp and erl_child_setup are x86-64, statically linked and have no INTERP.
• builder/cross-smoke.sh runs the cross-built beam.smp under qemu-x86_64 (user-mode emulation), boots the OTP start script, starts ssl and prints crypto:info_lib(). The build fails if OTP 29 or OpenSSL do not come up.
• scripts/assemble-rootfs.sh fails on any unexpected ELF.

The kernel build

scripts/build-asterinas.sh clones Asterinas at ASTERINAS_REF, applies every builder/asterinas-patches/*.patch, and runs cargo osdk build --release --boot-method vmm-direct --grub-boot-protocol multiboot --strip-elf inside the upstream asterinas/dev container (the exact tag is read from the checkout’s DOCKER_IMAGE_VERSION). OSDK rewrites e_machine to EM_386 so QEMU accepts the 64-bit multiboot ELF. The cargo registry is cached under build/asterinas-cache. About 4 minutes cold, 10 seconds for a rebuild after a one-line patch.

Caching

Each Docker stage is a cached layer keyed on its inputs. Changing app/ rebuilds only the release stage (about a minute). Changing OTP_TAG rebuilds otp-cross and otp-native (about 6 minutes). CI uses cache-from/to: type=gha.

Asterinas patches

scripts/build-asterinas.sh applies every builder/asterinas-patches/*.patch to the Asterinas checkout before building. The patches are small, were found by running this image, and should go upstream. Until then they are documented here so that nobody is surprised that the kernel is not pristine ec5bb9e.

0001: bind() to the unspecified address

Asterinas main (October 2026) resolves the interface for a bind() by looking the address up in the local routing table. 0.0.0.0 is in no table, so every wildcard bind fails with EADDRNOTAVAIL. That breaks gen_tcp:listen(Port, []), gen_udp:open(0, []) and OTP’s inet_res resolver, which binds its query socket to 0.0.0.0:0. (v0.18.1 had the same behaviour through a different code path.)

The patch (kernel/core/src/net/socket/ip/common.rs, resolve_bind_iface_and_config) treats an unspecified address as “the default interface”: the last registered one, which is the virtio NIC, falling back to loopback. It also rewrites the endpoint address to that interface’s own address, because aster-bigtcp dispatches incoming packets by (addr, port) and a socket left bound to 0.0.0.0 never receives anything (and, as a side effect, exhausted the kernel heap while retrying).

Consequence: a socket bound to 0.0.0.0 on Asterinas receives traffic on the NIC only, not on loopback. The sample app’s self-tests therefore connect to the NIC address instead of 127.0.0.1. The proper fix is a socket attached to all interfaces; this patch is the smallest change that makes OTP work.

0002: timerfd_settime must invalidate epoll readiness

Symptom: with QEMU TCG the image sometimes stalled right after /init exec’d beam.smp, or later mid-run. With one vCPU it stalled every time, with two about every second boot, with four about one in six. QEMU monitor samples showed one vCPU permanently in kernel mode, first apparently in handle_pending_signal, with more samples in epoll_pwait, ReadySet::poll and TimerfdFile::poll. The console log of a stalled run contained roughly 500,000 epoll_pwait calls: ERTS’s poll thread was spinning at 100 % and starving the schedulers.

ERTS uses a timerfd for sub-millisecond timeouts in erl_poll: arm it with timerfd_settime, epoll_wait(-1), then disarm with timerfd_settime(0), without ever read()ing the descriptor. Asterinas’s TimerfdFile::set_time resets the expiration counter but did not invalidate the Pollee’s cached readiness, so epoll_wait kept returning the disarmed timer as readable.

A 30-line C program (build/clk/init.c in the repository history; the loop is “arm 200 ms, epoll_wait, disarm, measure”) showed 19 of 20 waits returning in 0 ms on Asterinas and 0 of 20 on Linux. The fix is one line in kernel/core/src/time/timerfd.rs, set_time: self.pollee.invalidate() after ticks.store(0). After it: 0 spurious wake-ups, 8 of 8 clean boots at one and two vCPUs, and the memory floor dropped from 256 MB to 144 MB because the runaway poll thread no longer exhausted the kernel heap.

Lesson recorded for the next kernel: when a “livelock” log is dominated by one syscall returning immediately, suspect spurious readiness before locks, and write the ten-line reproducer first.

0003: runtime interface configuration (SIOCSIFADDR and friends)

Symptom: /init configures eth0 the way ifconfig does, with SIOCSIFADDR, SIOCSIFNETMASK, SIOCSIFFLAGS and SIOCADDRT. On Asterinas every one of them returned ENOTTY; the kernel only implemented the GET variants and hardcodes 10.0.2.15/24 via 10.0.2.2 for the virtio NIC in kernel/core/src/net/iface/init.rs. That happens to match QEMU’s user-mode network, so nothing visibly broke under QEMU, but on EC2 the address comes from DHCP and must be set at runtime.

The patch adds the setters:

• aster-bigtcp: Iface::set_ipv4_cidr and Iface::set_ipv4_gateway update smoltcp’s ip_addrs and default route under the interface lock.
• kernel/core/src/net/route: the two route managers move into an RwLock and route::reload() rebuilds the local/main tables from the interfaces’ current addresses, so bind() and output-interface lookups see the new address immediately.
• kernel/core/src/net/socket/ip/ioctl.rs: SIOCSIFADDR, SIOCSIFNETMASK, SIOCSIFBRDADDR (accepted; broadcast is derived), SIOCADDRT and SIOCDELRT for the default route (struct rtentry, gateway only).
• kernel/core/src/net/socket/util/ioctl.rs: SIOCSIFFLAGS is accepted (interfaces are always up).

Verified by booting with uniapp.ip=10.0.9.77/24 uniapp.gw=10.0.9.2 on a QEMU user network 10.0.9.0/24: the guest answers on the new address and the DNS and TLS probes pass, which they cannot with the compiled-in 10.0.2.15. Limits: one IPv4 address per interface, default route only.

0004: in-kernel DHCPv4 client (ip=dhcp)

Asterinas has no AF_PACKET sockets, so the DHCP client in /init cannot run there, and on EC2 the address is only available via DHCP. smoltcp, the network stack inside aster-bigtcp, ships a DHCPv4 client socket; this patch wires it in:

• Cargo.toml: enables smoltcp’s socket-dhcpv4 feature.
• aster-bigtcp: EtherIface::new_dhcp creates an interface without an IPv4 address and a Dhcpv4Socket next to it. Ingress UDP from port 67 to 68 is fed to the client, the client’s DISCOVER/REQUEST packets are emitted through the normal UDP path (so broadcast and the unspecified source work), and the client’s retry timers take part in next_poll_at_ms. While no address is set, unicast UDP to a not-yet-owned address is accepted, because servers may unicast the OFFER to yiaddr. On Configured the lease is applied with the setters from patch 0003, and CONFIG_GENERATION is bumped.
• kernel/core/src/net/route: the route tables rebuild lazily on the next lookup when CONFIG_GENERATION changed, since the lease arrives in the poll path, where the route lock cannot be taken.
• kernel/core/src/net/iface/init.rs: ip=dhcp on the command line selects new_dhcp for the virtio NIC; otherwise behaviour is unchanged.
• /proc/net/dhcp: one line per DHCP-configured interface, eth0 10.0.2.15/24 10.0.2.2 dns 10.0.2.3, or eth0 pending. Linux has no such file (it has no in-kernel DHCP client); it is how /init learns the nameserver.

Measured under QEMU: the lease arrives 33 ms after boot, /init logs net: eth0 10.0.2.15/24 gw 10.0.2.2 dns 10.0.2.3 (kernel dhcp), and make smoke-m1 passes without any uniapp.ip= parameter.

0005: the ENA network driver

EC2 Nitro instances have one NIC, the Elastic Network Adapter (PCI 1d0f:ec20, also 0ec2, 1ec2, ec21). Asterinas has no driver for it, so the image had no network on EC2 ([init] no ethernet interface found). Patch 0005 adds kernel/core/comps/ena, a component crate of about 1000 lines, and teaches net/iface/init.rs to use it for eth0 when there is no virtio-net.

Design, following ena_com.c in Linux so the pieces map one to one:

• regs.rs: the BAR0 register map.
• admin.rs: device reset, “readless” MMIO (the device answers register reads by DMA into a host buffer; direct reads are the fallback), a 32-entry admin queue polled without interrupts, GET_FEATURE, SET_FEATURE, CREATE_CQ, CREATE_SQ. The AENQ is allocated and registered because the device insists, but no event group is enabled.
• io.rs: 16-byte Tx/Rx descriptors, 8/16-byte completion descriptors, 128-entry rings in DmaCoherent memory with phase-bit completion.
• device.rs: AnyNetworkDevice for one queue pair. Rx buffers are 4 KiB pool segments handed to the device by req_id and refilled on completion; the MTU is set to 1500 so no frame spans two buffers. Tx uses one descriptor per packet (no meta descriptor, no offloads: smoltcp computes every checksum). Doorbells: Tx on every send, Rx and the CQ heads at the end of each poll, then the interrupt is unmasked.
• Interrupts: MSI-X vector 1 for the queue pair (vector 0 would be the admin queue and stays masked). Because MSI-X delivery on Nitro had not been exercised by this kernel before, the driver also raises the network softirqs from the timer tick every 4 ms; both paths are idempotent and the tick only bounds latency if a message is lost.
• Diagnostics: found ... and ... ready go through early_println!, so they appear at loglevel=error; if no ENA function is found the driver lists every unclaimed PCI function. Per-packet logging is at debug.

Measured on a t3.small: the lease arrives 1.6 s after power-on, BEAM is up at 4.4 s, and scripts/smoke-ec2.sh passes 8 of 8 twice in a row (TCP and TLS 1.3 echo from the internet, DNS and TLS client probes). Known limits: one queue pair, no LLQ, no RSS, no checksum or segmentation offload, no AENQ handling (link changes and keep-alives are ignored), no device reset after a fatal error.

Lesson from building it: the ENA loop is build, publish, launch, read the console, about six minutes per iteration and QEMU cannot shorten it (it has no ENA model). Two of the five iterations were spent on a tooling problem rather than the driver: make ami had silently rebuilt the kernel from the patch directory while a stale untracked file made patch 0004 fail to apply, so the image on EC2 lacked both DHCP and ENA. scripts/build-asterinas.sh now cleans the tree and aborts when a patch does not apply.

0006: power-off fallback and clock_settime

Two unrelated small changes, both needed to run unattended on EC2.

Restart: sys_reboot existed, and the kernel already registered the ACPI reset register and the i8042 controller as restart handlers (upstream’s register_restart_handler! registry, tried in descending priority). On Nitro neither exists (no reset register in the FADT, no keyboard controller), so reboot(2) fell through to machine_halt and the instance sat there. The patch fills in ostd::arch::power::try_restart (upstream left a TODO there) with a triple fault, the last resort Linux uses too: load an empty IDT and int3. The kernel registers it at Priority::LOW and moves the i8042 reset to Priority::new(1) so the order stays ACPI, i8042, triple fault. A Priority::LOW power-off handler falls back to a restart: without ACPI AML there is no S5 (until patch 0008), and a guest that asks to power off should reboot instead of hanging with a dead console. Under QEMU OSTD’s isa-debug-exit handler is registered at HIGH and wins.

Wall clock: there was no clock_settime/settimeofday; the realtime clock was boot time (from the RTC) plus the monotonic counter. The patch keeps that and adds an atomic offset that set_realtime stores and all realtime readers (the CLOCK_REALTIME* clocks and the vDSO bases) apply. Monotonic clocks are untouched, as on Linux. CAP_SYS_TIME is required. Verified with a 40-line C program: after clock_settime(2000000000), clock_gettime, gettimeofday (vDSO), CLOCK_REALTIME_COARSE and time() all report it and CLOCK_MONOTONIC is unchanged. On Nitro the firmware clock has been 0.4 to 2.7 s off at boot; /init corrects it with SNTP from Amazon Time Sync.

0007: NVMe on EBS

Symptom: nvme: Device initialization error: Err(CommandFailed) on every Nitro instance, after Create I/O Completion Queue (the third admin command). The driver worked under QEMU.

Cause: the EBS controller advertises CAP.MQES = 31 (32 queue entries); the driver asked for 64, the controller answered “Invalid Field in Command” (status 0x4205, SC 0x02). QEMU’s model allows 2048. The driver also never issued Set Features (Number of Queues), which the spec requires before creating I/O queues; EBS tolerates that, but the patch adds it since Linux does it and other controllers enforce it.

Fix: read CAP.MQES, cap the I/O queue size at it, give each ring a depth field so head/tail/phase wrap at the negotiated size while the storage stays QUEUE_DEPTH, and log CAP plus any rejected admin command through early_println! so the next controller quirk is visible at loglevel=error. Result: /dev/nvme1n1 (a data volume on /dev/sdf) mounts as ext2 on a t3.small and the boot counter survives a reboot.

A second fix landed in the same patch later: under write load (File.write plus a concurrent sync) the driver panicked with “SQ free_slots indicated space for this submit_with_items”. The device advances the SQ head when it fetches a command, before the completion arrives, so free_slots() can report a slot whose request is still outstanding; submit_with_items then refused it and the caller expected success. It now enqueues what fits, hands the rest back, and the caller waits for completions. The driver keeps one I/O queue: EBS limits the depth (32), not the count, and gp3’s baseline IOPS are served fine by it.

0008: ACPI power button and S5

Symptom: aws ec2 stop-instances left the instance in stopping for four minutes and reboot-instances took as long; both only completed when EC2 gave up and hard-reset. EC2 implements both by pressing the ACPI power button; the kernel had no idea the button existed, and poweroff could at best reset the machine (patch 0006), which stop-instances does not accept.

The patch adds two things, both without an AML interpreter:

• ostd collects the PM1a/PM1b event and control ports from the FADT and scans the DSDT and every SSDT for Name (_S5_, Package {...}), reading the two sleep-type constants. Nitro keeps _S5 in an SSDT with sleep type (0, 0); QEMU has it in the DSDT, also (0, 0). Unparsed or missing encodings are printed as raw bytes so the next firmware can be added.
• kernel/core/src/arch/x86/power.rs enables PWRBTN_EN, polls PM1_STS.PWRBTN_STS from the timer tick every 50 ms (the SCI stays masked at the IOAPIC, so no interrupt routing is needed), clears it and sends SIGPWR to PID 1, at most once per 5 s. A power-off handler at Priority::FIRMWARE writes SLP_TYP and SLP_EN to the PM1 control registers before falling back to a reset.

A side fix in ostd: the “running in QEMU” check accepted the KVMKVMKVM hypervisor signature, which Nitro also presents, so the QEMU isa-debug-exit handler was installed on EC2 and shadowed the ACPI path. It now also requires the firmware OEM ID BOCHS .

/init handles SIGPWR by sending SIGTERM to beam.smp (ERTS runs init:stop/0), waiting up to 20 s, then powering off; on Linux it watches /dev/input/event* for KEY_POWER instead, since that kernel reports the button as an input event. Measured on a t3.small: stop-instances reaches stopped in 14 to 20 s, reboot-instances is back with the next boot in 41 to 48 s, on both kernels.

0009: ENA driver, second round

Patch 0005 was the minimum that passes traffic. 0009 adds what a service behind a load balancer needs; all of it in kernel/core/comps/ena.

• AENQ. Link change, fatal error, warning, notification and keep-alive groups are enabled; the queue is drained every 100 ms. A keep-alive arrives every second; after 6 s without one (Linux’s KALIVE_TIMEOUT) the driver resets the device. The admin interrupt stays masked and the AENQ is polled (ena.aenq_irq=1 unmasks it; with it unmasked the guest hung after a reset, see below).
• Reset path. AdminQueue::reinit resets the device and re-programs the admin queue; the queue pairs, Rx buffers, RSS and AENQ are then rebuilt. Resets are requested from the health check and performed in the network softirq. ena.test_reset=SECONDS forces one. Measured on a t3.small: 2 ms for the reset, and the device’s own counters (GET_STATS) start moving again within 10 s, after which TCP from the internet works. If they do not move while the stack is sending, the driver reboots the machine, which is known to restore service.
• Checksum offload. STATELESS_OFFLOAD_CONFIG is queried; for IPv4 TCP/UDP frames the driver writes the pseudo-header sum into the L4 checksum field (Linux’s CHECKSUM_PARTIAL), sends one cached meta descriptor per queue with the header geometry and sets L4_CSUM_EN|L4_CSUM_PARTIAL. smoltcp’s checksum capabilities are set to None for TCP/UDP when the device also verifies on Rx; frames the device flags bad are dropped. Getting the pseudo-header right mattered: with the full checksum left in place the device produced wrong sums and every TLS handshake stalled.
• Queue pairs and RSS. ena.queues=N|auto (default auto: one per vCPU, max 8). Each pair has its own MSI-X vector; Rx is drained round robin; Tx is steered by a hash of the IPv4 5-tuple so one connection stays on one queue. RSS: Toeplitz over the same tuple, 128-entry indirection table over the Rx completion queues. The first version alternated Tx queues per packet, which reorders a flow’s segments on the wire and halved single-flow throughput (34 to 51 MB/s with two pairs vs 77 to 108 with one); with flow steering two pairs measure 77 to 95 MB/s, one pair 70 to 108, on t3.small and t3.medium, so multi-queue is the default. The echo benchmark is BEAM-bound at that level (1 KiB lines), the NIC has headroom.
• Tx completion accounting counts the descriptors each request used (meta + data) instead of trusting sq_head_idx.

Two things learnt the expensive way. First: never do MMIO reads or admin commands from the timer interrupt. The first version polled the AENQ and read DEV_STS from the tick while holding the device lock; a readless MMIO read can take milliseconds, and about one boot in five stalled right after DHCP with the network softirq locked out. Moving the health work into the softirq (the tick only sets a flag and raises it) fixed it. Second: a device that passes no traffic after a reset is not necessarily broken; the Nitro ENA needs a few seconds before its data path is live again, and hammering it with GET_STATS in that window made it worse.

aster-bigtcp gains TxPacket::data_writer for the in-place checksum patch.

Reporting upstream

The patches are git diff output against ec5bb9e, apply in order with git apply, and some depend on earlier ones (0004 on 0003, 0005 and 0009 on 0003/0004, 0008 on 0006). scripts/upstream-branches.sh turns them into commits on the fork github.com/bmalum/asterinas, one topic branch per upstreamable unit (fix/bind-unspecified-address, fix/timerfd-settime-readiness, feat/net-runtime-ifconfig-dhcp, feat/ena-driver, feat/x86-poweroff-clock-settime, fix/nvme-mqes-number-of-queues, feat/acpi-power-button-s5) plus elixir-unikernel-all; the commit messages are these sections.

Pull requests against asterinas/asterinas, opened 2026-10-10:

Checking the stack like upstream CI does

Upstream tests every pull request on its own, on x86-64, riscv64 and loongarch64, so each patch must be rustfmt- and clippy-clean at its point in the stack, not only with all nine applied (0003 once used an import that 0004 added; 0006 referred to the vDSO, which LoongArch lacks). scripts/aster-lint.sh REF ARCH... runs the same format_all.sh --check and clippy_check.sh workspace in the pinned dev image against any commit of build/aster-fork; --matrix "REF:ARCH ..." runs several. A run takes 5 to 10 minutes under Rosetta, so never block on it: start it with scripts/bg.sh start lint -- scripts/aster-lint.sh ... and poll with scripts/bg.sh wait lint 60. bg.sh kill lint also kills the labelled container; a cancelled docker run otherwise keeps holding cargo’s build-directory lock and every later build silently waits for it.

Upstream CI (make check) runs rustfmt and clippy with warnings as errors; the patches are formatted with tools/format_all.sh and clippy-clean as of the 2026-07-21 nightly (is_multiple_of, let chains, while let). Two tests in that CI are known flaky and unrelated to these changes: ostd::arch::vm::vmx::vmcs::test::switch_and_reuse_vmcs (ktest) and LTP epoll_wait04 (a 5 ms timing bound under TCG).

When one is merged upstream, bump ASTERINAS_REF and delete the patch; build-asterinas.sh fails loudly if a remaining patch no longer applies. 0005 is a new crate rather than a fix and would go upstream as a pull request adding kernel/core/comps/ena plus the two-driver selection in iface/init.rs. To send them: open issues on asterinas/asterinas with the C reproducer for 0002 and the gen_tcp:listen reproducer for 0001; the project’s CONTRIBUTING asks for a regression test under test/initramfs/src/regression. If upstream fixes land, bump ASTERINAS_REF and delete the patch; build-asterinas.sh fails loudly if a patch no longer applies.

Why these kernels

The research behind the kernel choice is in docs/RESEARCH.md in the repository; this is the short version.

What ERTS needs from a kernel

Threads with futexes, mmap/munmap (and optionally mremap), epoll or poll, timerfd, eventfd, pipes, Unix socket pairs, fork+execve for the erl_child_setup helper that ERTS starts unconditionally, BSD sockets with sendmsg/recvmsg, getrandom or /dev/urandom, a file system to load .beam files from, signals, and a tty for the shell. It does not need dlopen when NIFs are linked statically.

Asterinas (chosen)

A Linux-ABI compatible kernel written in Rust with a small unsafe core (“framekernel”), 245 syscalls on x86-64 at the pinned commit, virtio-net with a TCP/IP stack, initramfs, ramfs, ext2, SMP. An unmodified static musl beam.smp runs on it as PID 1. The project ships a Docker development container and a build tool (cargo osdk), moves fast, and the two gaps we hit were fixable with a few lines each. MPL-2.0.

Trade-offs: it is a kernel/user split, not a single binary; its memory floor is set by the fixed load address (128 MB) rather than by our image; it is pre-1.0 and some socket options and ioctls are missing.

Hermit (future, M3)

A true unikernel in Rust: the application is linked into libhermit.a and boots as one image. It has threads, futexes, mmap, poll, sockets over smoltcp and virtio-net, but no fork/execve, no pipe/socketpair, no select, no sigaction, and a Newlib-based libc. Running ERTS on it means patching ERTS (the erl_child_setup forker as a thread, pipe emulation, no mremap), porting OTP’s and OpenSSL’s configure to an x86_64-hermit target, and populating its memfs from an embedded archive. The nanovms/erlang patch set for the Nanos unikernel is the closest prior art. Months, not weeks. Everything in this repository (release layout, /init semantics, smoke tests) carries over and becomes the regression baseline.

Considered and rejected

Theseus (no POSIX), Redox (microkernel with relibc, a different porting effort), kerla and maestro (stale or immature), Unikraft, Nanos and OSv (not Rust). A BEAM reimplemented in Rust (BOSS, enigma, AtomVM-style) is a different project: it gives up unmodified OTP, which is the point here.

The Linux reference kernel

make run-m0 boots the same initramfs on a stock Linux 6.1 built by the Firecracker project for its CI (virtio, serial console and initramfs support compiled in, no modules). It is only a debugging aid: anything that works on Linux and fails on Asterinas is a kernel issue, and vice versa. Alpine’s linux-virt was tried first and rejected because it ships virtio_net as a module, which would have forced Linux-only files into the image.

Development workflow

Layout

app/            sample release (echo servers, boot probes)
builder/        Dockerfile, OTP cross config, kernel patches
docs/           this manual (docs/book) and the original research notes
init/           /init
scripts/        rootfs assembly, initramfs, kernel fetch/build, smoke test, dist
site/           landing page
.github/        CI: smoke tests, GitHub Pages

Edit, build, boot

Before a pull request: make smoke from a clean build/ (make clean first), and make docs if you touched the manual (broken links fail the build).

Debugging a boot

• uniapp.mode=erl gives a plain Erlang shell without Elixir’s CLI.
• uniapp.eval="io:format('~p~n',[inet:getifaddrs()])" runs an expression at boot (single quotes only inside the value).
• uniapp.code=embedded reproduces the Mix release default if lazy loading is suspected.
• loglevel=debug on Asterinas logs every syscall with its thread id; it is verbose (500k lines for a boot) but grep -c SYS_ per syscall tells you what the VM is doing when it looks stuck.
• Ctrl-a c enters the QEMU monitor; info registers -a shows where each vCPU is; addr2line -e target/x86_64-unknown-none/release/asterinas-osdk-bin inside the dev container maps kernel addresses to functions.
• Boot the same image with make run-m0 to see whether Linux agrees.

Updating upstreams

OTP: change OTP_TAG, check erts/configure still has the LIBZSTD line the Dockerfile patches (the sed is followed by a grep that fails if not), rebuild, run make smoke. Elixir: change ELIXIR_TAG, confirm bin/elixir still passes -user elixir and -s elixir start_cli (that is what /init hardcodes). Asterinas: change ASTERINAS_REF, rebuild; if a patch no longer applies, check whether it was merged upstream and delete it, otherwise rebase it.

Style

Shell scripts are bash with set -euo pipefail where they can be, and run on macOS’s bash 3.2 (no local x=$1 combined declarations, no mapfile). C is C11, warnings clean with -Wall -Wextra. Elixir follows mix format. Commit messages say what changed and why in the first line.

FAQ

Is this a unikernel? Not strictly. It is a kernel and one process with a syscall boundary between them. Everything else a Linux distribution ships is gone. A single-binary version on Hermit is the stretch goal (Why these kernels).

Why not just use a minimal Linux (Alpine, Nerves, Firecracker)? You can, and make run-m0 shows the same image doing exactly that. The point of the project is the memory-safe kernel underneath and the discipline of an image with three static binaries. The Linux path is kept as the control group.

Why not Nerves? Nerves is Linux (Buildroot) plus the BEAM as the main process, with a full userland underneath (busybox, erlinit). It is production-proven and the right answer for most embedded Elixir. This project removes the userland and swaps the kernel.

Can I run Phoenix? Yes, see A Phoenix application. Pure-BEAM dependencies only.

Can I use NIFs? Only if they are linked statically into beam.smp, the way crypto is. Runtime dlopen is not possible.

Does it do DHCP? Yes, by default. On Linux /init has a client; on Asterinas the kernel has one (ip=dhcp, patch 0004). uniapp.ip= sets a static address instead. See the networking chapter.

Can two images talk over Erlang distribution? Not yet: there is no epmd, and distribution is a non-goal of the MVP. -proto_dist with a custom EPMD module would be the way.

How do I get logs out? The console is the only output. Ship logs over the network from the application.

Why does IEx have no colours? TERM=dumb, because there is no terminfo database in the image.

How big is it really? 9.4 MB image plus 5.8 MB kernel. The VM needs 144 MB of RAM on Asterinas (kernel load address), 128 MB on Linux. beam.smp uses about 62 MB RSS idle.

Why is boot 3 to 4 seconds here when the goal says 2? Those numbers are from QEMU TCG on an Apple Silicon host, emulating x86-64 instruction by instruction. With KVM the BEAM’s JIT warm-up is a fraction of that; not yet measured.

Does it run on real hardware? Untested. Asterinas boots on x86-64 machines with virtio or supported NICs; nothing in the image cares.

Why Alpine for the toolchain? musl, and apk --arch x86_64 --root gives a complete x86-64 sysroot with static OpenSSL in one command, which is what makes the native-plus-cross build simple.

Can I build on an x86-64 machine? Yes; the same Dockerfile runs natively there and the “cross” step is a no-op in spirit (it still produces a static build against the sysroot).

Changelog

All notable changes to this project are recorded here. The format follows Keep a Changelog; versions follow Semantic Versioning.

Unreleased

Work towards running on EC2 (docs/GOALS-EC2.md).

Added

• scripts/mkdisk.sh and make ami: a GPT disk image with an EFI system partition (GRUB, kernel, initramfs) that boots via OVMF from an NVMe drive; make smoke-disk runs the assertions on it. KERNEL=linux builds the same image around a Linux kernel.
• scripts/build-linux-ec2.sh: Linux 6.1 with ENA, NVMe and EFI stub for the Linux-kernel AMI.
• DHCPv4 client in /init (Linux, AF_PACKET), used whenever uniapp.ip= is absent; falls back to the QEMU defaults.
• Asterinas patch 0003: SIOCSIFADDR, SIOCSIFNETMASK, SIOCSIFFLAGS, SIOCADDRT/SIOCDELRT so the address can be set at runtime.
• Asterinas patch 0004: in-kernel DHCPv4 client (ip=dhcp) with the lease published in /proc/net/dhcp.
• Asterinas patch 0005: aster-ena, a driver for the EC2 Elastic Network Adapter (one queue pair, MSI-X plus timer-tick polling, polled admin queue). The Asterinas image boots on a t3.small, gets its address from the VPC’s DHCP, and passes the TCP, TLS 1.3, DNS and TLS-client checks from the internet.
• scripts/ami-publish.py, scripts/smoke-ec2.sh, scripts/ec2-console.sh, scripts/smoke-ec2-iex.sh (IEx over the EC2 Serial Console) and make ami-publish / smoke-ec2 / ami-clean; DISK_MODE=iex images.
• Manual chapter “Running on EC2”; ENA driver notes in the patches chapter.

Fixed

• scripts/build-asterinas.sh aborts when a patch does not apply and cleans stale untracked files first; previously it built silently without the failed patches.

• /init is now a supervisor: it forks beam.smp, reaps orphans and on exit reboots (or powers off/halts, uniapp.on_exit=) the machine, so a crashed node restarts instead of hanging.

• /init reads EC2 instance metadata and user data (uniapp.imds=1); user-data key=value lines override the kernel command line. Identity in EC2_*/AWS_REGION, raw user data in /run/user-data.

• SNTP in /init (uniapp.ntp=, Amazon Time Sync by default on EC2): the clock is set at boot and resynced hourly.

• Data volume: uniapp.data=auto mounts an ext2 EBS volume at /data (UNIAPP_DATA, erl_crash.dump there); scripts/mkdata.sh builds the image, scripts/ami-publish.py --snapshot-only publishes it.

• Sample app: Uniapp.Cloudwatch ships the log to CloudWatch Logs and publishes EMF metrics (BootCount, Uptime, MemoryTotal, ProcessCount) with SigV4 and instance-role credentials; Uniapp.Data keeps a boot counter on /data.

• Asterinas patch 0006: reboot(2) resets the machine on Nitro (triple fault fallback), poweroff falls back to reset, clock_settime and settimeofday implemented.

• Asterinas patch 0007: NVMe honours CAP.MQES and issues Set Features (Number of Queues), so EBS volumes work.

• scripts/smoke-ec2.sh with DATA_SNAPSHOT: IAM role, data volume, user data, guest-initiated reboot, boot counter and CloudWatch assertions, then timed aws ec2 reboot-instances and stop-instances (23 in total); make smoke-ec2 wires it up.

• Asterinas patch 0008: ACPI power button (polled PM1_STS, delivered as SIGPWR to PID 1) and S5 power-off from the _S5 package found in the DSDT/SSDTs. aws ec2 stop-instances now stops in about 15 s and reboot-instances reboots in about 45 s instead of EC2’s 4-minute hard reset. /init handles SIGPWR (Asterinas) and KEY_POWER on evdev (Linux) by stopping the VM with SIGTERM and powering off; the Linux EC2 kernel gains ACPI_BUTTON.

• Sample app: Uniapp.Health serves /healthz (503 until ready, then 200 with JSON details) and /livez on 8080 for ALB/ASG health checks, and a throughput self-test (BENCH), also against a peer (uniapp.bench_peer).

• Asterinas patch 0009: ENA AENQ handling with keep-alive watchdog and a device reset path, TCP/UDP checksum offload, multiple queue pairs with RSS (ena.queues=, default one pair per vCPU; Tx steered by flow hash so a connection stays on one queue), and health work moved out of the timer interrupt. scripts/bench-ec2.sh measures NIC to NIC: 77 to 108 MB/s round trip between two t3.small.

• scripts/smoke-ec2.sh: health endpoint, BENCH/BULK, ENA ready and keep-alive assertions (29 in total); QUICK=1 short mode.

• Storage: uniapp.mounts= for additional ext2 volumes, LABEL= device specs (mkdata.sh takes a label), periodic sync in /init (uniapp.sync_s, default 30 s); a write without fsync survives a hard kill. Patch 0007 also fixes an NVMe submission-slot race that panicked the kernel under write load.

• /init sleeps in poll(2); the Linux power button (evdev) is seen in about 40 ms instead of up to 250 ms.

• IPv6 probes (uniapp.probe_ipv6=1): loopback works on both kernels; no SLAAC/DHCPv6, documented. Hibernation documented as unsupported.

• scripts/upstream-branches.sh: the patches as topic branches on the Asterinas fork; upstream pull requests asterinas/asterinas #3983 to #3989 opened from them. Asterinas base moved from 3d85cb4 to ec5bb9e (2026-09-17); patch 0006 uses upstream’s new restart/poweroff handler registry and fills in ostd::arch::power::try_restart, 0008 registers S5 at Priority::FIRMWARE. Patches are rustfmt- and clippy-clean.

Changed

• make run-*/smoke-* no longer pass uniapp.ip=; both kernels use DHCP.
• /init no longer execs beam.smp; the VM runs as PID 2 under the supervisor. ERL_CRASH_DUMP points to /data when a volume is mounted.

0.1.0 - 2026-10-07

First working release: milestones M0 (stock Linux), M1 (Asterinas) and M2 (TLS, size budget, CI) of GOALS.md.

Added

• Static x86-64 Erlang/OTP 29.1.1 (beam.smp with OpenSSL 3.5, zlib, zstd and the crypto/asn1 NIFs linked in) and Elixir 1.20.4, built natively with only ERTS cross-compiled; about 6 minutes cold.
• /init: a 300-line static C program replacing the release shell script, erlexec and an init system. Modes iex, app, erl; network, DNS and code-loading mode from the kernel command line.
• Sample release with TCP, UDP and TLS 1.3 echo servers and boot-time DNS and TLS client probes.
• Asterinas kernel build (3d85cb4, multiboot ELF) with two local patches: wildcard bind() and a timerfd_settime readiness fix.
• Stock Linux 6.1 reference kernel target for A/B debugging.
• scripts/smoke.sh: boots both modes on both kernels and asserts the GOALS.md criteria, including host-side TCP/UDP/TLS clients; GitHub Actions workflow running it.
• make check, make dist (versioned, checksummed bundle with run.sh), make help, make docs, make site.
• Manual (mdBook) and landing page, published to GitHub Pages.

Measurements

• Image 9.4 MB, kernel 5.8 MB, total 15.2 MB (budget 40 MB).
• Runs with 128 MB RAM on Linux, 144 MB on Asterinas.
• Kernel entry to /init 0.45 s (Asterinas) / 1.0 s (Linux) under QEMU TCG; application start 3 to 4 s under TCG.