DocsNodes

Get started

Nodes

Run sandboxes on a Linux machine of your own, a home server or a VPS, under the same deployment, CLI, API, SDKs and dashboard, with cloudex node add and cloudex new --on. What works there, ssh straight to it from your LAN or tailnet, how isolated it is, and what the machine can see.

A node is a Linux machine of your own, such as a home server or a VPS, that runs sandboxes for your deployment. An agent on it, cloudex-node, dials out to your deployment and stays connected. The machine needs one inbound port, on its LAN and tailnet addresses only, so that ssh from a computer near it can go straight there; that port answers only with a ticket from your deployment, and --direct off closes it. Sandboxes on a node work with the same CLI, API, SDKs and dashboard as sandboxes on Cloudflare. You pay for the machine instead of Cloudflare’s container time.

cloudex node add box                         # prints the command to run on the machine
cloudex node list                            # box online, with its CPUs, memory and isolation
cloudex new --on box --cpus 4 --memory 16G --name n1
cloudex ssh n1

Nodes are new in v0.2.2. The deployment and the CLI both need v0.2.2 or later. Direct connections and forks on nodes are new in v0.2.3: they need v0.2.3 on the deployment and the node’s agent, and direct connections need the v0.2.3 CLI too.

When a node is worth it

A node is worth it when you already have a machine with spare capacity, or when sandboxes would cost more on Cloudflare than a machine of your own:

  • Long-running sandboxes. A sandbox on a node has no 6-hour limit, and an idle time of 0 keeps it running until you stop it.
  • Bigger sandboxes. A sandbox on a node can have as many CPUs and as much memory as the machine has, not the 4 vCPU and 12 GiB of standard-4.
  • Sandboxes that stay up. Stopping and resuming take about a second, and the disk stays on the machine.
  • Working nearby. From the node’s LAN or tailnet, ssh, scp and editors go straight to it, as fast as plain ssh to the machine.

Each sandbox on a node still has its own Durable Object, and commands, previews and the dashboard reach it through your Worker, as for a sandbox on Cloudflare. Those count toward your Workers plan as usual. ssh from a computer on the node’s LAN or tailnet goes straight to the node instead. The container time is what you no longer pay.

Don’t use a node if:

  • You want to run untrusted code on plain Docker. Without Sysbox, code that escapes its container is root on the machine, and from there it reaches your LAN and tailnet. See Isolation.
  • The machine holds things a sandbox mustn’t reach, and you can’t give it Sysbox. The firewall keeps sandboxes off the host and the LAN, but under plain Docker it doesn’t protect against a kernel or container escape.
  • You need backups or --size. They aren’t available on nodes yet. See What differs.
  • The machine is arm64, has no systemd, or you have no root on it. Synology and Unraid aren’t supported yet.
  • The machine is far from you, or on a slow line. Unless you’re on its LAN or tailnet, every command and connection goes through Cloudflare, and saves go up over the machine’s own line either way. See Performance.

What you need

The installer checks each of these and stops with a message naming what’s missing:

  • Linux on x86-64 (amd64). arm64 machines aren’t supported yet.
  • Root, through sudo. The firewall needs it.
  • systemd, which loads the firewall at every boot.
  • nftables (nft) and iproute2 (ip). On Debian and Ubuntu: apt install nftables iproute2.
  • Docker Engine 28 or later, running. Earlier versions ignore the network priority that keeps the agent’s own route off sandbox networks, so the installer and the agent refuse them. It was tested with Docker 29 on Ubuntu 22.04.
  • At least 10 GB free in Docker’s data directory. The agent refuses new sandboxes below that, and keeps 5 GB free while they run (see Disk).
  • Outbound HTTPS to your deployment’s URL and to ghcr.io, where the agent and the sandbox image come from.
  • One inbound port, 7022, on the machine’s private LAN and tailnet addresses only, for direct connections. Nothing needs to reach it from the internet. --direct off opens no port at all.
  • A clock that keeps time, such as with NTP. A node whose clock is more than 60 seconds off is refused, and cloudex doctor warns from 30.
  • Optional: Sysbox, installed before the agent. Machines then run under Sysbox, which is safer and runs Docker inside. See Isolation.

Adding and removing nodes needs the admin key or an admin token.

Add a node

cloudex node add box

cloudex node add adds the node and prints the command to run on the machine, as root, with Docker running:

curl -fsSL https://cloudex.build/node | CLOUDEX_JOIN_TOKEN=cxj_... sudo --preserve-env=CLOUDEX_JOIN_TOKEN bash -s -- --api https://cloudex-api.<your-subdomain>.workers.dev --version v0.2.3 --digest sha256:...

The join token goes in the environment, not on the command line, so it isn’t in any process’s arguments on the machine. It works once, for 30 minutes; node add prints the time it expires. --digest pins the agent’s image: the installer pulls it by that digest, so Docker refuses an image that doesn’t match, and runs that exact image, never a tag that can move. cloudex deploy looks the digest up when it installs your deployment; a deployment that isn’t a release names the edge agent, without a digest. Adding the same name again before the node has joined gives a new token. --version is your deployment’s version, so the agent matches it. node add also prints a warning about plain Docker, which applies when the machine has no Sysbox.

Node names follow the rules for sandbox names: 2 to 32 lowercase letters, digits and hyphens, starting with a letter. They can’t start with nd-, which marks node ids. With --json, node add prints the node, the token, its expiry and the command.

In the dashboard, Nodes does the same for admins: Add node shows the command once.

What the installer does

The installer runs as root and changes the machine in these ways, in order. It says it worked only if every step did.

  1. Checks the machine, as listed in What you need, and whether Docker has Sysbox.
  2. Picks a pool for sandbox networks: a private /16 that collides with no route and no Docker network on the machine. --pool picks one yourself. It also picks the addresses for the direct port: the machine’s tailnet (100.64/10) IPv4 address on Tailscale’s interface, and its private (10/8, 172.16/12, 192.168/16) IPv4 addresses, leaving out Docker’s and other bridges and VPN tunnels (WireGuard, OpenVPN, ZeroTier, NetBird and the like). It prints each with its interface. If one is a network you don’t want reaching the port, such as a cloud provider’s private network or a shared LAN, run it again with --direct-addr for each address you do want; --direct off picks none.
  3. Installs the firewall: /etc/cloudex-node/firewall.nft and a systemd unit, cloudex-node-firewall.service, which loads it now, at every boot, and whenever nftables.service starts, restarts or reloads. A timer, cloudex-node-firewall-check.timer, loads it again within 30 seconds if anything else removed it. It’s one nftables table, inet cloudex, which touches only traffic to and from the sandboxes’ bridges (named cx…). Docker’s restarts don’t touch it.
  4. Joins your deployment with the token, from a one-off container that reads it on its standard input, so the token is in no process’s arguments and not in the agent’s settings. Then it starts the agent as a container, cloudex-node, with --restart unless-stopped, its own Docker network and a volume, both also named cloudex-node. The agent has the Docker socket, a read-only file system, no capabilities, at most 1 GiB of memory and 1024 processes. Unless --direct off, the agent then runs a small relay, cloudex-node-direct, which Docker publishes the direct port on, on those addresses only. The relay has no Docker socket, no capabilities, a read-only file system and 64 MiB of memory; it passes connections to the agent and holds nothing.
  5. Waits for the agent to come online and pass its isolation self-test. The agent pulls the sandbox image first, about 700 MB.
  6. Boots a throwaway sandbox, checks from inside it that the internet answers and the host and metadata addresses don’t, and removes it.

It ends with this node is installed and online, then the plain-Docker warning if the machine has no Sysbox, the direct port’s addresses, and the command that uninstalls it. On the test machine it took 23 to 29 seconds. cloudex node list then shows the node online.

Each sandbox on the node is a container named cloudex-<sandbox id> on a network of its own. Everything the agent makes carries build.cloudex.* labels. It reads, changes and deletes only containers, networks and volumes with those labels, and never prunes, so your own containers are safe. The installer keeps what it needs to update and uninstall in /etc/cloudex-node.

Option Effect
--version vX.Y.Z The agent’s version; node add gives your deployment’s
--digest sha256:… Pull the agent’s image by this digest, and refuse any other; node add gives it for a release
--pool CIDR The pool for sandbox networks, such as 10.250.0.0/16, inside 10.0.0.0/8 or 172.16.0.0/12
--block CIDR Another IPv4 range sandboxes may not reach; repeatable
--machine-image REF Run sandboxes from this image instead of the deployment’s
--allow-image REPO Another image repository sandboxes may run; repeatable. Without it, only ghcr.io/lucataco/cloudex-base.
--keep-free-gb N The floor for Docker’s free disk, 5 GB by default. See Disk.
--disk-gb N Cap each sandbox’s disk at N GB, where Docker’s storage driver can. See Disk.
--direct auto or off The direct port. auto, the default, publishes it on the machine’s tailnet IPv4 address and its private IPv4 addresses on LAN interfaces (not bridges or VPN tunnels), and prints them; off opens no port, and ssh goes through your deployment.
--direct-addr IP Publish the direct port on exactly this private or tailnet IPv4 address; repeatable. Use it on a VPS, with the tailnet address.
--direct-port N The direct port, 7022 by default
--unisolated Testing only, without root: no firewall. See Isolation.
--uninstall, --keep-images, --yes See Uninstall

Updating the agent

After cloudex deploy updates your deployment, run the installer again on each node with the new version. It needs no token: it keeps the node’s identity, its pool, its direct port’s settings and its sandboxes, and replaces the running agent. To pin the image, add --digest with the one cloudex node add prints for that version.

curl -fsSL https://cloudex.build/node | sudo bash -s -- --version v0.2.3 --digest sha256:...

Running sandboxes keep running while the agent is replaced, and the new agent joins their networks again as it starts. Open ssh sessions and commands end with the old agent. A node updated from v0.2.2 gets its direct port this way; until then, ssh to it goes through your deployment.

The agent doesn’t update itself. cloudex doctor warns about a node whose agent’s version differs from the deployment’s, and the deployment refuses an agent whose protocol it doesn’t speak, saying which version to install.

Make sandboxes on a node

cloudex new --on box --name n1
cloudex new --on box --cpus 4 --memory 16G --name big
cloudex run --on box -- npm test

--on takes a node’s name or id. --cpus (from 0.25) and --memory (such as 512M or 16G, from 512 MiB) set the sandbox’s limits on the node, up to what the node has. Without them, a sandbox gets its size’s CPUs and memory: 1 CPU and 6 GiB for standard-2, at most the node’s CPUs and half its memory. --cpus and --memory work only with --on. On Cloudflare, pick a --size instead.

cloudex new --on prints where the sandbox runs and its limits. On a node under plain Docker, it repeats the warning about trusted code. The other flags of cloudex new work as on Cloudflare.

cloudex list has a WHERE column: cloudflare, the node’s name, or the name with (offline) or (removed). cloudex info shows where a sandbox runs, its machine’s CPUs and memory, and its isolation.

A sandbox stays on the node it was made on. To move it, see Move between a node and Cloudflare.

In the dashboard

New sandbox has a Where choice when the deployment has nodes, with each node’s isolation, and CPUs and memory for a node. The sandbox list has a Where column, and a sandbox’s page shows where it runs, its machine and its isolation. Nodes, for admins, lists the nodes with their state, isolation, host and sandboxes, adds and removes them.

From code

import { Cloudex } from "cloudex";

const cx = new Cloudex();
const nodes = await cx.nodes.list(); // name, state, runtime, isolation, cpus, memoryMb, ...
const box = await cx.create({ name: "n1", node: "box", cpus: 4, memoryMb: 16384 });
console.log(box.placement); // { provider: "node", node: { name: "box", online: true, ... }, cpus: 4, memoryMb: 16384 }
from cloudex import Cloudex

cx = Cloudex()
nodes = cx.nodes.list()
box = cx.create(name="n1", node="box", cpus=4, memory_mb=16384)
print(box.placement, box.isolation)

nodes.get(name) adds the image the agent has and, for admins, a round trip to it, and nodes.add(name) and nodes.remove(name), with force, need an admin key. See SDKs.

When a node refuses a sandbox

Message starts with Why
node box hasn't joined yet The install command hasn’t run on it, or failed
node box was removed cloudex node rm removed it; add it again with cloudex node add box
node box is offline (last seen …) Its agent isn’t connected. See Offline.
node box's isolation self-test fails (…) The firewall isn’t working; it names the checks that failed. See Troubleshooting.
node box is still pulling the sandbox image The agent is fetching the image after an install or an update; try again when cloudex node info box says it’s present
an own-only token can't place sandboxes on node box Own-only tokens can’t use plain-Docker nodes. See Who may use a node.
cpus: node box has 8 CPUs --cpus or --memory asked for more than the node has
Docker's disk on this node has … GB free Fewer than 10 GB are free for Docker on the machine for a new sandbox or a fork’s copy, or fewer than 5 GB (the node’s floor) for a resume. See Disk.
a fork of n1 stays on node box A fork was asked for on Cloudflare or another node. Save the sandbox and start from the save instead; see Move.
a fork of n1 would have … image layers Forks of forks go about 100 levels deep at most. Make a named save of it and start from that.

What works, what differs

On a node
ssh, ssh-config, scp, rsync, editors, port forwarding The same. From the node’s LAN or tailnet they go straight to the node, elsewhere through your deployment.
exec, background commands, cp, the files API The same. Uploads are at most 100 MB, as on Cloudflare.
Preview links The same, WebSockets included
The dashboard’s terminal and desktop The same. The desktop’s first start installs it over the machine’s own line.
Settings, --repo, ~/.cloudex/start The same
Agents and provider keys The same. Provider calls go through the node’s agent; see What a node can see.
The cloudex command inside a sandbox The same, cloudex stop inside included
Idle stop The same rules. --idle 0 means never. While the node is offline, idle stop can’t act.
The 6-hour limit None
stop Stops the machine in about half a second. The disk stays on the node: no save and no backup.
resume Starts the machine again with its disk, in 1 to 2 seconds
Named saves Work. A save goes up over the machine’s own line, and records the node it was made on.
new --from <save> --on <node> Works, from a save made on Cloudflare or on any node
Sizes --cpus and --memory instead, for new, fork and new --from. resume --size is refused.
Forks, new --from <sandbox> The same, on the same node, running or stopped. A running sandbox pauses while its disk is copied. See Forks on a node.
Backups, resume --from-backup None: the disk stays on the node. Make a named save for a copy off the node.
Docker inside a sandbox With Sysbox only. Under plain Docker, there’s no Docker in the sandbox.
Disk Docker’s disk on the machine, shared by its sandboxes. See Disk.
delete Removes the machine and its network. While the node is offline, only with --force.

Disk

A node’s sandboxes share Docker’s disk on the machine, usually its root file system. On most machines (overlayfs on ext4) Docker can’t cap one sandbox’s share, so the agent watches the free space every 5 seconds instead. When it falls under the node’s floor, 5 GB unless the installer’s --keep-free-gb says otherwise, the agent stops the running sandbox that has written the most, and that sandbox’s notice says why. It stops another only while the space keeps falling. Resume a stopped sandbox once there’s room again, and delete what filled it.

The watchdog acts after the floor is crossed: a sandbox that writes fast can take the free space a little further down in the seconds before it stops. In testing, a sandbox writing about 135 MB/s was stopped with the free space 0.3 to 0.7 GB under the floor.

Where Docker’s storage driver has quotas (btrfs, zfs, or overlay2 on xfs mounted with pquota), the installer’s --disk-gb N also caps each new sandbox’s disk at N GB. Elsewhere the option does nothing, and the watchdog is the limit.

Fork images. A fork copies its original’s disk into an image on the node, which the forks made by that request share. The image takes about as much space as the original had written, or about twice that on Docker’s containerd image store, which Docker 29 uses on new installs. A fork is refused unless the copy leaves at least 10 GB free. The image goes with the last sandbox that uses it, and one that nothing uses is removed within the hour. On Docker’s older overlay2 store, an image that a fork of a fork came from stays until that one’s image goes too.

Stopping sandboxes doesn’t free fork images. When the watchdog stops a sandbox while fork images hold more than 1 GB, the notice says how much, and deleting forks frees it. cloudex node info shows how many fork images there are and their size.

Connections to the agent

A sandbox reaches its node’s agent for cloudex.internal and its provider calls. Each sandbox may hold at most 256 connections to the agent at once, and have 64 requests in flight; more are closed as they come, so one sandbox can’t use up the agent for the others. A preview’s WebSocket message is at most 1 MiB. Your deployment holds at most 64 open streams (commands, preview requests and connections) to one sandbox on a node; more wait up to 15 seconds for one to end.

Direct connections

ssh to a sandbox on a node goes straight to the node when your computer can reach it, on the same LAN or over a tailnet such as Tailscale. Elsewhere it goes through your deployment, as before. You don’t choose: the CLI tries both and takes the first that works.

What goes direct is everything that runs over cloudex ssh’s connection:

  • cloudex ssh, with port forwarding (-L) and remote commands;
  • plain ssh, scp, rsync and editors, through the block that cloudex ssh-config prints;
  • cloudex prompt and cloudex attach;
  • cloudex proxy, for any TCP port.

What doesn’t: the dashboard, its terminal and desktop, preview links, cloudex exec, run and cp, agents without a terminal, and the API and SDKs. They go through your deployment. For a web server in the sandbox, forward its port over ssh, which goes direct:

cloudex ssh n1 -- -L 3000:localhost:3000

How it works

The installer publishes one port on the node, 7022, on the machine’s private LAN and tailnet IPv4 addresses. For each connection, the CLI asks your deployment where the sandbox is. For a sandbox on a node, the deployment answers with the node’s addresses, its public key, and a ticket good for one connection to that sandbox and port, for 60 seconds.

The CLI then tries the addresses, starting them 50 ms apart: first those on your own LAN subnet, then tailnet addresses if your computer is on a tailnet, then other private addresses. Each try is a TLS connection whose certificate must carry the node’s key; anything else, such as another machine that has the address now, a VPN that swallows it, or a captive portal, fails that try. The first that proves to be the node gets the ticket, and the node checks it before it connects anything. If none has proved itself about 300 ms after the CLI asked, the CLI starts the path through your deployment too, and uses whichever connects first. When the node has no direct port, such as an older agent or --direct off, it goes through your deployment at once.

When the path through your deployment wins on a network, for the next 10 minutes the CLI starts it at once there, and still tries the node beside it. The CLI keeps the node’s addresses for 10 minutes too, so the next connection tries them before your deployment answers.

Both paths reach the same sshd, so ssh checks the sandbox’s host key either way, and known_hosts has one entry for both. Your deployment still decides who may connect: the ticket is minted only for a token that may use the sandbox, and only while it’s running.

A direct session counts as activity for idle stop, as on the other path. Revoking the token that opened it ends it within about 90 seconds (6 to 40 seconds in testing): each ticket names its token, and the node ends every session of a token that ended. cloudex stop ends it at once, and cloudex node rm ends it and closes the port. It doesn’t drop when you run cloudex deploy, or when the node’s own connection to your deployment does: in testing, a session lived through a 2-minute cut of the node’s line to Cloudflare, and a new one opened during it. While the node’s connection is down, though, a revoked token’s session lasts until it’s back.

Checking it

CLOUDEX_DEBUG=1 cloudex ssh n1 -- true
cloudex proxy: [  0.0ms] asking the API
cloudex proxy: [374.2ms] cx-7f3a2k is on node box (nd-k2m4p6q8), direct [192.168.1.20 100.101.102.103] port 7022
cloudex proxy: [375.5ms]   192.168.1.20:7022     lan-onlink    1.3ms  verified
cloudex proxy: [375.9ms] chose direct 192.168.1.20:7022 (lan-onlink)

CLOUDEX_DEBUG=1 prints each try, how long it took, why it failed, and which path was chosen. CLOUDEX_DIRECT sets the paths the CLI uses:

CLOUDEX_DIRECT Effect
auto (the default) Try the node and your deployment, as above
off Only through your deployment, as in v0.2.2
only Only straight to the node, or fail. For checks and debugging.

cloudex node info box shows the direct port’s addresses and whether it’s listening, off or failed, with the reason, to admins; other tokens see the node without them, since a sandbox’s own connection gets the addresses it needs. A node installed with --direct off shows Direct off. cloudex doctor tries a TLS connection to each online node’s addresses from your computer, with no ticket, and says which answer; with a token that isn’t an admin’s, it says it skipped that check.

Addresses and networks

The installer picks the addresses once. If the machine’s LAN address changes, such as with DHCP, ssh falls back to your deployment until you run the installer again, which picks them again. A DHCP reservation for the machine, or its tailnet address, avoids that. On a VPS, the private addresses are the provider’s network, which your computer can’t reach: give the tailnet address with --direct-addr.

The relay starts on its own, so a missing address doesn’t stop the agent or its sandboxes. If an address isn’t there yet, such as a tailnet that comes up after Docker, the relay fails alone, node info says failed with Docker’s reason, and the agent tries again every 30 seconds. If the port is published on a public address or on all addresses, such as by hand, the agent refuses it and says so in node info, until the installer runs again.

Your computer’s network What happens
The node’s LAN Direct, on the LAN address
A tailnet the node is on Direct, on the tailnet address
A full-tunnel VPN LAN addresses go into the tunnel and fail; through your deployment, or the tailnet address if it works
A split-tunnel VPN Direct, as on the LAN
A Tailscale exit node The LAN is blocked unless Tailscale’s “Allow local network access” is on; the tailnet address still works
Another VPN in 100.64/10, such as Cloudflare WARP Its addresses can look like tailnet ones; the key check refuses any machine that isn’t the node
A captive portal The tries fail. So does your deployment, until you sign in to the network.
macOS Local Network privacy When the terminal app isn’t allowed to reach the local network, LAN tries can fail at once with no route to host, which CLOUDEX_DEBUG points out, and ssh goes through your deployment. Allow the app in System Settings, Privacy & Security, Local Network.

The CLI doesn’t know whether Tailscale reaches the node directly or through one of its relays (DERP). A relayed tailnet path still wins when it answers first, and can be slower for big copies than your deployment.

Move between a node and Cloudflare

A sandbox stays where it was made. To move one, save it, then start a sandbox from the save where you want it:

cloudex save n1 n1-copy                        # on the node
cloudex new --from n1-copy --name n1-cf        # on Cloudflare
cloudex new --from n1-copy --on box --name n2  # or on a node, the same one or another

It works in both directions, and between nodes. A Cloudflare sandbox’s save starts on a node the same way, with --on. The new sandbox has all of the saved disk, because both sides run the same image. It has a new id and new preview links, and the old sandbox keeps running until you stop or delete it.

  • Speed. A save of a small sandbox on the node took about 2 seconds on the test machine. Starting a node’s save on Cloudflare took 6 to 13 seconds, and a Cloudflare save on the node 7 to 9. A large save is limited by the machine’s upload speed.
  • Device files that Docker in a Cloudflare sandbox keeps under its data directory can’t be made on a node, so they’re left out of the restore, and the new sandbox’s notice counts them. Docker’s images in it may need pulling again.
  • Trust. A save made on a node holds whatever that node put in it. Before you start it on Cloudflare, treat it like code from that node. cloudex saves shows the node each save was made on in its ON column, and so do new --from, the dashboard and the SDKs (a save’s node).

When a node is offline

A node is offline when its agent isn’t connected: the machine is off, its network is down, or the agent or Docker stopped. cloudex node list shows it offline at once when the agent closes its connection, and within about 45 seconds when the connection just goes silent.

While it’s offline:

  • cloudex list shows its sandboxes as box (offline), with the state they had. cloudex info still works.
  • Anything that needs the machine, such as ssh, exec, stop or resume, fails with node box is offline (last seen 2026-11-11 14:02 UTC). When only the node’s line to your deployment is down and your computer can still reach the node, ssh to a running sandbox can still go direct, if your CLI has connected to it since it last started.
  • The machines keep running on the host, if it’s up. Idle stop can’t stop them until the node is back.
  • A change to a running sandbox’s settings is refused, and isn’t kept.
  • cloudex delete is refused. cloudex delete --force deletes the sandbox at once, and the node removes its machine and network when it’s back. cloudex node info lists those as pending deletes.

When the agent connects again, which it does by itself, the node is online within a second or two. The agent reports its machines, and each sandbox settles with what really happened:

  • A machine that stopped while the node was offline marks its sandbox stopped, with a notice that its disk is kept. cloudex resume starts it.
  • A sandbox you’d stopped whose machine was started on the host by hand is stopped again.
  • A machine that’s gone marks a running sandbox error. Resuming it starts over from the image, without its files, and its notice says so.
  • A container with the agent’s labels that belongs to no sandbox is left alone and listed as an orphan in cloudex node info.

Reboots and restarts

The firewall loads at boot, before Docker, and Docker starts the agent again. Sandboxes don’t start by themselves: each one shows stopped, with a notice that its machine stopped without being asked to and its disk is kept. Resume them. The same happens when Docker restarts and stops its containers. Restarting only the agent’s container leaves the machines running.

In testing, the node was online within 3 seconds of the machine being back up, and a resume took about 2 seconds, with the files.

Deploys

Nodes reconnect by themselves when you run cloudex deploy. As for sandboxes on Cloudflare, busy connections to a node’s sandboxes through your deployment, such as ssh sessions, move to the new version within about 5 minutes and may drop then. Direct ssh sessions don’t go through your deployment, so a deploy doesn’t touch them.

Remove a node

cloudex node rm box

cloudex node rm disconnects the node’s agent at once, and its key stops working, so it can’t connect again. It asks first; --yes skips the question. The node stays listed as revoked until you run cloudex node rm again, which forgets it.

A node that still has sandboxes is refused unless you add --force. Then:

  • If the node is online, its machines are deleted first. The sandboxes stay listed, in state error, saying their machines went with the node, until you delete them.
  • If the node is offline, its machines can’t be deleted then. The sandboxes stay listed in state error until you delete them. When the agent connects again, the deployment tells it the node was removed, and it stops every machine. Their disks stay on the host until you uninstall the agent.

A sandbox on a removed node answers 410 to everything but delete, and deleting it needs nothing from the node.

node rm doesn’t uninstall anything on the host. The agent stops its machines once it hears it was removed, and then asks again only every 30 minutes; cloudex-node status on the machine says the node was removed. To add the same machine again, uninstall it first, then cloudex node add and run the new command: the installer refuses to run over a removed node.

Isolation

Every sandbox on a node runs in a Docker container on a network of its own, with the firewall around it. How far it’s kept from the host depends on the runtime:

Isolation When Root in a sandbox is Docker inside Use it for
sysbox Docker has Sysbox when a sandbox is made An unprivileged user on the host Works Code you don’t fully trust
docker (plain Docker) No Sysbox Root on the host’s kernel, held in by the container Doesn’t work Code you trust only
none The node joined with --unisolated As plain Docker, with no firewall Doesn’t work Testing only
vm A sandbox on Cloudflare Root in a VM of its own Works (host network) Anything

cloudex node list, cloudex info, the dashboard and the API show a node’s and a sandbox’s isolation. A plain-Docker node shows docker (trusted code only).

What the firewall does

The firewall applies under both runtimes. From a sandbox:

  • The internet and DNS answer.
  • The host itself, your LAN (private address ranges), your tailnet (100.64.0.0/10), link-local and cloud metadata addresses such as 169.254.169.254, multicast and reserved ranges, and other sandboxes don’t. Neither do the host’s public on-link networks as they were at install, or anything you add with --block.
  • Nothing can open a connection into a sandbox, and sandbox networks have no IPv6.
  • A packet from a sandbox whose source address isn’t on its own network is dropped, so a sandbox that takes another’s address can’t have answers sent there.

A sandbox also reaches its node’s agent, for cloudex.internal and its provider calls, and the agent answers it only from its own network.

The self-test checks this from inside throwaway sandboxes, rather than reading the rules: when the agent starts, every hour after, and after an error. It probes the host, its gateway, its DNS servers and tailnet addresses, the direct port, metadata, another sandbox, and what root can do inside, and it checks that the internet and DNS answer. If anything that should be closed answers, the node’s isolation shows as failing, it makes and starts no sandboxes until a self-test passes, and it stops the sandboxes that are running, each with a notice that says why. A pass doesn’t count for ever: if no self-test has finished for about two hours, the node starts no sandboxes until one passes.

The firewall is one nftables table. systemctl restart nftables, nft flush ruleset, or a firewall manager that rewrites the whole rule set removes it, and with it Docker’s own rules:

  • nftables.service starting, restarting or reloading loads the table again right after, since its unit is tied to that service.
  • Anything else is caught within 30 seconds by cloudex-node-firewall-check.timer, which loads it again.
  • Meanwhile, the agent knocks on the host from a sandbox network every 5 seconds, a knock the table rejects. When the host answers instead, the agent marks the node failing, stops its running sandboxes, each with a notice that says why, and runs its self-test again. In testing, after nft flush ruleset a running sandbox could reach the host for 1 to 4 seconds before it was stopped, and the table was back 9 to 34 seconds later; after systemctl restart nftables the table was back within a quarter of a second, and the sandbox never reached the host.

A flush also removes Docker’s rules, so sandboxes lose the internet until Docker’s rules are back (systemctl restart docker restores them). Once a self-test passes again, resume the sandboxes. ufw, firewalld and iptables-legacy weren’t tested.

The direct port

The direct port is the node’s only inbound port. Docker publishes it on the addresses the installer chose, all private or tailnet IPv4, and the agent refuses to serve it when it finds it published on a public address or on all addresses. Behind it are the relay, which holds nothing, and the agent, which takes connections only from the relay.

Anything on your LAN or tailnet can connect to it and complete a TLS handshake, and that’s all. The agent reads one line after the handshake, at most 1 KiB within 5 seconds, and connects nothing without a valid ticket: one your deployment signed with this node’s own secret, for a sandbox running on it, unused and less than 60 seconds old. Wrong, reused, expired and missing tickets get an error and the connection closes; sshd never sees them. The agent holds at most 64 handshakes at once and 16 from one address, so a flood from one machine can delay others’ connections by a few seconds but not take the node down; in testing, 2000 connections at once left the agent at 10 MiB, and a real ssh right after went through.

Sandboxes on the node can’t reach the port: the firewall keeps them off the host’s addresses, and the self-test checks that every hour. Under --unisolated they can, and get the same TLS handshake and nothing more.

On a machine with Tailscale, Tailscale’s own NAT makes tailnet clients reach the agent from Docker’s gateway address rather than their own. That needs no setup. They share one budget, 64 of the agent’s 128 handshakes at once, kept for them; other addresses share the other 64, at most 16 each. To narrow who can reach the port, use Tailscale’s access rules, --direct-addr to publish it on fewer addresses, or --direct off.

Plain Docker

Under plain Docker, the container is all that keeps root in a sandbox off the host. Sandboxes drop some capabilities and never run privileged, with host paths, the Docker socket or the host’s network. But a kernel bug or a container escape gives code in the sandbox root on the host, and from there your LAN and tailnet are in reach, whatever the firewall says.

Use a plain-Docker node only for code you trust. An agent that reads web pages, issues or pull requests can be told what to run by them. That includes code from pull requests, unknown repos, and agents that fetch and run things from the internet. Use Sysbox, or Cloudflare, for those.

Sysbox

Sysbox maps root in a sandbox to an unprivileged user on the host, and lets Docker run inside. Install it before the agent, or run the installer again after installing it; sandboxes made from then on use it, and older ones keep the runtime they were made with. Sysbox’s installer rewrites Docker’s daemon.json and restarts Docker, and version 0.7.1 also needed the fuse3 package, which it doesn’t declare.

Sysbox still shares the host’s kernel. Root in a sandbox can change its own routes and send raw packets on its own network, where only the agent and the host’s bridge address are. The firewall and the agent’s checks hold against that (a forged source address, for one, is dropped), but a kernel bug can still reach the host.

–unisolated

--unisolated joins a node without the firewall, for testing on a machine where you have no root. Sandboxes on it reach the host, its LAN, its tailnet and its direct port. It shows as none (testing).

Who may use a node

Who Can
The admin key, admin tokens Add and remove nodes, and place sandboxes on any node
user tokens Place sandboxes on any node, and see the nodes
Own-only user tokens The same, except on plain-Docker and unisolated nodes, which they’re refused: CI runs code from pull requests. They may use Sysbox nodes.
read tokens See the nodes

What a node can see

A node runs your sandboxes, so it can see and change everything in them: their disks, processes and network traffic. It also gets their settings, which the deployment decrypts and sends at every start, as it does to a Cloudflare container.

Provider calls. On a node, a sandbox’s calls to an AI provider go to the node’s agent, which passes them to your deployment, which adds the provider key and sends them on. The sandbox’s HTTPS connection ends at the agent, with a certificate from the node’s own certificate authority, which can only sign the providers’ host names. So the agent sees the agents’ prompts and the providers’ answers in plain text. It never sees the provider keys: requests leave it with the placeholder key, and the real key is added inside your Worker.

Saves. A node writes the named saves its sandboxes make, and reads the saves they start from, while that runs. A save made on a node holds whatever that node put in it.

A node can’t:

  • read provider keys, the admin key, tokens or the deployment’s database;
  • reach another node’s sandboxes or a Cloudflare sandbox;
  • read or write other saves, or backups;
  • use your deployment as a proxy to anywhere but the providers’ APIs and cloudex.internal;
  • open direct connections to another node’s sandboxes: each node’s tickets are signed with a secret of its own.

Someone who takes over a node can still spend your provider keys for requests made in its sandboxes’ names, for as long as it has a sandbox running. Set a spending limit with each provider if that matters, and remove a node you no longer trust with cloudex node rm.

Performance

Every command to a sandbox on a node goes through your deployment, as for a sandbox on Cloudflare, plus a round trip to the node and Docker’s own time for each command. So does ssh, unless it goes straight to the node.

Through your deployment

These were measured with the CLI on a Mac in Florida on Wi-Fi, and a node on an AWS VM in us-east-1, which reached Cloudflare in Ashburn (IAD). The Cloudflare figures are from a standard-2 sandbox in the same hour.

Node Cloudflare
cloudex exec -- true, start to end 221 ms median, 241 ms p90 163 ms, 178 ms
A command’s round trip on a kept-alive connection 99–141 ms median 73 ms
ssh connection set-up 343 ms median 315 ms
Keystroke round trip in ssh 63–68 ms median 9 ms
scp, 256 MiB, up / down 23 / 28 MiB/s 59 / 57 MiB/s
cloudex cp, 90 MiB, up / down 13–18 / 62–66 MiB/s 22 / 19 MiB/s
Create 2.3 s
Stop 0.45 s
Resume 1.1–2.0 s

A node that’s far from you or from Cloudflare adds that distance to every round trip, unless ssh goes direct. cloudex node info shows where the node connects to Cloudflare and a round trip to its agent, and the deployment makes each of the node’s sandboxes a Durable Object in the node’s region, where Cloudflare has one. For long sessions, work inside tmux, with cloudex attach or the dashboard’s terminal, so a dropped connection doesn’t end what’s running.

Direct

A direct connection is as fast as plain ssh to the machine: the TLS around it cost nothing measurable. These were measured with the CLI on the same Mac in Florida. The tailnet figures are v0.2.3’s, against a VPS in New Jersey over Tailscale, with both peers connected directly, in the same runs as the figures through your deployment. The LAN figures are from the prototype of the same design, against a home server over Wi-Fi.

LAN Tailnet, far VPS Through your deployment, to the same VPS
Keystroke round trip in ssh, median 6.9 ms (plain ssh: 6.4) 30 ms (plain ssh: 30) 41 ms
scp, up / down 46 / 41 MiB/s, 1 GiB (plain scp: 45 / 42) 39–43 / 35–41 MB/s, 256 MiB 16–21 / 25–34 MB/s
cloudex ssh -- true, start to end, median 550 ms 736 ms

On a test VM on the same Mac, which is as close as a node gets, a keystroke took 1 ms and scp ran at 52–60 MB/s, against 18–44 through your deployment.

When a direct try fails, falling back costs up to about 300 ms more than going through your deployment at once; in testing, between nothing and 330 ms, for a stale address, a closed port, another machine at the address, and a node that accepts and then stalls. Later connections on that network start through your deployment at once, for 10 minutes.

Troubleshooting

From your computer:

  • cloudex node info box shows the node’s state and when it was last seen, its agent’s version, its host (CPUs, memory, free disk), its runtime and isolation, when the self-test last ran and which checks failed, the sandbox image and whether it’s present or still pulling, where it connects to Cloudflare, a round trip to it, its direct port’s addresses and state (for admins), its fork images, its clock if it’s off, pending deletes and orphans.
  • cloudex doctor warns about nodes that never joined, are offline, run an agent of another version, fail their self-test, have under 15 GB of Docker disk free, or whose clock is more than 30 seconds off. It also tries each online node’s direct port from your computer, and says whether it answers.

On the machine:

sudo docker exec cloudex-node cloudex-node status          # connected? exits 1 when not
sudo docker logs --tail 50 cloudex-node                    # the agent's log
sudo docker exec cloudex-node cloudex-node selftest        # the self-test now, as JSON; exits 1 when it fails
sudo docker exec cloudex-node cloudex-node selftest --machine   # boot and check a throwaway sandbox
sudo docker ps -a --filter name=cloudex-node-direct       # the direct port's relay
sudo systemctl status cloudex-node-firewall                # the firewall's unit
sudo nft list table inet cloudex                           # the firewall's rules

status says whether the agent is connected, since when, and if it isn’t, the last error. status --json adds the node’s id, its deployment and its last self-test.

Message What it means
that join token is unknown, used or expired A token works once, for 30 minutes. Run cloudex node add box again for a new one.
this node has joined already The machine has an agent’s identity. To join again, uninstall first.
this deployment speaks node protocol … install the agent of the deployment's version Run the installer with the --version it names
node box's isolation self-test fails (…) The firewall isn’t loaded, or something else answers from inside a sandbox. Check systemctl status cloudex-node-firewall, then the checks it names.
machines on this node may run only ghcr.io/lucataco/cloudex-base The deployment’s image is from another repository. Run the installer with --allow-image.
every machine pool the installer knows collides with a route or a Docker network Give a free private /16 with --pool
Docker … is too old, nodes need nftables, run it as root A requirement in What you need is missing
the node's clock is … s ahead (or behind), in the agent’s log More than 60 seconds is refused. Set the machine’s time with NTP.
this node was removed from the deployment, in status and the agent’s log cloudex node rm removed it; the agent stopped its sandboxes. Uninstall, then add it again.
Direct … (failed): published on a public address in node info The direct port is published on a public address or all addresses, such as by hand. Run the installer again.
Direct … (failed) with Docker’s reason, such as cannot assign requested address An address the installer chose isn’t on the machine now. The agent tries again every 30 seconds; if the address changed, run the installer again.
Direct … (off): waiting for the deployment's secret in node info The agent has just started and hasn’t heard from your deployment yet. It clears within seconds of the node coming online.
Direct off: turned off on this node (installed with --direct off) As it says. Run the installer again with --direct auto or --direct-addr to open the port.
A sandbox’s notice: Node box stopped its machine: … The agent stopped it itself: Docker’s disk fell under the floor, the firewall’s table disappeared, or the self-test found the firewall open. The notice says which.

Uninstall

curl -fsSL https://cloudex.build/node | sudo bash -s -- --uninstall

On the machine, --uninstall removes everything the installer made: the agent, the direct port’s relay, every sandbox’s container and network with their disks, fork images, the agent’s volume, the firewall and its unit, /etc/cloudex-node, and the images it pulled, unless they were there before. It asks first, saying how many sandboxes it removes; --yes skips the question. --keep-images keeps the images it pulled, but not fork images: they’re sandboxes’ disks. If a fork’s copy is under way, it waits for it to finish first, so that its image isn’t left behind.

Remove the node from your deployment too, with cloudex node rm box. Doing that first, while the node is online, deletes its sandboxes’ machines through the agent (--force if it still has sandboxes), and the sandboxes stay listed in error until you delete them. If you uninstall first, the node shows offline until you remove it.

For agents and LLMs: this page as Markdown,llms.txt and all docs in one file.