Part 2: Installing and Patching Sysbox for a Locked-Down Cluster
Part 1 covered why any of this is worth doing. This part is the actual runbook: what it took to get Sysbox running on a GKE cluster with no public internet access from any node or pod, reaching the outside world only through preconfigured virtual repositories in an internal Artifact Registry.
Step 1: Install the base runtime
Standard stuff, at least on paper. Sysbox ships as a privileged DaemonSet that runs once per labeled node. It mounts host paths like /etc, /run, /usr/bin, /usr/local/bin, /var/lib, and the systemd unit directories, then runs an installer script that lays down the Sysbox binaries, the systemd services, and a node label. RBAC is minimal: a ServiceAccount, a ClusterRole scoped to get and patch on nodes, a ClusterRoleBinding tying the two together.
kubectl label node <node-name> sysbox-install=yes kubectl apply -f rbac/sysbox-rbac.yaml kubectl apply -f runtime-class/sysbox-runtimeclass.yaml kubectl apply -f daemonset/sysbox-deploy-k8s.yaml Check it worked with: systemctl status sysbox sysbox-mgr sysbox-fs kubectl get runtimeclass sysbox-runc kubectl get nodes -l sysbox-runtime=running
If all three services are active and the node carries the label, the base install is done. This part is genuinely easy. The rest of it isn’t.
Step 2: Fix the installer for a node with no internet
Here’s where the standard guidance stops applying. The stock installer script assumes it can reach the open internet to grab a few packages during setup. On a node with no outbound route at all, that assumption just doesn’t hold, and it doesn’t fail cleanly either. It hangs, or breaks somewhere unhelpful.
The fix is a small overlay image. Rebase from an internal mirror of the installer image, add the one utility the deploy step actually needs (rsync, pulled through the internal registry rather than a public one), and swap in a patched installer script. The patched version stops assuming it can apt-install anything. Instead, it checks for the specific binaries Sysbox needs at runtime, fusermount3 and iptables, and fails immediately with a real error message if either is missing, instead of hanging while it tries to fetch them. It still runs the normal Sysbox checks around user-namespace procfs mounting and loading kernel modules like shiftfs when needed.
This overlay approach isn’t specific to Sysbox, either. Any DaemonSet installer that assumes internet access needs roughly this same treatment on a cluster that has none.
Step 3: The first real failure
Base install healthy, services up, label present. Then the first actual Sysbox pod failed on sandbox creation. The error traced back to the runtime’s rootfs handling, specifically mounting binfmt_misc onto proc/sys/fs/binfmt_misc. It turned out the sandbox’s root filesystem simply didn’t have that proc path yet, and the default mkdir followed by mount logic had no fallback for it.
Step 4: Patch the runtime
This ended up touching two Sysbox components.
In sysbox-runc, a dedicated procMount operation was added so binfmt_misc stops being handled like a normal rootfs mkdir followed by mount. Instead, the init helper remounts /proc, works from the namespace-aware /proc/1/root view, creates the destination there, and mounts it safely by file descriptor. Sysctl writes were also relaxed to tolerate a missing key, since some proc sysctl entries genuinely don’t exist in this environment, and failing hard on that was pure noise.
sysbox-fs needed a broader fix. A whole class of /proc/sys/* passthrough operations (lookup, open, read, write, readdir, readlink, setattr) were failing, and the real cause was that they weren’t carrying enough process context into the namespace layer to correctly reproduce the caller’s root, working directory, uid and gid, and capabilities. Once that context was attached, and the user namespace was deliberately stripped for generic proc-sys passthrough specifically, the permission and personality mismatches went away. A read-only filesystem edge case in temporary procfs mounting also needed a fallback instead of just failing outright.
Both got rebuilt from source with multi-stage, multi-platform Dockerfiles, producing Linux amd64 binaries that replaced the host copies. Restart the services and it’s done:
systemctl restart sysbox sysbox-mgr sysbox-fs systemctl status sysbox sysbox-mgr sysbox-fs
With that in place, a basic Sysbox pod finally reached Running, and cat /proc/self/uid_map inside it showed a real user-namespace mapping. Small thing to check, but it’s the tell that isolation is actually working and not just assumed.
 | Anuj Tuli, Chief Technology Officer Anuj specializes in developing and delivering vendor-agnostic solutions that avoid the “rip-and-replace” of existing IT investments. He has worked on Cloud Automation, DevOps, Cloud Readiness Assessments, and Migration projects for healthcare, banking, ISP, telecommunications, government and other sectors. He leads the development and management of Cloud Automation IP (intellectual property) and related professional services. During his career, he held multiple roles in the Cloud and Automation, and DevOps domains. With certifications in AWS, VMware, HPE, BMC and ITIL, Anuj offers a hands-on perspective on these technologies. Like what you read? Follow Anuj on LinkedIn at https://www.linkedin.com/in/anujtuli/ |