Part 3: Proving It Works
Parts 1 and 2 covered why this matters and how the runtime got healthy. But a healthy runtime isn’t the same thing as a proven one. The only claim worth making to a test team is “your actual test suite will pass here,” and that took three separate checks, each one harder to fake than the last.
Check 1: Does the sandbox even work
Simplest possible test: a pod pinned to runtimeClassName: sysbox-runc, scheduled onto a labeled node.
apiVersion: v1 kind: Pod metadata: name: sysbox-test spec: runtimeClassName: sysbox-runc nodeSelector: sysbox-runtime: running containers: - name: test image: ubuntu:24.04 command: ["/bin/bash", "-c", "cat /proc/self/uid_map; sleep 3600"]
A real user-namespace mapping, something like 0 100000 65536, confirms the pod is running in its own isolated UID range and not sharing the host’s root. If this doesn’t show up correctly, nothing past this point matters.
Check 2: Does nested Docker actually pull and run something
This one runs /sbin/init as the entrypoint, which is required so the nested, systemd-managed Docker daemon actually comes up inside the sandbox, with hostUsers: false and a Docker client config mounted in for the internal registry.
kubectl create secret generic docker-config-ar \ --from-file=config.json=./docker-config.json \ -n default
One thing bit us here that’s worth calling out on its own. A placeholder credentials file (anything with a literal “{AUTH}” string still sitting in it) mounts into the pod just fine. No error. Looks correct. Then every single nested pull fails, and the failure message doesn’t point back at the credential at all. The secret needs a real base64-encoded oauth2accesstoken entry, generated fresh, because Artifact Registry tokens are short-lived. That has to be part of whatever rotates credentials automatically, not something set once and forgotten.
Second gotcha, smaller but just as annoying: pulling a public Docker Hub image through a virtual repository requires the /library/ segment for official images. docker-remote/hello-world fails. docker-remote/library/hello-world works. One path segment, and it cost real debugging time before anyone remembered the convention.
Once both were right, a pull-and-run against the internal registry succeeded and printed the expected Docker Hub greeting. That’s proof the nested daemon itself works, not just the sandbox around it.
Check 3: Does the library actually work, not just the daemon
This is the check that matters most and gets skipped most often. Nested Docker working doesn’t mean Testcontainers works, because Testcontainers talks to Docker through its own bundled client, and that client can disagree with the daemon about API versions even when the daemon is completely fine. That’s more or less exactly what happened: an older Testcontainers dependency threw client version 1.32 is too old. Minimum supported API version is 1.40, while the nested daemon was sitting there healthy on API 1.55. Nothing wrong with Sysbox, or with Docker. The client shipped inside the old Testcontainers jar just hadn’t caught up. The fix was pinning the test application to a current Testcontainers release, 2.0.5, with a client that actually negotiates a modern API.
The final proof was a small Java app, built into a fat jar with Maven, that pulls a base image through the internal repo, starts a container, runs a command inside it, and prints a marker on success:
docker run --rm \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /root/.docker:/root/.docker:ro \ -e DOCKER_CONFIG=/root/.docker \ -e DOCKER_HOST=unix:///var/run/docker.sock \ -e TESTCONTAINERS_RYUK_DISABLED=true \ internal-docker-repo/testcontainers-java-smoke:latest
It connected, reported Server Version: 29.7.1 and API Version: 1.55, ran the command, and printed its own success marker with a clean exit code. That’s the actual proof: not “the daemon runs,” but “a real Testcontainers workload runs.”
What’s left
Functionally, this is done. Unprivileged pod, isolated nested daemon, authenticated pulls through internal repos only, a real Testcontainers run passing. What’s left is cleanup: stripping temporary debug logging out of the patched binaries and re-running all three checks against the cleaned build before it ships into shared CI images.
Questions people actually ask
Does this need privileged: true? No, and that’s really the whole point of using Sysbox instead of a DinD sidecar. Isolation comes from a real Linux user namespace, not from trusting the container not to misbehave.
People also ask whether this works behind a fully air-gapped registry with no exceptions at all. It does, as long as the virtual repository actually mirrors what the test suite needs and the mounted credentials are current. The token failure above is the most common way this breaks in practice.
A fair question: if nested Docker was working, why did Testcontainers still fail? Because the failure lived in the library’s bundled client, not the runtime. Worth checking those two things separately rather than assuming a working daemon means a working test suite.
One last question: is any of this GKE-specific? Some of it, yes. The Artifact Registry details and the node-labeling steps are GKE conventions. The runtime fixes themselves sit lower than that. They’re about proc-filesystem handling and namespace passthrough, and they’d apply to any locked-down, no-egress Kubernetes environment regardless of which managed provider is running it.
 | Anuj Tuli, Chief Technology Officer Anuj specializes in developing and delivering vendor-agnostic solutions that avoid the “rip-and-replace” of existing IT investments. He has worked on Cloud Automation, DevOps, Cloud Readiness Assessments, and Migration projects for healthcare, banking, ISP, telecommunications, government and other sectors. He leads the development and management of Cloud Automation IP (intellectual property) and related professional services. During his career, he held multiple roles in the Cloud and Automation, and DevOps domains. With certifications in AWS, VMware, HPE, BMC and ITIL, Anuj offers a hands-on perspective on these technologies. Like what you read? Follow Anuj on LinkedIn at https://www.linkedin.com/in/anujtuli/ |