bug: VM driver fails to create rootfs for images with files >2 GiB due to mke2fs -d bug

[zakkak]

Agent Diagnostic

Investigated the ProvisioningFailed error when creating sandboxes from large local images.

• Root cause traced to crates/openshell-driver-vm/src/rootfs.rs:format_ext4_image_from_dir which calls mke2fs -d to populate an ext4 image from a staging directory.
• The staging directory contains source-rootfs.tar (~4 GiB) copied in by stage_guest_image_payload at crates/openshell-driver-vm/src/driver.rs:4094-4104.
• mke2fs -d in e2fsprogs 1.47.3 has a signed 32-bit overflow bug: any file with >2 GiB of non-zero data fails with __populate_fs: Ext2 file too big. Zero-filled files pass because mke2fs skips zero blocks as sparse.
• Confirmed the 2 GiB boundary empirically (2048 MiB succeeds, 2049 MiB fails). Block size, filesystem features, and image size have no effect.
• Additionally, e2fs_tool_candidates in rootfs.rs only searches macOS Homebrew paths (/opt/homebrew/opt/e2fsprogs, /usr/local/opt/e2fsprogs) after the bare tool name. On Linux, when the driver process runs with a restricted $PATH (e.g. missing /usr/sbin), the bare name lookup fails and no Linux-standard paths are tried, resulting in a misleading "not found" error.
• Upstream bug report draft is at /tmp/mke2fs-bug-report.md for filing against tytso/e2fsprogs.

Description

Actual behavior: openshell sandbox create --from localhost/<image>:latest fails when using the VM compute driver with microVMs:

ProvisioningFailed: failed to create ext4 rootfs image from .../.openshell-overlay-staging-...:
__populate_fs: Ext2 file too big while writing file "source-rootfs.tar"
mke2fs: Ext2 file too big while populating file system

On some Linux systems the initial error is instead:

/usr/local/opt/e2fsprogs/bin/mkfs.ext4 not found. Install e2fsprogs (mke2fs/mkfs.ext4) and retry

This occurs because the tool search only checks macOS Homebrew paths and does not include standard Linux paths like /usr/sbin, /sbin, or /usr/bin.

Expected behavior: Sandbox creation succeeds for any valid container image regardless of rootfs tar size, and mke2fs/mkfs.ext4 is found on standard Linux installations.

This only affects the VM compute driver (compute_drivers = ["vm"]), which uses mke2fs -d to build ext4 rootfs images. The Podman and Docker drivers are not affected.

Reproduction Steps

1. Configure the gateway to use the VM compute driver (see gateway.toml below).
2. Build or pull a local container image whose exported rootfs tar exceeds 2 GiB of non-zero content.
3. Run:

   openshell sandbox create \
     --name test \
     --cpu 2 \
     --memory 2Gi \
     --from localhost/<image>:latest

4. Observe the ProvisioningFailed error.

Environment

• OS: Fedora 44 (Linux 7.0.10-201.fc44.x86_64)
• e2fsprogs: 1.47.3
• OpenShell: 0.0.55 / 0.0.56-dev

gateway.toml (using microVMs via the VM compute driver):

[openshell]
version = 1

[openshell.gateway]
bind_address = "0.0.0.0:17670"
compute_drivers = ["vm"]

[openshell.drivers.vm]
vcpus = 8
mem_mib = 16384

Logs

Adding the output after applying the following patch


diff --git a/crates/openshell-driver-vm/src/rootfs.rs b/crates/openshell-driver-vm/src/rootfs.rs
index 904ed8cd..cb3d222d 100644
--- a/crates/openshell-driver-vm/src/rootfs.rs
+++ b/crates/openshell-driver-vm/src/rootfs.rs
@@ -458,7 +458,7 @@ fn round_up_to_mib(bytes: u64) -> u64 {
 }
 
 fn format_ext4_image_from_dir(source: &Path, image_path: &Path) -> Result<(), String> {
-    let mut last_error = None;
+    let mut errors = Vec::new();
     for tool in ["mke2fs", "mkfs.ext4"] {
         for candidate in e2fs_tool_candidates(tool) {
             let label = candidate.display().to_string();
@@ -467,6 +467,8 @@ fn format_ext4_image_from_dir(source: &Path, image_path: &Path) -> Result<(), St
                 .arg("-F")
                 .arg("-t")
                 .arg("ext4")
+                .arg("-b")
+                .arg("4096")
                 .arg("-E")
                 .arg("root_owner=0:0")
                 .arg("-d")
@@ -476,7 +478,7 @@ fn format_ext4_image_from_dir(source: &Path, image_path: &Path) -> Result<(), St
             match output {
                 Ok(output) if output.status.success() => return Ok(()),
                 Ok(output) => {
-                    last_error = Some(format!(
+                    errors.push(format!(
                         "{label} failed with status {}\nstdout: {}\nstderr: {}",
                         output.status,
                         String::from_utf8_lossy(&output.stdout),
@@ -484,18 +486,18 @@ fn format_ext4_image_from_dir(source: &Path, image_path: &Path) -> Result<(), St
                     ));
                 }
                 Err(err) if err.kind() == std::io::ErrorKind::NotFound => {
-                    last_error = Some(format!("{label} not found"));
+                    errors.push(format!("{label} not found"));
                 }
                 Err(err) => {
-                    last_error = Some(format!("run {label}: {err}"));
+                    errors.push(format!("run {label}: {err} (kind: {:?})", err.kind()));
                 }
             }
         }
     }
     Err(format!(
-        "failed to create ext4 rootfs image from {}: {}. Install e2fsprogs (mke2fs/mkfs.ext4) and retry",
+        "failed to create ext4 rootfs image from {}: [{}]. Install e2fsprogs (mke2fs/mkfs.ext4) and retry",
         source.display(),
-        last_error.unwrap_or_else(|| "no ext4 formatter found".to_string())
+        errors.join("; ")
     ))
 }
 
@@ -736,6 +738,9 @@ fn run_debugfs(image_path: &Path, command: &str) -> Result<(), String> {
 
 fn e2fs_tool_candidates(tool: &str) -> Vec<PathBuf> {
     let mut candidates = vec![PathBuf::from(tool)];
+    for dir in ["/usr/sbin", "/sbin", "/usr/bin"] {
+        candidates.push(Path::new(dir).join(tool));
+    }
     for root in ["/opt/homebrew/opt/e2fsprogs", "/usr/local/opt/e2fsprogs"] {
         candidates.push(Path::new(root).join("sbin").join(tool));
         candidates.push(Path::new(root).join("bin").join(tool));


Output


✗ Error: ProvisioningFailed: failed to create ext4 rootfs image from /home/zakkak/.local/state/openshell/vm-driver/images/sandbox-prepared-rootfs-ext4-umoci-v3-openshell-0.0.56-dev.8-gb41e0df4-sha256-7f5f917046be984da8aa6821cff29b225138033bc0c0387730feca99c5e97b16.staging-1780583285334-0/.openshell-overlay-staging-357730-1780583291284: [mke2fs failed with status exit status: 1
stdout: 
stderr: __populate_fs: Ext2 file too big while writing file "source-rootfs.tar"
mke2fs: Ext2 file too big while populating file system
; /usr/sbin/mke2fs failed with status exit status: 1
stdout: 
stderr: __populate_fs: Ext2 file too big while writing file "source-rootfs.tar"
mke2fs: Ext2 file too big while populating file system
; /sbin/mke2fs failed with status exit status: 1
stdout: 
stderr: __populate_fs: Ext2 file too big while writing file "source-rootfs.tar"
mke2fs: Ext2 file too big while populating file system
; /usr/bin/mke2fs failed with status exit status: 1
stdout: 
stderr: __populate_fs: Ext2 file too big while writing file "source-rootfs.tar"
mke2fs: Ext2 file too big while populating file system
; /opt/homebrew/opt/e2fsprogs/sbin/mke2fs not found; /opt/homebrew/opt/e2fsprogs/bin/mke2fs not found; /usr/local/opt/e2fsprogs/sbin/mke2fs not found; /usr/local/opt/e2fsprogs/bin/mke2fs not found; mkfs.ext4 failed with status exit status: 1
stdout: 
stderr: __populate_fs: Ext2 file too big while writing file "source-rootfs.tar"
mkfs.ext4: Ext2 file too big while populating file system
; /usr/sbin/mkfs.ext4 failed with status exit status: 1
stdout: 
stderr: __populate_fs: Ext2 file too big while writing file "source-rootfs.tar"
mkfs.ext4: Ext2 file too big while populating file system
; /sbin/mkfs.ext4 failed with status exit status: 1
stdout: 
stderr: __populate_fs: Ext2 file too big while writing file "source-rootfs.tar"
mkfs.ext4: Ext2 file too big while populating file system
; /usr/bin/mkfs.ext4 failed with status exit status: 1
stdout: 
stderr: __populate_fs: Ext2 file too big while writing file "source-rootfs.tar"
mkfs.ext4: Ext2 file too big while populating file system
; /opt/homebrew/opt/e2fsprogs/sbin/mkfs.ext4 not found; /opt/homebrew/opt/e2fsprogs/bin/mkfs.ext4 not found; /usr/local/opt/e2fsprogs/sbin/mkfs.ext4 not found; /usr/local/opt/e2fsprogs/bin/mkfs.ext4 not found]. Install e2fsprogs (mke2fs/mkfs.ext4) and retry

Agent-First Checklist

• I pointed my agent at the repo and had it investigate this issue
• I loaded relevant skills (e.g., debug-openshell-cluster, debug-inference, openshell-cli)
• My agent could not resolve this — the diagnostic above explains why