feat(manifests): add 8 core-derived ACP agent manifests

[Yiminnn]

What

The 8 core-derived ACP agent manifests that benchflow's manifest loader (load_agent_manifest, merged in #14) consumes — the agents-side half of the decouple "running-through" path. Each manifest.toml is the data-only projection of a core AGENTS entry; benchflow merges it with that entry's host-side shim-only fields (subscription_auth, credential_files, …) so a manifest-loaded agent == the original core agent.

Agents: claude-agent-acp, codex-acp, gemini, mimo, openclaw, opencode, openhands, pi-acp.

How it fits

• Consumed by benchflow when BENCHFLOW_AGENTS_DIR points at this tree — see feat(agents): all-paths decouple core — manifest loader (8 ACP) + omnigent session-factory seam (additive, gated) benchflow#825 (the core consumer loader + additive merge-mode). Additive / gated: no effect on benchflow's built-in AGENTS unless that env is set (strangler-fig).
• codex-acp ships the self-writing launcher (c675b3d): writes ~/.codex/auth.json from OPENAI_API_KEY only when the key is set, leaving ChatGPT-subscription mode (host auth.json) untouched. Pairs with the benchflow-core subscription fix in feat(agents): all-paths decouple core — manifest loader (8 ACP) + omnigent session-factory seam (additive, gated) benchflow#825.

Verified — e2e citation-check, both docker + daytona

All 8 load from these manifests, and the shim-only merge == orig core entry on every dataclass field. End-to-end via benchflow#825:

agent	result
pi-acp / opencode / openhands / mimo	reward 1.0 (deepseek)
gemini	runs via manifest (GEMINI_API_KEY)
codex-acp (ChatGPT subscription)	1.0 on both backends after the #825 subscription fix (gpt-5.5, 7 tool calls)
claude-agent-acp (subscription)	1.0 daytona
openclaw	1.0 daytona

Test plan

• manifest load + shim-only merge == orig for all 8 (benchflow loader tests).
• e2e citation-check on docker + daytona via BENCHFLOW_AGENTS_DIR.

[greptile-apps]

Greptile Summary

Adds eight manifest.toml files — one per ACP agent (claude-agent-acp, codex-acp, gemini, mimo, openclaw, opencode, openhands, pi-acp) — that benchflow's manifest loader consumes when BENCHFLOW_AGENTS_DIR is set, acting as the agents-side data-only projection of the corresponding core AGENTS entries.

• Each manifest declares install_cmd (bootstraps Node.js and installs the agent package), launch_cmd (starts the ACP server), protocol, requires_env, and an [env_mapping] section that translates BENCHFLOW_PROVIDER_* vars into agent-native env vars.
• Several manifests embed non-trivial logic as base64-encoded shell/Python proxy scripts baked into install_cmd, handling provider config-file generation at launch time (mimo, opencode, openclaw, pi-acp).
• opencode/manifest.toml is missing the BENCHFLOW_PROVIDER_API_KEY = "OPENAI_API_KEY" entry in [env_mapping]; the opencode-proxy reads OPENAI_API_KEY to populate the provider JSON, so authentication will silently fail unless the key is already in the environment by another means.

Confidence Score: 4/5

Safe to merge with the caveat that opencode's API key mapping should be verified or fixed before relying on it in key-mapped deployments.

Seven of the eight manifests are structurally sound and consistent with each other. The one concrete concern is opencode's missing BENCHFLOW_PROVIDER_API_KEY = OPENAI_API_KEY env_mapping entry — mimo is structurally identical and has it, and the opencode-proxy reads OPENAI_API_KEY directly; without the mapping the agent config is written without credentials. The e2e tests passed, but likely because the test environment had OPENAI_API_KEY already set in the process environment rather than relying on benchflow's mapping. The remaining findings (unescaped printf in codex-acp and openhands, duplicated bootstrap in pi-acp) are low-risk hardening items.

opencode/manifest.toml — verify or add the missing BENCHFLOW_PROVIDER_API_KEY mapping. codex-acp/manifest.toml and openhands/manifest.toml — the printf-based JSON construction is worth hardening against unusual key values.

Important Files Changed

Filename	Overview
opencode/manifest.toml	Missing BENCHFLOW_PROVIDER_API_KEY → OPENAI_API_KEY env_mapping entry; the opencode-proxy reads OPENAI_API_KEY to populate the provider JSON, so without this mapping the agent will launch without an API key.
codex-acp/manifest.toml	launch_cmd writes auth.json via unescaped printf string interpolation; API keys with JSON special chars would produce malformed output, though standard OpenAI keys are alphanumeric.
openhands/manifest.toml	launch_cmd builds agent_settings.json via shell printf string interpolation; LLM_MODEL or LLM_API_KEY with embedded quotes would produce malformed JSON, but real-world values are unlikely to trigger this.
pi-acp/manifest.toml	Duplicate Node.js bootstrap block in install_cmd inflates manifest size; uses @latest for pi-related packages (unpinned). Python launcher handles env_mapping internally so no env_mapping section needed.
openclaw/manifest.toml	Ships a large (~50 KB) base64-encoded Python ACP shim; uses openclaw@latest (unpinned), but the conditional skip-if-present install mitigates reproducibility concerns.
mimo/manifest.toml	Ships embedded base64 shell/Python proxy script for provider config; has complete env_mapping with both BASE_URL and API_KEY entries; no issues found.
claude-agent-acp/manifest.toml	Clean manifest with pinned npm version, proper requires_env, and complete env_mapping; no issues found.
gemini/manifest.toml	Standard manifest with pinned @google/gemini-cli@0.42.0, requires_env, and env_mapping; no issues found.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant BF as benchflow
    participant ML as manifest loader
    participant MF as manifest.toml
    participant AG as agent process

    BF->>ML: load_agent_manifest(BENCHFLOW_AGENTS_DIR)
    ML->>MF: read contract_version, name, install_cmd, launch_cmd, env_mapping
    ML-->>BF: manifest dataclass

    BF->>BF: additive merge(manifest, core AGENTS entry)
    note over BF: shim-only fields added from core

    BF->>AG: run install_cmd
    BF->>BF: "apply env_mapping BENCHFLOW_PROVIDER_* to agent-native vars"
    BF->>AG: exec launch_cmd with mapped env vars
    AG-->>BF: ACP stdio session

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant BF as benchflow
    participant ML as manifest loader
    participant MF as manifest.toml
    participant AG as agent process

    BF->>ML: load_agent_manifest(BENCHFLOW_AGENTS_DIR)
    ML->>MF: read contract_version, name, install_cmd, launch_cmd, env_mapping
    ML-->>BF: manifest dataclass

    BF->>BF: additive merge(manifest, core AGENTS entry)
    note over BF: shim-only fields added from core

    BF->>AG: run install_cmd
    BF->>BF: "apply env_mapping BENCHFLOW_PROVIDER_* to agent-native vars"
    BF->>AG: exec launch_cmd with mapped env vars
    AG-->>BF: ACP stdio session

Loading

Comments Outside Diff (1)

1. opencode/manifest.toml, line 15-17 (link)

   Missing API key mapping in env_mapping

   The [env_mapping] section only maps BENCHFLOW_PROVIDER_BASE_URL → OPENAI_BASE_URL, but is missing BENCHFLOW_PROVIDER_API_KEY = "OPENAI_API_KEY". The opencode-proxy script (decoded from the base64 blob in install_cmd) reads OPENAI_API_KEY directly to populate opts["apiKey"] in the provider JSON config — if OPENAI_API_KEY is never injected via env_mapping, the config is written without an API key and authentication will silently fail at runtime. The structurally identical mimo/manifest.toml has both mappings and serves as the reference.

_{Reviews (1): Last reviewed commit: "fix(codex-acp): manifest launch_cmd self..." | Re-trigger Greptile}

[greptile-apps]

Unescaped API key in printf JSON construction

launch_cmd uses printf '{"OPENAI_API_KEY": "%s"}' "$OPENAI_API_KEY" to write auth.json. If OPENAI_API_KEY ever contains ", \, or a newline (e.g., a key passed in from a CI secret with trailing whitespace or embedded quotes), the resulting file is malformed JSON and codex will fail to authenticate. The same pattern appears in openhands/manifest.toml with LLM_MODEL and LLM_API_KEY. Consider piping through python3 -c "import json,sys,os; print(json.dumps({...}))" or similar for safe serialization.

[greptile-apps]

Entire Node.js bootstrap block duplicated in install_cmd

The install_cmd contains the full Node.js download-and-install bootstrap (the if [ ! -x "$BF_NODE_DIR/bin/node" ]; then … fi block) twice — once before installing @mariozechner/pi-coding-agent and again verbatim before installing pi-acp. The second occurrence is a no-op at runtime (the node binary is already present), but it roughly doubles the manifest size and makes the command harder to audit. Consider factoring the bootstrap into a shared variable or a helper, or omitting the repeated copy.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!