[DO NOT MERGE] Run-only: gb200 dsr1 measured power+temp (canonical NVIDIA) by arygupt · Pull Request #1791 · SemiAnalysisAI/InferenceX

arygupt · 2026-06-15T21:43:16Z

⚠️ DO NOT MERGE — run-only validation (canonical NVIDIA via gb200)

Purpose: get canonical NVIDIA measured dsr1 power+temp on the healthy gb200 runner, sidestepping the wedged gb300-nv fleet.

Carries #1574's self-contained consumer code (aggregate_power.py/process_result.py + tests) on top of main, plus the gb200 launcher perfmon wiring and a single 1-job changelog entry (dsr1-fp4-gb200-dynamo-sglang-powercheck). Matrix = exactly one gb200 job (1k/1k, conc 8, 1×prefill TP4 + 2×decode TP4) — verified with generate_sweep_configs.

Same proven pipeline as the gb300 CoreWeave validation (run 27170607165): launch_gb200-nv.sh clones the perfmon fork (gb200 dsr1 recipes + perfmon machinery), injects monitoring: + a 90-min health_check, and stages perf_samples_*.csv + GPU_METRICS_CSV_GLOB → Process result → aggregate_power.py patches the agg JSON with avg_power_w + prefill/decode + per-worker temp/util/mem.

Why not merge: duplicates #1574's aggregate_power.py. Closed once the data lands.

Known risk to watch on first run: the Oracle/watchtower gb200 cluster may not cross-mount compute→head, so per-node perf_samples (written on compute) may need the shared-FS treatment the minimax path already uses (flagged inline in the launcher).

🤖 Generated with Claude Code

Note

Medium Risk
Touches published agg JSON telemetry and multinode CI/runner cleanup; mistakes could skew power metrics or runner health, though aggregation is best-effort and the PR is run-only.

Overview
Run-only validation for measured DeepSeek-R1 disaggregated power/temperature on NVIDIA: three new changelog keys (dsr1-fp4-gb200|b200|gb300-dynamo-sglang-powercheck) each expand to one 1k/1k job mirroring existing disagg topologies.

Multinode launchers (launch_gb200-nv.sh, launch_gb300-nv.sh, launch_b200-dgxc.sh) gain a dsr1 path that clones SemiAnalysisAI/srt-slurm@feat/inferencex-perfmon, enables recipe monitoring: (and longer health_check where needed), copies perf_samples_*.csv into the workspace, and sets GPU_METRICS_CSV_GLOB before log cleanup.

Result processing extends aggregate_power.py to accept multiple perfmon CSVs, parse role/worker filenames, add temp/util/mem and workers[], support disagg per-stage joules/power, and resolve bench windows from srt-slurm date when Unix timestamps are missing. process_result.py prefers the glob env var (no stale single-CSV fallback) and passes disagg into aggregation.

CI hardening in benchmark-multinode-tmpl.yml: timeout-wrapped Slurm cancel/drain with a 120s cap and sudo rm -rf benchmark_logs pre/post to avoid hung squeue and root-owned leftovers poisoning runners.

Large test additions lock multinode glob wiring, disagg agg fields, and vendor CSV shapes. Documented as do not merge (duplicates #1574 consumer code).

^{Reviewed by Cursor Bugbot for commit 9bf533c. Bugbot is set up for automated code reviews on this repo. Configure here.}

github-actions · 2026-06-15T21:43:25Z

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.

If additional help is needed, PR authors can reach out to core maintainers over Slack.

cursor · 2026-06-15T21:45:10Z

    total_system_energy_j = avg_power_w * num_gpus * duration
-    joules_per_output_token = total_system_energy_j / total_output
    total_tokens = total_output + total_input
+    joules_per_output_token = total_system_energy_j / total_output  # cluster fallback


Desync samples inflate joules

Medium Severity

Multinode energy uses avg_power_w * num_gpus * duration while num_gpus counts every GPU seen across all timestamp buckets, but per-bucket averages can come from only a subset of nodes when perfmon polls don’t align. That pairing can overstate joules_per_* and disagg stage energy even when per-GPU wattage looks correct.

Additional Locations (1)

utils/aggregate_power.py#L374-L398

^{Reviewed by Cursor Bugbot for commit c0b79f4. Configure here.}

github-actions · 2026-06-15T21:57:18Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27578362429
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27578362429

github-actions · 2026-06-15T22:09:20Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27578777750
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27578777750



+# Back-compat shim — some external callers may have imported _parse_power.
+_parse_power = _parse_numeric_cell


cursor · 2026-06-16T21:24:38Z

+    git clone https://github.com/SemiAnalysisAI/srt-slurm.git "$SRT_REPO_DIR"
+    cd "$SRT_REPO_DIR"
+    git checkout feat/inferencex-perfmon
+    export PERFMON_ENABLED=1


Gb200 dsr1 lacks fp4 guard

Medium Severity

The new perfmon fork branch matches all dynamo-sglang/dsr1 runs, not only the fp4 powercheck. launch_b200-dgxc.sh gates the same campaign on PRECISION == "fp4", but gb200 also switches fp8 configs (e.g. dsr1-fp8-gb200-dynamo-sglang) to SemiAnalysisAI/srt-slurm@feat/inferencex-perfmon and sets PERFMON_ENABLED=1, replacing the validated cquil11/srt-slurm-nv path for unrelated jobs on this commit.

^{Reviewed by Cursor Bugbot for commit 742665d. Configure here.}

cursor · 2026-06-16T21:24:38Z

+    if ! grep -q '^health_check:' "$CONFIG_PATH"; then
+        printf '\nhealth_check:\n  max_attempts: 540\n  interval_seconds: 10\n' >> "$CONFIG_PATH"
+        echo "[perfmon] injected health_check (90min ceiling) into $CONFIG_PATH"
+    fi


Existing health_check not bumped

Medium Severity

On the perfmon path, gb200 only appends a 90-minute health_check when the recipe has no top-level health_check: key. Typical dsr1 recipes already define health_check with max_attempts: 360 (60 minutes), so the launcher leaves that limit unchanged. The b200 path forcibly sets four-space max_attempts to 540 for the same warmup rationale, so gb200 powercheck may run with a shorter ready ceiling than intended.

^{Reviewed by Cursor Bugbot for commit 742665d. Configure here.}

github-actions · 2026-06-16T21:45:20Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27649212722
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27649212722

github-actions · 2026-06-17T05:51:56Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27650031789
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27650031789

cursor · 2026-06-17T21:37:36Z

+        echo "[perfmon] WARNING: zero recipe YAMLs found under recipes/ — power data will be MISSING from this run." >&2
+    else
+        echo "[perfmon] injected monitoring: into $INJECTED_COUNT of $FOUND_COUNT recipes."
+    fi


gb300 dsr1 branch too broad

High Severity

The new perfmon path triggers on MODEL_PREFIX == dsr1 only, without requiring dynamo-sglang. Any gb300-nv job with prefix dsr1—including dsr1-fp8-gb300-dynamo-trt—clones the perfmon fork, sets PERFMON_ENABLED, and rewrites all recipe YAMLs, instead of the sa-submission TRT path used before.

^{Reviewed by Cursor Bugbot for commit b357089. Configure here.}

cursor · 2026-06-17T21:37:36Z

+    if [ "$perf_csv_count" -gt 0 ]; then
+        mkdir -p "$GITHUB_WORKSPACE/perf_samples"
+        cp "$LOGS_DIR"/perf_samples_*.csv "$GITHUB_WORKSPACE/perf_samples/"
+        echo "GPU_METRICS_CSV_GLOB=$GITHUB_WORKSPACE/perf_samples/perf_samples_*.csv" >> "$GITHUB_ENV"


gb300 omits nested perf CSVs

Medium Severity

gb300-nv staging counts and copies perf_samples_*.csv only from the top of LOGS_DIR, while gb200 and b200 use recursive find. If srt-slurm writes perfmon CSVs under job log subdirectories—as b200’s launcher assumes—gb300 powercheck runs can finish green but skip measured power aggregation.

Additional Locations (2)

runners/launch_b200-dgxc.sh#L418-L422

runners/launch_gb200-nv.sh#L483-L487

^{Reviewed by Cursor Bugbot for commit b357089. Configure here.}

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 6 total unresolved issues (including 5 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 12dfbb8. Configure here.}

cursor · 2026-06-18T01:01:22Z

+                float(duration),
+                int(total_output),
+                int(total_input),
+            )


Bench window UTC versus local

Medium Severity

When multinode bench JSON lacks Unix window fields, tier-2 windowing treats the date string as UTC while perfmon/nvidia-smi CSV timestamps are parsed as naive local time. On runners whose timezone differs from how date was written, the filter window can miss all samples and skip power patching despite valid CSVs.

^{Reviewed by Cursor Bugbot for commit 12dfbb8. Configure here.}

github-actions · 2026-06-18T01:27:32Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27729704236
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27729704236

…en pattern) Mirrors the gb300 wiring: the dynamo-sglang + dsr1 path now clones the SemiAnalysisAI perfmon fork (which ships the gb200 dsr1 recipes at recipes/gb200-fp?/<seq>/*.yaml — the exact CONFIG_FILE the dsr1-fp?-gb200-dynamo-sglang configs reference — PLUS the perfmon machinery) instead of cquil11/srt-slurm-nv. Injects monitoring: + health_check (90min ceiling for the ~35min dsr1 warmup), and stages perf_samples_*.csv + GPU_METRICS_CSV_GLOB for Process-result. Targets canonical NVIDIA measured power+temp on the HEALTHY gb200 runner, sidestepping the wedged gb300-nv fleet. Run is gated. Known risk to validate on first run: the Oracle/watchtower gb200 cluster may not cross-mount compute->head, so per-node perf_samples written on compute may need the shared-FS treatment the minimax path already uses. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…nical NVIDIA) Path-B-style run-only branch off main to get CANONICAL NVIDIA dsr1 power+temp on the HEALTHY gb200 runner, sidestepping the wedged gb300-nv fleet. Carries #1574's self-contained consumer code (aggregate_power.py/process_result.py + tests) + a 1-job changelog entry (dsr1-fp4-gb200-dynamo-sglang-powercheck). gb200 launcher perfmon wiring is the prior commit. Matrix verified = exactly 1 gb200 job. DO NOT MERGE (duplicates #1574). Run-only to harvest measured data; close after it lands. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…ical NVIDIA) Wire perfmon into launch_b200-dgxc.sh's dsr1-fp4 dynamo-sglang branch: clone SemiAnalysisAI/srt-slurm@feat/inferencex-perfmon (ships recipes/ b200-fp4/1k1k.yaml + nvidia-smi perfmon) instead of NVIDIA/srt-slurm@main, insert monitoring: INTO the recipe base: block (survives the zip_override), bump base.health_check.max_attempts to 540 (90min), and stage perf_samples_*.csv + GPU_METRICS_CSV_GLOB before the outputs cleanup. Add dsr1-fp4-b200-dynamo-sglang-powercheck (1k1k stp_lowlat[0], 1P TP4 + 5D TP8, conc[16]) -> matrix expands to ONE b200 job. The b200-multinode pool (dgxc-slurm_{7,8,9}) health is unverified (idle 60+ sweeps) but is the best of the wedged/idle NVIDIA multinode pools. Run-only; do not merge. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Extends the measured-power campaign (PR #1574) to the NVIDIA gb300-nv runner for DeepSeek-R1. The dsr1 branch in launch_gb300-nv.sh clones SemiAnalysisAI/srt-slurm@feat/inferencex-perfmon (NVIDIA PR #35), recursively injects monitoring: into every recipe (find -type f, not a flat glob), and stages per-node perf_samples_*.csv to $GITHUB_WORKSPACE so the Process-result step's aggregate_power.py patches the agg JSON with measured per-phase board power. Adds dsr1-fp4-gb300-dynamo-sglang-powercheck: a minimal single-job validation (1k/1k, conc 8, 1xP TP4 + 2xD TP4) of the plumbing before the full dsr1-disagg-NVIDIA sweep. Once this lands on a main that already carries #1574's aggregate_power consumer code, the changelog entry triggers exactly 1 gb300 job. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…runner The 'Slurm cleanup (pre-run)' drain loop waited for jobs named after the runner with NO timeout; a stuck/CG zombie scancel can't reap hangs it forever (observed 13-20min+ on gb200/b200/gb300-nv dsr1 legs). Bound the drain to 120s -> force KILL + proceed, and cap the benchmark_logs rm with timeout 60 (stale-NFS guard). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…angs) The 120s deadline added previously sits in the loop BODY, but on the NVIDIA clusters squeue hangs in the while-CONDITION's $(squeue ...), so the body never runs. Wrap scancel/squeue in 'timeout 30' so each call returns and the loop can progress to the deadline + force-KILL. gb300-cw (CoreWeave) cleanup clears fine; gb300-nv/gb200/b200 wedge -> NVIDIA-cluster slurm specific. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

github-actions · 2026-06-18T01:48:07Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27730497283
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27730497283

github-actions · 2026-06-18T02:07:02Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27731212420
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27731212420

arygupt requested a review from a team June 15, 2026 21:43

arygupt requested review from jgangani and kedarpotdar-nv as code owners June 15, 2026 21:43

github-project-automation Bot added this to InferenceMAX Board Jun 15, 2026

arygupt added the sweep-enabled label Jun 15, 2026

github-code-quality Bot found potential problems Jun 15, 2026

View reviewed changes

Comment thread utils/aggregate_power.py Fixed

cursor Bot reviewed Jun 15, 2026

View reviewed changes

arygupt added sweep-enabled and removed sweep-enabled labels Jun 15, 2026

arygupt force-pushed the feat/dsr1-gb200-powercheck-run branch from c0b79f4 to 742665d Compare June 16, 2026 21:22

arygupt added sweep-enabled and removed sweep-enabled labels Jun 16, 2026

github-code-quality Bot found potential problems Jun 16, 2026

View reviewed changes

Comment thread utils/aggregate_power.py

# Back-compat shim — some external callers may have imported _parse_power.

_parse_power = _parse_numeric_cell

cursor Bot reviewed Jun 16, 2026

View reviewed changes

arygupt force-pushed the feat/dsr1-gb200-powercheck-run branch from 742665d to 06f8f37 Compare June 16, 2026 21:25

arygupt added sweep-enabled and removed sweep-enabled labels Jun 16, 2026

arygupt force-pushed the feat/dsr1-gb200-powercheck-run branch from 06f8f37 to b357089 Compare June 17, 2026 21:35

cursor Bot reviewed Jun 17, 2026

View reviewed changes

arygupt force-pushed the feat/dsr1-gb200-powercheck-run branch from b357089 to 1df33dc Compare June 17, 2026 21:38

arygupt added sweep-enabled and removed sweep-enabled labels Jun 17, 2026

arygupt mentioned this pull request Jun 17, 2026

fix: restore trailing newline in perf-changelog.yaml (unblocks all changelog sweeps) #1815

Open

arygupt force-pushed the feat/dsr1-gb200-powercheck-run branch from 1df33dc to 12dfbb8 Compare June 18, 2026 00:59

arygupt added sweep-enabled and removed sweep-enabled labels Jun 18, 2026

cursor Bot reviewed Jun 18, 2026

View reviewed changes

arygupt added sweep-enabled and removed sweep-enabled labels Jun 18, 2026

arygupt and others added 6 commits June 17, 2026 18:42

arygupt force-pushed the feat/dsr1-gb200-powercheck-run branch from 86125c4 to 9bf533c Compare June 18, 2026 01:42

arygupt added sweep-enabled and removed sweep-enabled labels Jun 18, 2026



		# Back-compat shim — some external callers may have imported _parse_power.
		_parse_power = _parse_numeric_cell

Conversation

arygupt commented Jun 15, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ DO NOT MERGE — run-only validation (canonical NVIDIA via gb200)

Uh oh!

github-actions Bot commented Jun 15, 2026

Uh oh!

Uh oh!

cursor Bot Jun 15, 2026

Choose a reason for hiding this comment

Desync samples inflate joules

Uh oh!

github-actions Bot commented Jun 15, 2026

Uh oh!

github-actions Bot commented Jun 15, 2026

Uh oh!

cursor Bot Jun 16, 2026

Choose a reason for hiding this comment

Gb200 dsr1 lacks fp4 guard

Uh oh!

cursor Bot Jun 16, 2026

Choose a reason for hiding this comment

Existing health_check not bumped

Uh oh!

github-actions Bot commented Jun 16, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

Uh oh!

cursor Bot Jun 17, 2026

Choose a reason for hiding this comment

gb300 dsr1 branch too broad

Uh oh!

cursor Bot Jun 17, 2026

Choose a reason for hiding this comment

gb300 omits nested perf CSVs

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Jun 18, 2026

Choose a reason for hiding this comment

Bench window UTC versus local

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

arygupt commented Jun 15, 2026 •

edited by cursor Bot

Loading