Skip to content

From NVIDIA Megatron-LM for visibility#18

Open
RaymondLi0 wants to merge 7212 commits into
bigcode-project:multi-query-attentionfrom
NVIDIA:main
Open

From NVIDIA Megatron-LM for visibility#18
RaymondLi0 wants to merge 7212 commits into
bigcode-project:multi-query-attentionfrom
NVIDIA:main

Conversation

@RaymondLi0

@RaymondLi0 RaymondLi0 commented Jan 24, 2023

Copy link
Copy Markdown
Collaborator

No description provided.

@RaymondLi0 RaymondLi0 changed the base branch from multi-query-attention to before-merge June 20, 2023 20:12
@RaymondLi0 RaymondLi0 changed the base branch from before-merge to multi-query-attention June 20, 2023 20:12
pavelgein and others added 27 commits May 15, 2026 20:17
@pavelgein @asolergi-nv
[fix] Use MSC for checking checkpoint existence (#4251)
f2070ee 
Signed-off-by: Pavel Gein <pavel.gein@gmail.com>
Co-authored-by: Antoni-Joan Solergibert <asolergibert@nvidia.com>
@gdengk
Reorder mtp_post_process after attention backward in 1F1B schedule pl…
212f885 
…an (#4695)

Co-authored-by: Gao Deng <gdeng@login-lyris01.lyris.clusters.nvidia.com>
@Wohox
[Main][feat] Support A2A Overlap for Megatron-FSDP (#3797)
e9d9a4c
@xrennvidia @Phlip79
add is_torch_min_version in fsdp src (#4812)
7371e01 
Signed-off-by: Xiaowei Ren <xren@nvidia.com>
Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com>
@gdengk @ericharper @Phlip79
Add high-priority A2A stream and HybridEP preprocessing SMs (#4694)
54f90af 
Co-authored-by: Gao Deng <gdeng@login-lyris02.lyris.clusters.nvidia.com>
Co-authored-by: Eric Harper <eharper@nvidia.com>
Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com>
@buptzyb @claude
Refactor CUDA graph API: decompose cuda_graph_scope into full_iterati…
e41b370 
…on impl, inference scope, and per-layer capture modules (#4292)

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@github-actions
chore(beep boop 🤖): Bump (main) (2026-05-18)
a1c5f5d
@dimapihtar
Tokenizers updates (#4780)
ab28544 
Signed-off-by: dimapihtar <dpykhtar@nvidia.com>
@dimapihtar
Fix no nvrx tests (#4847)
5096459 
Signed-off-by: dimapihtar <dpykhtar@nvidia.com>
@yashaswikarnati
Thread custom process groups through MoE grad finalization (#4782)
aa786b7
@shanmugamr1992 @claude
Fix unit tests (#4689)
648b916 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@shanmugamr1992 @claude @chtruong814
Tests/dynamic inference functional coverage (#4761)
6c3fa6d 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Charlie Truong <chtruong@nvidia.com>
@github-actions
Update copy-pr-bot.yaml [skip ci]
74daec9
@janEbert
Fix oncall references (#4722)
952efb4
@balasaajay
Update golden values for nightly functional tests (#4850)
b2a8ec7
@athitten
fix(inference): size DynamicInferenceContext KV layer_map for non-uni…
43fb2f5 
…form PP (#4775)

Co-authored-by: Abhishree Thittenamane <athittenaman@cw-dfw-cs-001-login-01.cm.cluster>
Co-authored-by: root <root@pool0-01849.cm.cluster>
@kevalmorabia97
Modernize post-training modelopt example scripts (#4807)
14aaa7e 
Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com>
@github-actions
chore: rotate oncall schedule
38986a9
@shanmugamr1992 @claude
test: add inference performance test harness for GPT 583M, hybrid 2B,… (
23ba357 
#4806)

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@chtruong814
ci: Prevent shell trace in parts of _run_training.sh (#4884)
1ffcd1d 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@wujingyue
Ignore Vim swap files (#4860)
14dabeb
@shjwudp
M-FSDP: Make fine_grained_param_gather configurable for MXFP8 to enab…
b5d143f 
…le performance–memory trade-offs (#4181)
@kamran-nvidia
MimoOptimizer: fix distributed checkpoint save and load for non-coloc…
68d64f7 
…ated MIMO (#4801)

Signed-off-by: Kamran Jafari <kjafarisadeg@nvidia.com>
@github-actions
Update copy-pr-bot.yaml [skip ci]
5df3e4e
@deepakn94 @claude
Route non-Muon params through DistributedOptimizer (#4771)
0044db1 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@chtruong814
ci: Gate optional CI jobs with repository variables (#4907)
990fa69 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@Phlip79 @claude
Fix tokenizers bug in nightly (#4833)
e69a1ea 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
tdene and others added 30 commits June 16, 2026 19:32
@tdene
Make Megatron RL only materialize last token logit (#4551)
b00cad1 
Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>
@jalbericiola @tdene
Profiling (#3110)
a12484b 
Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>
Co-authored-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>
@sraman-rgb
Support fused MLA QKV checkpoint reload (#5310)
2b90b3f 
Signed-off-by: sraman <sraman@nvidia.com>
@wujingyue
Add minimal DBuffer implementation (#4835)
000dc1c 
Signed-off-by: Jingyue Wu <wujingyue@gmail.com>
@HollowMan6
[split 1/5] Fix packed THD RoPE under CP (#5243)
d30e165 
Signed-off-by: Hollow Man <hollowman@opensuse.org>
@github-actions
Update copy-pr-bot.yaml [skip ci]
49737fd
@wujingyue
Document agent PR commit sign-off and signing (#5381)
2e1183a 
Signed-off-by: Jingyue Wu <wujingyue@gmail.com>
@wujingyue
Remove unused distributed pytest markers (#5380)
2463dbe 
Signed-off-by: Jingyue Wu <wujingyue@gmail.com>
@lhb8125
[feat] Support fine-grained activation offloading in fused group mlp (#…
5c660c3 
…5082)

Signed-off-by: hongbinl <hongbinl@nvidia.com>
@yashaswikarnati
Thread tensor-parallel group into the RADIO patch embedder (#5371)
bd381ac 
Signed-off-by: ykarnati <ykarnati@nvidia.com>
@yashaswikarnati @claude
Add MimoModel.zero_grad_buffer delegating to active DDP submodules (#…
a00c0de 
…5372)

Signed-off-by: ykarnati <ykarnati@nvidia.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
@HollowMan6
[split 3/5] Refactor absorbed MLA projection handling (#5245)
7604f28 
Signed-off-by: Hollow Man <hollowman@opensuse.org>
@github-actions
chore: rotate oncall schedule
5182aa6
@chtruong814
ci: Remove sync skills workflow (#5091)
41dbab4 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@balasaajay
Add flaky marker to fine-grained activation offloading test (#5350) (#…
ad5a93b 
…5368)

Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
@balasaajay
Revert "Remove checkpoint-time GPU cache reclaim workaround (#5170)" (#…
be82829 
…5366)

Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
@balasaajay
Update goldens for weekly tests after pytorch and TE bumps. (#5399)
a50252b 
Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
@yashaswikarnati @claude
Add MIMO runtime setup: per-role RNG seeding and DDP wrapping (#5285)
d1410e1 
Signed-off-by: ykarnati <ykarnati@nvidia.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
@tdene @jalbericiola
Add --mamba-training-ssm-states-dtype argument (#5309)
6142ee4 
Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>
Co-authored-by: Jorge Albericio <jalbericiola@nvidia.com>
@github-actions
chore(beep boop 🤖): Bump (main) (2026-06-22)
2cb1b80
@lmcafee-nvidia
Fix Mamba prefix match for chunked prefill (#4758)
57c484e 
Signed-off-by: Lawrence McAfee <lmcafee@nvidia.com>
@wdykas @claude
Disag MR2: Refit into multiple destination pools and tied-embedding +…
ae2efd5 
… UVM fixes (#5187)

Signed-off-by: wdykas <wdykas@nvidia.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
@wdykas @claude
Disag MR1: Add inference shard specs and pg-collection building (#5186)
fc4597c 
Signed-off-by: wdykas <wdykas@nvidia.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
@yashaswikarnati @claude
Support the MIMO cross-grid path in training loop (#5373)
e1c4495 
Signed-off-by: ykarnati <ykarnati@nvidia.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
@shanmugamr1992
Stabilize hybrid_2b GB200 perf test against run-to-run noise (#5364)
6bd392f
@Phlip79
Consistent oncall schedule (#5404)
b6b44a7 
Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com>
@wdykas
Disag MR3: Add heterogeneous KV/Mamba reshard planners (#5188)
93a7642 
Signed-off-by: wdykas <wdykas@nvidia.com>
@yashaswikarnati @claude
Add RADIO vision encoder wrapper for MIMO example (#5397)
2a46893 
Signed-off-by: ykarnati <ykarnati@nvidia.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
@santhnm2
Clean up MTP inference control flow (#5418)
76f6ccc 
Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com>
@yashaswikarnati @claude
Add MIMO dual gradient finalization (colocated + non-colocated) (#5286)
f8170b4 
Signed-off-by: ykarnati <ykarnati@nvidia.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.