Skip to content

Materialize Letta core/archive export benchmark#211

Merged
yvette-carlisle merged 1 commit into
mainfrom
y/elf-xy-984
Jun 19, 2026
Merged

Materialize Letta core/archive export benchmark#211
yvette-carlisle merged 1 commit into
mainfrom
y/elf-xy-984

Conversation

@yvette-carlisle

Copy link
Copy Markdown
Member

Summary

  • add Docker-contained Letta core/archive export-readback smoke task and optional Letta compose profile
  • publish the June 19 Letta materialization evidence report and JSON companion
  • update external adapter manifest/tests so all six Letta core/archive scenarios are typed blocked until live export/readback source ids exist

Benchmark

  • cargo make smoke-letta-core-archive-export-readback: 6 jobs, 0 pass, 0 wrong_result, 6 blocked, 14/14 evidence/source-ref/quote coverage
  • cargo make real-world-memory-core-archival: 6 jobs, 6 pass, 0 blocked, 0 wrong_result, evidence coverage 1.0

Validation

  • python -m py_compile scripts/letta-core-archive-export-readback-smoke.py
  • cargo make smoke-letta-core-archive-export-readback
  • jq empty apps/elf-eval/fixtures/report_snapshots/2026-06-19-letta-core-archive-export-readback-report.json && jq empty apps/elf-eval/fixtures/real_world_external_adapters/memory_projects_manifest.json
  • cargo test -p elf-eval --test real_world_job_benchmark letta_core_archive_export_readback_report_preserves_blocked_gates -- --test-threads=1
  • cargo test -p elf-eval --test real_world_job_benchmark real_world_report_includes_external_adapter_coverage_manifest -- --test-threads=1
  • cargo test -p elf-eval --test real_world_job_benchmark external_adapter_run_summarizes_nonzero_scenario_losses -- --test-threads=1
  • cargo test -p elf-eval --test real_world_job_benchmark external_adapter_manifest_rejects_unmeasured_win_loss_scenario_outcomes -- --test-threads=1
  • cargo test -p elf-eval --test real_world_job_benchmark generated_json_report_renders_markdown -- --test-threads=1
  • cargo make real-world-memory-core-archival
  • cargo make fmt-check
  • cargo make check-docs
  • decodex docs check
  • cargo make lint
  • cargo make check

Comment thread scripts/letta-core-archive-export-readback-smoke.py Fixed
Comment thread scripts/letta-core-archive-export-readback-smoke.py Fixed
@yvette-carlisle yvette-carlisle merged commit 66e9313 into main Jun 19, 2026
12 checks passed
@yvette-carlisle yvette-carlisle deleted the y/elf-xy-984 branch June 19, 2026 06:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant