Make MCP indexing observable and tools available immediately

[Neverdecel]

Addresses dev-team findings about CodeRAG-as-MCP-tool feeling broken at
broad-root (e.g. /home) scale: "enabled" but not "usable", opaque status,
and tools that aren't reachable in time inside Hermes/Claude Code/Codex.

• Serve the MCP protocol immediately. run_mcp() moved warm-up (which
  downloads/loads the embedding model on first run) and the initial index
  into one background bootstrap thread, so mcp.run() — and thus tools/list
  — is reached at once. The toolset is registered synchronously in
  build_mcp(), so clients see all five tools right away instead of timing
  out behind the model download. This is the CodeRAG-side cause of the
  "tools registered but unusable" race.

• Live, pollable index progress. New thread-safe IndexProgress (types.py)
  threaded from Indexer.index(live=...) up through CodeRAG.index() to the
  index_status tool, which now returns a progress object: state
  (idle/scanning/indexing/optimizing/ready/failed), files_discovered,
  files_to_index, files_indexed, chunks, current_path, elapsed, last_error.
  files_discovered ticks up during the long pre-embed scan, so a big index
  reads as "scanning" instead of a stuck 0. reindex drives the same object.

• Earlier partial results. During a live (MCP background) index, buffered
  rows are committed every ~5s so dense search returns hits before the
  8192-chunk flush boundary. Gated on live is not None, so the CLI, watcher
  and tests keep today's single-flush-at-end batching (defaults unchanged).

• MCP best-practice polish. ToolAnnotations(readOnlyHint=True) on
  search_code/search_files/get_file/index_status; reindex marked non-read-only.

• Concurrency safety. Serving before warm-up means a query can arrive mid
  bootstrap; guard CodeRAG's lazy provider/store/searcher/indexer
  construction with a reentrant build lock so two threads can't build a
  second conflicting LanceStore.

Tests: live-progress + failure-path tests in test_indexer.py; live
progress, annotations, and reindex-progress assertions in test_mcp.py.

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_015SPsWy8a63EpYMFDJjVE1e

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 84.00000% with 20 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
coderag/surfaces/mcp_server.py	43.75%	18 Missing ⚠️
coderag/indexer.py	91.66%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!