Documentation Coverage
Documentation is the canonical account of ReplayLab's current user experience. If a user-facing capability is missing from this matrix, treat that as a documentation gap and update the docs before calling the capability complete.
Statuses:
- Implemented: documented, runnable or inspectable, and covered by tests or examples.
- Prepared: documented and implemented as a maintainer workflow, but not yet verified by the external service it targets.
- Partial: implemented but the workflow still has a known UX gap.
- Planned: intentionally not implemented yet.
| Capability | User Interface | Documentation | Example or Notebook | Test Coverage | Status |
|---|---|---|---|---|---|
| Understand why ReplayLab exists | MkDocs, README | docs/why-replaylab.md, README.md |
examples/tutorials/* |
tests/unit/test_tutorial_notebooks.py |
Implemented |
| Install and smoke-check packages | CLI, package metadata | README.md, docs/release-checklist.md, CHANGELOG.md |
scripts/release_smoke.py |
tests/unit/test_release_metadata.py, GitHub package smoke job |
Implemented |
Prepare 0.1.0a4 release candidate |
Package metadata, release scripts, scenario wheelhouse installs | README.md, docs/release-checklist.md, CHANGELOG.md, PLAN.md |
python scripts/release_rehearsal.py --channel github --version 0.1.0a4 |
tests/unit/test_release_metadata.py, tests/unit/test_release_rehearsal.py, tests/unit/test_run_scenario.py, package smoke job |
Implemented |
Publish 0.1.0a4 public alpha |
GitHub release, TestPyPI, PyPI | README.md, docs/release-checklist.md, CHANGELOG.md, PLAN.md |
https://github.com/Symblee/ReplayLab/releases/tag/v0.1.0a4, python -m pip install replaylab==0.1.0a4 |
GitHub release asset hash check, TestPyPI install smoke, PyPI install smoke, http-loopback-pypi --package-version 0.1.0a4, support-bot-demo-local --package-version 0.1.0a4 |
Implemented |
Prepare 0.1.0a3 release candidate |
Package metadata, release scripts, scenario wheelhouse installs | README.md, docs/release-checklist.md, CHANGELOG.md, PLAN.md |
python scripts/release_rehearsal.py --channel github --version 0.1.0a3 |
tests/unit/test_release_metadata.py, tests/unit/test_release_rehearsal.py, tests/unit/test_run_scenario.py, package smoke job |
Implemented |
Publish 0.1.0a3 public alpha |
GitHub release, TestPyPI, PyPI | README.md, docs/release-checklist.md, CHANGELOG.md, PLAN.md |
https://github.com/Symblee/ReplayLab/releases/tag/v0.1.0a3, python -m pip install replaylab==0.1.0a3 |
GitHub release asset hash check, TestPyPI install smoke, PyPI install smoke, http-loopback-pypi --package-version 0.1.0a3 |
Implemented |
| Rehearse GitHub prerelease assets | Release script, GitHub Actions | README.md, docs/release-checklist.md, CHANGELOG.md |
scripts/release_rehearsal.py |
tests/unit/test_release_rehearsal.py, GitHub package smoke job |
Implemented |
| Install from GitHub public-alpha prerelease | GitHub release assets, pip | README.md, docs/release-checklist.md, CHANGELOG.md |
https://github.com/Symblee/ReplayLab/releases/tag/v0.1.0a4 |
GitHub asset hash check, clean virtualenv install smoke | Implemented |
| Publish to TestPyPI and PyPI | GitHub Actions, trusted publishers | README.md, docs/release-checklist.md |
.github/workflows/publish.yml |
tests/unit/test_publish_workflow.py, TestPyPI and PyPI install smoke |
Implemented |
| Install from PyPI public alpha | pip | README.md, docs/release-checklist.md, CHANGELOG.md |
python -m pip install replaylab==0.1.0a4 |
Clean Python 3.13 PyPI install smoke | Implemented |
Verify post-release 0.1.0a3 confidence |
pip, scenario runner, paid OpenAI check | docs/release-checklist.md, PLAN.md |
python -m pip install replaylab, python scripts/run_scenario.py run openai-real-pypi --paid --keep-workspace --package-version 0.1.0a3 |
Clean pinned/unpinned PyPI install smoke, no-key-dogfood, http-loopback-pypi --package-version 0.1.0a3, support-bot-demo-local --package-version 0.1.0a3, paid OpenAI scenario validation |
Implemented |
| Dogfood the PyPI alpha against real OpenAI | Maintainer script, PyPI install | docs/release-checklist.md, PLAN.md |
scripts/real_openai_dogfood.py |
tests/unit/test_real_openai_dogfood.py, manual OpenAI validation |
Implemented |
| Dogfood the PyPI alpha against real HTTP clients | Maintainer script, PyPI install | docs/release-checklist.md, PLAN.md |
scripts/real_http_dogfood.py |
tests/unit/test_real_http_dogfood.py, manual loopback HTTP validation |
Implemented |
| Run scenario-driven validation | Maintainer script, e2e tests | docs/scenario-validation.md, docs/release-checklist.md, AGENTS.md |
scripts/run_scenario.py |
tests/unit/test_run_scenario.py, tests/e2e/test_scenarios.py |
Implemented |
| Run no-key deterministic scenario | Scenario runner | docs/scenario-validation.md, docs/quickstart.md |
python scripts/run_scenario.py run no-key-dogfood |
tests/e2e/test_scenarios.py |
Implemented |
| Run fresh PyPI first-project scenario | Scenario runner | docs/scenario-validation.md, docs/tutorials/first-pypi-project.md |
python scripts/run_scenario.py run new-user-pypi-local --keep-workspace --package-version 0.1.0a4 |
tests/unit/test_new_user_onboarding_scenario.py, tests/unit/test_run_scenario.py, manual loopback scenario validation |
Implemented |
| Run customer support-bot demo scenario | Scenario runner | docs/scenario-validation.md, docs/tutorials/replaylab-in-10-minutes.md |
python scripts/run_scenario.py run support-bot-demo-local --keep-workspace --package-version 0.1.0a4 |
tests/unit/test_support_bot_demo_scenario.py, tests/unit/test_run_scenario.py, manual loopback scenario validation |
Implemented |
| Run HTTP loopback scenario | Scenario runner | docs/scenario-validation.md, docs/release-checklist.md |
python scripts/run_scenario.py run http-loopback-pypi --keep-workspace |
tests/unit/test_run_scenario.py, manual loopback scenario validation |
Implemented |
| Run HTTP edge-case scenario | Scenario runner | docs/scenario-validation.md, docs/release-checklist.md, docs/support-matrix.md |
python scripts/run_scenario.py run http-edge-cases-local --keep-workspace |
tests/unit/test_http_edge_cases_scenario.py, tests/unit/test_run_scenario.py, manual loopback scenario validation |
Implemented |
| Run auto-instrumentation scenario | Scenario runner | docs/scenario-validation.md, docs/tutorials/asgi-fastapi-middleware.md |
python scripts/run_scenario.py run auto-instrumentation-local --keep-workspace |
tests/unit/test_auto_instrumentation_scenario.py, tests/unit/test_run_scenario.py, manual loopback scenario validation |
Implemented |
| Run ASGI/FastAPI middleware scenario | Scenario runner | docs/scenario-validation.md, docs/tutorials/asgi-fastapi-middleware.md |
python scripts/run_scenario.py run asgi-fastapi-local --keep-workspace |
tests/unit/test_asgi_fastapi_scenario.py, tests/unit/test_run_scenario.py, manual loopback scenario validation |
Implemented |
| Run ASGI lifecycle scenario | Scenario runner | docs/scenario-validation.md, docs/tutorials/asgi-fastapi-middleware.md |
python scripts/run_scenario.py run asgi-lifecycle-local --keep-workspace |
tests/unit/test_asgi_lifecycle_scenario.py, tests/unit/test_run_scenario.py, manual loopback scenario validation |
Implemented |
| Run worker/job decorator scenario | Scenario runner | docs/scenario-validation.md, docs/tutorials/worker-job-adapter.md |
python scripts/run_scenario.py run job-worker-local --keep-workspace |
tests/unit/test_job_worker_scenario.py, tests/unit/test_run_scenario.py, manual loopback scenario validation |
Implemented |
| Run job lifecycle scenario | Scenario runner | docs/scenario-validation.md, docs/tutorials/worker-job-adapter.md |
python scripts/run_scenario.py run job-lifecycle-local --keep-workspace |
tests/unit/test_job_lifecycle_scenario.py, tests/unit/test_run_scenario.py, manual loopback scenario validation |
Implemented |
| Run PydanticAI compatibility scenario | Scenario runner | docs/scenario-validation.md, docs/tutorials/pydantic-ai-compatibility.md |
python scripts/run_scenario.py run pydantic-ai-local --keep-workspace |
tests/unit/test_framework_compat_scenarios.py, tests/unit/test_run_scenario.py, manual loopback scenario validation |
Implemented |
| Run LangGraph compatibility scenario | Scenario runner, local app trace-shape validator | docs/scenario-validation.md, docs/tutorials/langgraph-compatibility.md |
python scripts/run_scenario.py run langgraph-local --keep-workspace |
tests/unit/test_framework_compat_scenarios.py, tests/unit/test_run_scenario.py, tests/unit/app/test_trace_projection.py, manual loopback scenario validation |
Implemented |
| Run external LangGraph realism scenario | Scenario runner, external example project, local app product checks, visual trace gate | docs/scenario-validation.md, docs/tutorials/langgraph-compatibility.md |
python scripts/run_scenario.py run langgraph-external-local --keep-workspace |
tests/unit/test_framework_compat_scenarios.py, tests/unit/test_run_scenario.py, tests/unit/app/test_trace_projection.py, manual external-project loopback scenario validation |
Implemented |
| Run LangChain ChatOpenAI compatibility scenario | Scenario runner, local app product surface checks | docs/scenario-validation.md, docs/tutorials/langchain-openai-compatibility.md |
python scripts/run_scenario.py run langchain-openai-local --keep-workspace |
tests/unit/test_framework_compat_scenarios.py, tests/unit/test_run_scenario.py, manual loopback scenario validation |
Implemented |
| Run Anthropic Messages compatibility scenario | Scenario runner, local app trace-shape validator | docs/scenario-validation.md, docs/tutorials/anthropic-compatibility.md, docs/support-matrix.md |
python scripts/run_scenario.py run anthropic-local --keep-workspace |
tests/unit/test_provider_llm_adapters.py, tests/unit/test_provider_auto_patch_parity.py, tests/unit/app/test_provider_protocol_previews.py, tests/unit/test_framework_compat_scenarios.py, tests/unit/test_run_scenario.py, manual loopback scenario and visual app validation |
Implemented |
| Run Gemini Generate Content compatibility scenario | Scenario runner, local app trace-shape validator | docs/scenario-validation.md, docs/tutorials/gemini-compatibility.md, docs/support-matrix.md |
python scripts/run_scenario.py run gemini-local --keep-workspace |
tests/unit/test_provider_llm_adapters.py, tests/unit/test_provider_auto_patch_parity.py, tests/unit/app/test_provider_protocol_previews.py, tests/unit/test_framework_compat_scenarios.py, tests/unit/test_run_scenario.py, manual loopback scenario and visual app validation |
Implemented |
| Run OpenAI streaming compatibility scenario | Scenario runner, local app trace-shape validator | docs/scenario-validation.md, docs/tutorials/openai-responses.md, docs/support-matrix.md |
python scripts/run_scenario.py run openai-streaming-local --keep-workspace |
tests/unit/test_openai_adapter.py, tests/unit/test_provider_llm_common.py, tests/unit/test_framework_compat_scenarios.py, tests/unit/test_run_scenario.py, manual loopback scenario validation |
Implemented |
| Run Anthropic streaming compatibility scenario | Scenario runner, local app trace-shape validator | docs/scenario-validation.md, docs/tutorials/anthropic-compatibility.md, docs/support-matrix.md |
python scripts/run_scenario.py run anthropic-streaming-local --keep-workspace |
tests/unit/test_provider_llm_adapters.py, tests/unit/test_provider_llm_adapter_edges.py, tests/unit/test_provider_llm_common.py, tests/unit/test_framework_compat_scenarios.py, tests/unit/test_run_scenario.py, manual loopback scenario validation |
Implemented |
| Run Gemini streaming compatibility scenario | Scenario runner, local app trace-shape validator | docs/scenario-validation.md, docs/tutorials/gemini-compatibility.md, docs/support-matrix.md |
python scripts/run_scenario.py run gemini-streaming-local --keep-workspace |
tests/unit/test_provider_llm_adapters.py, tests/unit/test_provider_llm_adapter_edges.py, tests/unit/test_provider_llm_common.py, tests/unit/test_framework_compat_scenarios.py, tests/unit/test_run_scenario.py, manual loopback scenario validation |
Implemented |
| Run AI diagnosis loopback scenario | Scenario runner | docs/scenario-validation.md, docs/tutorials/ai-assisted-diagnosis.md |
python scripts/run_scenario.py run ai-diagnosis-loopback --keep-workspace |
tests/unit/test_ai_diagnosis_scenario.py, tests/unit/test_run_scenario.py, manual loopback scenario validation for report explanation, report-diff explanation, and instrumentation planning |
Implemented |
| Run HTML report scenario | Scenario runner | docs/scenario-validation.md, docs/quickstart.md |
python scripts/run_scenario.py run html-report-local --keep-workspace |
tests/unit/test_run_scenario.py, manual deterministic scenario validation |
Implemented |
| Run HTML report failure scenario | Scenario runner | docs/scenario-validation.md, docs/quickstart.md |
python scripts/run_scenario.py run html-report-failure-local --keep-workspace |
tests/unit/test_run_scenario.py, manual deterministic scenario validation |
Implemented |
| Run React viewer scenario | Scenario runner | docs/scenario-validation.md, docs/tutorials/local-react-viewer.md |
python scripts/run_scenario.py run react-viewer-local --keep-workspace |
tests/unit/test_run_scenario.py, manual deterministic scenario validation |
Implemented |
| Run viewer-first local workflow scenario | Scenario runner | docs/scenario-validation.md, docs/tutorials/local-react-viewer.md |
python scripts/run_scenario.py run viewer-first-local --keep-workspace |
tests/unit/test_run_scenario.py, manual deterministic scenario validation |
Implemented |
| Run guided local workflow scenario | Scenario runner | docs/scenario-validation.md, docs/quickstart.md |
python scripts/run_scenario.py run guided-local-workflow --keep-workspace |
tests/unit/test_run_scenario.py, manual deterministic scenario validation |
Implemented |
| Run local app shell scenario | Scenario runner | docs/scenario-validation.md, docs/concepts/local-artifacts.md |
python scripts/run_scenario.py run local-app-shell --keep-workspace |
tests/unit/test_local_app.py, tests/unit/test_run_scenario.py, manual deterministic scenario validation |
Implemented |
| Run local app workflow scenario | Scenario runner | docs/scenario-validation.md, docs/concepts/local-artifacts.md |
python scripts/run_scenario.py run local-app-workflow --keep-workspace |
tests/unit/test_local_app.py, tests/unit/test_run_scenario.py, frontend report-verdict tests, manual deterministic scenario validation |
Implemented |
| Run failure-story scenario | Scenario runner | docs/scenario-validation.md, docs/concepts/local-artifacts.md |
python scripts/run_scenario.py run failure-story-local --keep-workspace |
tests/unit/test_inspection.py, tests/unit/test_run_scenario.py, manual deterministic scenario validation |
Implemented |
| Run paid external PydanticAI realism scenario | Scenario runner, external example project, local app product checks | docs/scenario-validation.md, docs/tutorials/pydantic-ai-compatibility.md |
python scripts/run_scenario.py run pydantic-ai-external-paid --paid --keep-workspace --max-provider-calls 10 |
tests/unit/test_framework_compat_scenarios.py, tests/unit/test_run_scenario.py, manual paid external-project scenario validation with retry/cap policy |
Implemented |
| Run paid OpenAI scenario | Scenario runner | docs/scenario-validation.md, docs/release-checklist.md |
python scripts/run_scenario.py run openai-real-pypi --paid --keep-workspace |
tests/unit/test_run_scenario.py, manual paid OpenAI validation |
Implemented |
| Configure startup instrumentation | Python SDK | docs/concepts/integration-model.md, docs/tutorials/openai-responses.md, docs/tutorials/http-capture-replay.md |
examples/tutorials/openai_responses_capture_replay.ipynb, examples/tutorials/http_capture_replay.ipynb |
tests/unit/test_same_process_auto_instrumentation.py, tests/integration/test_same_process_startup.py |
Implemented |
| Simplify startup auto-instrumentation | Python SDK, CLI | docs/index.md, docs/concepts/integration-model.md, docs/support-matrix.md |
auto_patch_integrations="auto", --auto-patch-integrations auto |
tests/unit/test_replaylab_init.py, tests/unit/test_cli_run.py, tests/unit/test_cli_replay.py, tests/integration/test_same_process_startup.py |
Implemented |
| Scope request, job, or session capture | Python SDK | docs/concepts/integration-model.md, docs/index.md |
docs/tutorials/openai-responses.md, docs/tutorials/http-capture-replay.md |
tests/unit/test_capture_lifecycle.py, tests/integration/test_same_process_startup.py |
Implemented |
| Complete first PyPI project onboarding | pip, Python SDK, CLI | docs/tutorials/first-pypi-project.md, README.md |
app.py and provider.py snippets in docs/tutorials/first-pypi-project.md |
tests/unit/test_new_user_onboarding_scenario.py, new-user-pypi-local --package-version 0.1.0a4 |
Implemented |
| Run 10-minute support-bot demo | pip, Python SDK, local app, React viewer | docs/tutorials/replaylab-in-10-minutes.md, README.md |
support_bot.py generated by support-bot-demo-local |
tests/unit/test_support_bot_demo_scenario.py, support-bot-demo-local --package-version 0.1.0a4, current-checkout local app rehearsal |
Implemented |
| Capture ASGI/FastAPI request scopes | Python SDK helper and middleware | docs/concepts/integration-model.md, docs/tutorials/asgi-fastapi-middleware.md, docs/support-matrix.md |
scripts/asgi_fastapi_scenario.py, scripts/auto_instrumentation_scenario.py, scripts/asgi_lifecycle_scenario.py |
tests/unit/test_asgi_middleware.py, tests/integration/test_asgi_middleware_integration.py, tests/unit/test_asgi_fastapi_scenario.py, tests/unit/test_auto_instrumentation_scenario.py, tests/unit/test_asgi_lifecycle_scenario.py |
Implemented |
| Capture worker/job scopes | Python SDK decorator | docs/concepts/integration-model.md, docs/tutorials/worker-job-adapter.md, docs/support-matrix.md |
scripts/job_worker_scenario.py, scripts/job_lifecycle_scenario.py |
tests/unit/test_jobs_adapter.py, tests/integration/test_jobs_integration.py, tests/unit/test_job_worker_scenario.py, tests/unit/test_job_lifecycle_scenario.py |
Implemented |
| Validate PydanticAI compatibility | Scenario runner | docs/tutorials/pydantic-ai-compatibility.md, docs/support-matrix.md, docs/mvp-limitations.md |
scripts/pydantic_ai_scenario.py |
tests/unit/test_framework_compat_scenarios.py, manual loopback scenario validation |
Implemented |
| Validate LangGraph compatibility | Scenario runner | docs/tutorials/langgraph-compatibility.md, docs/support-matrix.md, docs/mvp-limitations.md |
scripts/langgraph_scenario.py |
tests/unit/test_framework_compat_scenarios.py, manual loopback scenario validation |
Implemented |
| Validate LangChain ChatOpenAI Responses compatibility | Scenario runner, local app product surface checks | docs/tutorials/langchain-openai-compatibility.md, docs/support-matrix.md, docs/mvp-limitations.md |
scripts/langchain_openai_scenario.py |
tests/unit/test_framework_compat_scenarios.py, tests/unit/test_run_scenario.py, manual loopback scenario validation |
Implemented |
| Run no-key local dogfood workflow | CLI | docs/quickstart.md |
examples/dogfood_mvp |
tests/integration/test_dogfood_mvp.py |
Implemented |
| Capture OpenAI Responses calls | Python SDK, CLI wrapper | docs/tutorials/openai-responses.md, docs/support-matrix.md |
examples/tutorials/openai_responses_capture_replay.ipynb, python scripts/run_scenario.py run openai-streaming-local --keep-workspace |
tests/unit/test_openai_adapter.py, tests/unit/test_provider_llm_common.py, tests/integration/test_openai_integration.py |
Implemented |
| Capture Anthropic Messages calls | Python SDK, CLI wrapper | docs/tutorials/anthropic-compatibility.md, docs/support-matrix.md |
python scripts/run_scenario.py run anthropic-local --keep-workspace, python scripts/run_scenario.py run anthropic-streaming-local --keep-workspace |
tests/unit/test_provider_llm_adapters.py, tests/unit/test_provider_llm_adapter_edges.py, tests/unit/test_provider_auto_patch_parity.py, tests/unit/app/test_provider_protocol_previews.py, tests/unit/test_framework_compat_scenarios.py |
Implemented |
| Capture Gemini Generate Content calls | Python SDK, CLI wrapper | docs/tutorials/gemini-compatibility.md, docs/support-matrix.md |
python scripts/run_scenario.py run gemini-local --keep-workspace, python scripts/run_scenario.py run gemini-streaming-local --keep-workspace |
tests/unit/test_provider_llm_adapters.py, tests/unit/test_provider_llm_adapter_edges.py, tests/unit/test_provider_auto_patch_parity.py, tests/unit/app/test_provider_protocol_previews.py, tests/unit/test_framework_compat_scenarios.py |
Implemented |
| Capture HTTP calls | Python SDK, CLI wrapper | docs/tutorials/http-capture-replay.md, docs/support-matrix.md |
examples/tutorials/http_capture_replay.ipynb |
tests/unit/test_http_adapters.py, tests/integration/test_http_integration.py |
Implemented |
| Persist local capsules and payloads | Python SDK | docs/concepts/local-artifacts.md, docs/mvp-limitations.md |
tests/golden/capsules/with_payloads |
tests/unit/test_capsule_storage.py, tests/integration/test_capsule_storage_integration.py |
Implemented |
| List capsules | CLI, Python SDK | docs/quickstart.md, docs/concepts/local-artifacts.md |
examples/dogfood_mvp |
tests/unit/test_capsule_listing.py, tests/integration/test_dogfood_mvp.py |
Implemented |
| Inspect capsules | CLI, Python SDK | docs/quickstart.md, docs/concepts/local-artifacts.md |
examples/dogfood_mvp |
tests/unit/test_inspection.py, tests/integration/test_dogfood_mvp.py |
Implemented |
| Replay provider calls locally | CLI, Python SDK | docs/tutorials/openai-responses.md, docs/tutorials/http-capture-replay.md, docs/support-matrix.md |
examples/tutorials/* |
tests/unit/test_replay_runtime.py, tests/integration/test_cli.py |
Implemented |
| Follow actionable CLI next steps | CLI | docs/quickstart.md, docs/scenario-validation.md, README.md |
python scripts/run_scenario.py run no-key-dogfood |
tests/integration/test_cli.py, no-key-dogfood scenario assertions |
Implemented |
| Inspect and compare replay reports | CLI, Python SDK | docs/quickstart.md, docs/tutorials/generate-pytest-regression.md |
examples/dogfood_mvp |
tests/unit/test_inspection.py, tests/integration/test_dogfood_mvp.py |
Implemented |
| Diff replay reports before and after a fix | CLI, Python SDK | docs/quickstart.md, docs/tutorials/generate-pytest-regression.md |
examples/dogfood_mvp |
tests/unit/test_inspection.py, tests/integration/test_cli.py |
Implemented |
| Export replay reports as HTML | CLI, Python SDK | docs/quickstart.md, docs/concepts/local-artifacts.md, docs/mvp-limitations.md |
python scripts/run_scenario.py run html-report-local --keep-workspace |
tests/unit/test_report_html.py, tests/integration/test_cli.py, tests/unit/test_run_scenario.py |
Implemented |
| Export replay report diffs as HTML | CLI, Python SDK | docs/quickstart.md, docs/concepts/local-artifacts.md, docs/mvp-limitations.md |
python scripts/run_scenario.py run html-report-failure-local --keep-workspace |
tests/unit/test_report_html.py, tests/integration/test_cli.py, tests/unit/test_run_scenario.py |
Implemented |
| Export replay reports with React viewer | CLI, Python SDK | docs/tutorials/local-react-viewer.md, docs/quickstart.md, docs/concepts/local-artifacts.md |
python scripts/run_scenario.py run react-viewer-local --keep-workspace |
tests/unit/test_replay_viewer.py, tests/integration/test_cli.py, tests/unit/test_run_scenario.py |
Implemented |
| Export replay report diffs with React viewer | CLI, Python SDK | docs/tutorials/local-react-viewer.md, docs/quickstart.md, docs/concepts/local-artifacts.md |
python scripts/run_scenario.py run react-viewer-local --keep-workspace |
tests/unit/test_replay_viewer.py, tests/integration/test_cli.py, tests/unit/test_run_scenario.py |
Implemented |
| Open replay reports with viewer-first local workflow | CLI, React viewer | docs/tutorials/local-react-viewer.md, docs/quickstart.md, docs/concepts/local-artifacts.md |
python scripts/run_scenario.py run viewer-first-local --keep-workspace |
tests/unit/test_replay_viewer.py, tests/integration/test_cli.py, tests/unit/test_run_scenario.py |
Implemented |
| Run guided local replay workflow | CLI, React viewer | docs/quickstart.md, docs/tutorials/local-react-viewer.md, docs/scenario-validation.md |
replaylab workflow local <capsule_id_or_path> -- <app command> |
tests/integration/test_cli.py, tests/unit/test_run_scenario.py, guided-local-workflow scenario |
Implemented |
| Browse local artifacts in product app | CLI, React product app, FastAPI local backend | docs/quickstart.md, docs/concepts/local-artifacts.md, docs/tutorials/local-react-viewer.md |
replaylab app --local-store-root .replaylab |
tests/unit/test_local_app.py, tests/integration/test_cli.py, local-app-shell scenario, external PydanticAI realism gate |
Implemented |
| Browse product-level captured-run ownership | React product app, FastAPI local backend | README.md, docs/concepts/local-artifacts.md, docs/tutorials/local-react-viewer.md, docs/scenario-validation.md |
GET /api/workspace, GET /api/projects/{project_id}, GET /api/captured-runs/{captured_run_id} |
tests/unit/test_local_app.py, apps/web/src/test/app.test.tsx, local-app-workflow scenario product-artifact ownership assertions |
Implemented |
| Complete the app-owned local loop | React product app, FastAPI local backend | README.md, docs/quickstart.md, docs/tutorials/local-react-viewer.md, docs/scenario-validation.md |
Select a captured run in replaylab app, use the Local workflow hub to inspect trace, run regression replay or live experiment, generate a provider replay guard, export or preview upload safety, and reopen evidence after refresh |
apps/web/src/test/app.test.tsx, local-app-workflow scenario app-owned loop persistence assertions |
Implemented |
| Inspect captured-run LLM and tool I/O | React product app, FastAPI local backend | docs/quickstart.md, docs/concepts/local-artifacts.md, docs/tutorials/local-react-viewer.md, docs/scenario-validation.md |
replaylab app --local-store-root .replaylab, then select a captured run from the run list side inspector or open a full captured-run page |
tests/unit/test_local_app.py, apps/web/src/test/app.test.tsx, local-app-shell scenario trace I/O assertions, external PydanticAI realism gate with split-pane trace explorer and formatted input/output/tool-result previews |
Implemented |
| Run regression replays and live experiments from product app | React product app, FastAPI local backend, run profile/action API | docs/quickstart.md, docs/concepts/local-artifacts.md, docs/tutorials/local-react-viewer.md, docs/scenario-validation.md |
python scripts/run_scenario.py run local-app-workflow --keep-workspace; deterministic live experiment coverage in local app tests |
tests/unit/test_local_app.py, tests/unit/test_run_scenario.py, apps/web/src/test/app.test.tsx, local-app-workflow scenario with run profiles, action endpoints, product captured-run ownership, captured-run regression replay history, persisted live experiment history, experiment labels/hypotheses, applied input/prompt/model/tool/provider-target variant metadata, provider conversion compatibility reports, app-managed setup candidates, artifact-derived providers, no command/provider form in the product path, and external PydanticAI app check |
Implemented |
| Preview upload safety before cloud | React product app, FastAPI local backend, CLI dry-run | README.md, docs/quickstart.md, docs/documentation-coverage.md, PLAN.md |
replaylab app --local-store-root .replaylab; replaylab cloud preview-upload captured_run <captured_run_id> --local-store-root .replaylab --format json |
tests/unit/test_cloud_preflight.py, apps/web/src/test/app.test.tsx, local-app-workflow scenario upload-preview assertions for captured runs, regression replays, and live experiments |
Implemented |
| Upload evidence to Cloud Free (manual + opt-in auto-sync) | React product app, local app backend actions/API, hosted Cloud Free API | README.md, docs/quickstart.md, docs/support-matrix.md, docs/mvp-limitations.md, PLAN.md |
In replaylab app, open Upload preview, click Upload now, and optionally enable Cloud Sync in the app shell |
tests/unit/test_services.py, tests/unit/test_local_app.py, apps/web/src/test/app.test.tsx |
Implemented |
| Understand failed replay story | CLI, local app, React viewer, static HTML | docs/quickstart.md, docs/concepts/local-artifacts.md, docs/tutorials/local-react-viewer.md, docs/scenario-validation.md |
python scripts/run_scenario.py run failure-story-local --keep-workspace |
tests/unit/test_inspection.py, tests/unit/test_local_app.py, tests/unit/test_replay_viewer.py, tests/unit/test_report_html.py, failure-story-local scenario |
Implemented |
| Explain replay reports with AI | CLI, Python SDK | docs/tutorials/ai-assisted-diagnosis.md, docs/mvp-limitations.md |
python scripts/run_scenario.py run ai-diagnosis-loopback --keep-workspace |
tests/unit/test_ai_assistance.py, tests/unit/test_cli_ai.py, tests/unit/test_ai_diagnosis_scenario.py |
Implemented |
| Explain replay report diffs with AI | CLI, Python SDK | docs/tutorials/ai-assisted-diagnosis.md, docs/mvp-limitations.md |
uv run replaylab report diff-explain <baseline_report> <candidate_report> --dry-run |
tests/unit/test_ai_assistance.py, tests/unit/test_cli_ai.py, tests/unit/test_ai_diagnosis_scenario.py |
Implemented |
| Plan instrumentation with AI | CLI, Python SDK | docs/tutorials/ai-assisted-diagnosis.md, docs/scenario-validation.md |
uv run replaylab ai plan-instrumentation --app-root . --providers openai,requests,httpx --dry-run |
tests/unit/test_ai_assistance.py, tests/unit/test_cli_ai.py, tests/unit/test_ai_diagnosis_scenario.py |
Implemented |
| Inspect replay safety preflight | Local app, product API | docs/concepts/replay-safety.md, docs/concepts/tool-resolution.md, docs/support-matrix.md, docs/mvp-limitations.md |
tool-resolution-local, unsupported-effect-local, raw-socket-effect-local, queue-effect-local, unsupported-http-client-local, escape-scope-local, sandbox-escape-local, sandbox-runtime-image-local, sandbox-hardening-local, and sandbox-adversarial-local scenarios; captured-run and report detail pages show provider boundaries, OpenAI model tools, protocol tool requests/results, advisory implementation candidates when source can be inspected, explicit execution-tool wrapper evidence, observed HTTP effect stack attribution, advisory tool effect maps, read-only effect policy proposals, saved project policy review state, opt-in HTTP effect control evidence, opt-in local-effect control evidence, opt-in SQLite database-effect control evidence, opt-in raw-socket network-effect control evidence, opt-in queue/pubsub effect control evidence, opt-in unsupported HTTP client control evidence, hardened local-container sandbox containment evidence, unsupported-effect scope detection including native/FFI and process-escape blockers, safe workflow readiness, and generation availability or blockers |
tests/unit/test_execution_tool_control.py, tests/unit/test_safety_preflight.py, tests/unit/test_sandbox_runtime.py, tests/unit/test_sandbox_image.py, tests/unit/test_unsupported_effects.py, tests/unit/test_unsupported_effect_escape_scope.py, tests/unit/test_unsupported_http_clients.py, tests/unit/test_database_effects.py, tests/unit/test_network_effects.py, tests/unit/test_queue_effects.py, tests/unit/test_tool_resolution.py, tests/unit/test_tool_effect_map.py, tests/unit/test_effect_policy.py, tests/unit/test_http_adapters.py, tests/unit/test_local_effects.py, tests/unit/test_safe_workflow_readiness.py, tests/unit/app/test_server_api.py, apps/web/src/test/app.captured-run-evidence.test.tsx, apps/web/src/test/app.safety-preflight.test.tsx |
Implemented |
| Generate pytest provider replay guards | Local app, CLI, Python SDK | docs/tutorials/generate-pytest-regression.md, docs/concepts/local-artifacts.md, docs/concepts/replay-safety.md, docs/quickstart.md |
Provider replay guard and diagnostic provider replay guard rows below | tests/unit/test_pytest_generator.py, tests/unit/test_local_app.py, tests/unit/test_product_artifacts.py, apps/web/src/test/app.test.tsx, tests/unit/test_cli_generate_test.py, tests/integration/test_cli.py |
Implemented |
| Generate clean provider replay guards | Local app, CLI, Python SDK | docs/tutorials/generate-pytest-regression.md, docs/concepts/local-artifacts.md, docs/concepts/replay-safety.md, docs/quickstart.md |
examples/tutorials/generate_pytest_regression.ipynb, tests/golden/generated_tests/test_support_bot_replay.py, local-app-workflow scenario provider replay guard |
tests/unit/test_pytest_generator.py, tests/unit/test_local_app.py, tests/unit/test_product_artifacts.py, apps/web/src/test/app.test.tsx, tests/unit/test_cli_generate_test.py, tests/integration/test_cli.py |
Implemented |
| Generate diagnostic provider replay guards | Local app, CLI compatibility for final provider failures | docs/tutorials/generate-pytest-regression.md, docs/mvp-limitations.md, docs/concepts/replay-safety.md, docs/quickstart.md |
local-app-workflow scenario diagnostic provider replay guard from a problem replay; generated pytest diagnostic helper |
tests/unit/test_pytest_generator.py, tests/unit/test_local_app.py, tests/unit/test_product_artifacts.py, apps/web/src/test/app.test.tsx, tests/unit/test_cli_generate_test.py, tests/integration/test_cli.py |
Implemented |
| Generate readiness-gated safe workflow regressions | Local app, CLI, Python generator | docs/tutorials/generate-pytest-regression.md, docs/concepts/replay-safety.md, docs/mvp-limitations.md, docs/support-matrix.md, docs/scenario-validation.md |
tool-resolution-local scenario covers the controlled evidence chain and can generate safe workflow regression when the prepared hardened sandbox image is present; sandbox-runtime-image-local builds and checks the default image and proves positive contained replay; sandbox-hardening-local builds a recipe-backed local-path-dependency image and proves hardened sandbox evidence; sandbox-adversarial-local proves bounded sandbox escape probes are contained, refused, or reported as readiness blockers; unsupported-effect-local proves linked SQLite evidence refuses safe workflow generation before DB control, then reviewed/enforced SQLite policy permits generation; raw-socket-effect-local proves direct socket evidence blocks generation in observe mode and becomes an explicit block in enforce mode; queue-effect-local proves enqueue/publish evidence blocks generation in observe mode and becomes an explicit block in enforce mode; unsupported-http-client-local proves unsupported HTTP client evidence blocks generation in observe mode and becomes an explicit block in enforce mode; escape-scope-local proves linked native/FFI and process-escape evidence refuses safe workflow generation while provider replay guard generation remains available; sandbox-escape-local proves sandbox evidence appears and blocks readiness when containment cannot complete; generated pytest installs reviewed effect policy and replays with HTTP/local-effect/database-effect/network-effect/queue-effect/unsupported-HTTP-client enforcement plus sandbox mode enforce |
tests/unit/test_pytest_generator.py, tests/unit/test_cli_generate_test.py, tests/unit/test_sandbox_runtime.py, tests/unit/test_sandbox_image.py, tests/unit/test_safe_workflow_readiness.py, tests/unit/test_unsupported_effects.py, tests/unit/test_unsupported_effect_escape_scope.py, tests/unit/test_unsupported_http_clients.py, tests/unit/test_database_effects.py, tests/unit/test_network_effects.py, tests/unit/test_queue_effects.py, tests/unit/app/test_regression_generation_actions.py, apps/web/src/test/app.generated-actions.test.tsx |
Implemented |
| Wire generated pytest guards into CI | Local app, GitHub Actions, pytest | docs/tutorials/generate-pytest-regression.md, docs/quickstart.md, README.md, docs/mvp-limitations.md |
local-app-workflow scenario writes a minimal GitHub Actions workflow and verifies generated-guard CI readiness |
tests/unit/test_ci_readiness.py, tests/unit/test_product_artifacts.py, apps/web/src/test/app.test.tsx, local-app-workflow scenario CI-readiness assertions |
Implemented |
| Understand support boundaries | MkDocs | docs/support-matrix.md, docs/mvp-limitations.md |
docs/quickstart.md |
tests/unit/test_tutorial_notebooks.py, MkDocs strict build |
Implemented |
| Run replay without app-specific capture guard | Python SDK, CLI replay | docs/concepts/integration-model.md, docs/mvp-limitations.md, docs/tutorials/openai-responses.md, docs/tutorials/http-capture-replay.md |
examples/tutorials/openai_responses_capture_replay.ipynb, examples/tutorials/http_capture_replay.ipynb |
tests/unit/test_capture_lifecycle.py, tests/unit/test_replaylab_init.py, tests/integration/test_same_process_startup.py, tests/unit/test_tutorial_notebooks.py |
Implemented |
| Hard Celery, RQ, and APScheduler adapters | Future SDK adapters | PLAN.md, docs/mvp-limitations.md |
Not yet available | Not yet available | Planned |
| Cloud hosted runners, account auth, billing, and team collaboration | Future SaaS | PLAN.md, docs/mvp-limitations.md, docs/support-matrix.md |
Not yet available | Not yet available | Planned |
Maintenance Rule
When a task changes a user-facing capability, update this matrix in the same change. If a capability is technically implemented but the matrix has no documentation, example, or test coverage, treat the task as incomplete.