Quality · Implemented
Evaluation, observability, and recovery
How the system is measured, debugged, and changed without uncontrolled policy updates.
Evaluation system
Versioned datasets cover retrieval, concept extraction, graph traversal, diagnosis, mastery, review, tutoring, tool selection, workflow routing, checkpoints, resilience, security, and integration boundaries.
- Retrieval: Recall@K, MRR, nDCG, context precision, scope correctness, and insufficiency accuracy.
- Diagnosis: route, gap type, mastery state, next action, grounding, and repeatability.
- Tutoring: one-question rule, premature-answer rate, policy compliance, and completion routing.
- Runtime: checkpoint completeness, interruption/resume, retry correctness, and idempotency.
Controlled improvement cycleReviewed failures become regression cases. Changes are promoted only after the frozen suite passes.
Observability
Operational events and spans record workflow versions, node transitions, prompt versions, tool calls, evidence IDs, retries, latency, validation outcomes, and safe failure categories. Credentials, raw provider errors, full source documents, and hidden reasoning are excluded.
Failure recovery
- Public pages remain available when private infrastructure is unavailable.
- Vector and graph projections can be rebuilt from Postgres.
- Checkpointed workflows resume from the latest valid state.
- Optional notebook panels fail independently.
- Recoverable input is preserved after API failure.