How we improved the system using Opik
We use Opik by Comet for observability, evaluation, and systematic improvement of our focus coach. Here’s how it works.
Observability
Every LLM call in the app is traced to Opik. When you talk to the Coach or generate a Weekly report, we create a trace with the user message, the model request, and the reply. Each OpenRouter call (including tool-use rounds in the coach) is logged as a child span.
Evaluation
We defined evaluation datasets and run evaluations that log to Opik. For the Coach, we have 8 test cases: general advice, data-specific questions (e.g. “How did I do this week?”), and cases where we give no data so the coach should ask to clarify. For the Weekly summary, we have 3 synthetic weeks. We score outputs with AnswerRelevance: does the answer address the question?
Systematic improvement
We use Opik evaluation to show before vs after. We run the same coach eval with two prompts: a deliberately weak “regression” prompt (generic assistant, no focus rules) and our real focus-agent prompt. The regression run gets lower AnswerRelevance scores; the baseline run gets higher scores. In Opik we compare the two experiments to see the improvement.
Opik is by Comet. We use it for the hackathon to demonstrate excellent observability, evaluation, and data-driven improvement of our focus app.