At the final course call, one of the students thanked me not for the lectures and not for the homework. For the log reviews: “That’s something I would never have thought of myself, and it’s a really useful tool.” That was the moment I realized a side experiment in reviewing process had grown into a product. Here is the story of how feedback from the first cohort turned into RetroPair, an open source plugin I released on GitHub this week.
Where it started: “homework solves itself in one prompt”
I finished the first cohort of the AI-Driven Development course with an uneasy aftertaste, and wrote a full retrospective on my Telegram channel at the time. Students independently told me the same thing: the assignments were spelled out in such detail that they solved in a single prompt. You get the answer, submit it, move on. I was grading the result, while the students were learning something other than what actually needed training.
That is also where I formulated the key takeaway from the first cohort: developer problems with AI are not tool problems. They are the absence of a personal working system. Coding takes 20-30% of the time; everything else is framing the task, planning, verifying the result, deciding what to delegate to the agent and what to keep. Nobody trains those 70-80%, because the finished code only shows the outcome - the path to it stays offscreen.
And where is that path visible? In session logs. Claude Code, Codex, Cursor - they all write transcripts of the dialogue between human and agent. That is ready-made material: how a person frames a task, reacts to errors, verifies results or takes them at face value, recovers after a failed attempt. So I decided: in the second cohort, I would review not just the homework but the process.
How do you review hundreds of sessions
From the second cohort on, students submitted homework together with logs from their coding agents: Claude Code, Codex, Cursor. Two parallel cohorts, Level 1 and Level 2. The scale became tangible fast: one student accumulated over five hundred log files over the course, another had fifty-one sessions in the final module alone. Reading that by hand does not scale - and it should not.
The process grew organically, as the course went on. First came simple skills: collect the logs, filter out noise, find the right files in a student’s repository. Then sanitization - keys, tokens and personal data get scrubbed from the logs before any agent ever sees them. Then a two-stage review: collector agents read sessions in batches and bring back verbatim quotes with addresses, and a separate judge agent evaluates the work against eight criteria. Every verdict must stand on a quote from the log; no quote, no score. At the end of the chain sits a critic configured as an experienced developer reading the feedback draft, deciding whether you would apply it tomorrow or throw it out.
What this scoring is for. It describes not the agent’s code but the human’s process: how deliberately they frame tasks, verify results, calibrate trust and leave escape routes. Each criterion runs on a 0 to 4 scale with described anchors, so a “3” means the same thing for different people across different modules. The criteria:
- Task framing. How explicitly a person sets the context, boundaries and definition of done before the agent starts working.
- Result verification. Whether the agent’s answer is taken at face value or checked against an observable run and the person’s own criteria.
- Trust calibration. Whether a person tells apart what can be delegated wholesale from where their own acceptance threshold and control are needed.
- Context management. How the agent’s window is managed: what to feed as input, what to pin in a file, when to open a fresh session.
- Work decomposition. Whether the task is cut into steps with checkpoints or handed to the agent as one chunk.
- Owning decisions. Whether choices between even options stay with the human, or the agent’s recommendations get silently accepted.
- Changing approach after failure. Whether the lever changes after a repeated failure, instead of repeating the same action hoping for different behavior.
- Reversibility. Whether there are return points: commits, branches, isolated experiments before risky runs.
The point is not a single score but the trajectory: you can see what a person is training and what is lagging.
A separate mechanic is recommendations. After every review the student got concrete suggestions, and in the next module the agents checked whether they had applied them, issuing a verdict: heard, partially, not visible. The result was not a one-off grade but a trajectory.
What it looked like on a live example
In one review, the judge found a pattern: the agent asked the student twice, explicitly, what to do about the keys found in the code, and warned that this should be decided before publishing the branch. The person replied about something else in both places and published the branch - with the question still hanging. The review produced a recommendation: close any explicit agent question and any “this should be decided before…” with a separate reply before publishing - either a decision or a named reason to leave it as is. A module later, the same review pipeline looked at the new sessions with exactly that question in mind: heard or not. That is a loop no static analyzer has - advice does not evaporate, it gets checked on the next cycle.
What the students said
At the final calls of both cohorts, log reviews were named one of the most useful parts of the course. Interestingly, they turned out useful even for people who had been working with coding agents for about two years: there were plenty of insights they had never thought about mid-work. Quoting from memory, no names.
A tech lead: “I really enjoyed the collection of my sessions. That’s something I would never have thought of myself. It was fascinating to see how I use AI from the outside, with scoring.”
A developer who received three such reviews: “The most useful part was the list of mistakes at the top of the report and the recommendations for what to do when the agent gets stuck in a loop. I tried to recall what my problems were and dig into them.”
Another student, about the personal report: “When you read it and understand what to pay attention to and where to work better - it’s not a black box anymore.”
And the most telling comment came from an engineer building his own development pipeline with reviewers, critics and coders: he said he wants to embed this kind of review into his pipeline, so that agents review the work of agents on a permanent basis. And a tech lead at the second final call looked at the group report’s radar chart across eight axes and suggested scaling it from a study group to a whole company: measuring how teams actually adopt working with AI, instead of reporting installed license counts.
The RetroPair plugin
By the end of the course I had a working process: skills, agents, scripts, a sanitizer, HTML reports. All of it existed as homemade wiring around the students’ repository structure. I packaged it into the RetroPair plugin for Claude Code.
What it does:
| Feature | What you get |
|---|---|
| Log collection and sanitization | Logs are collected locally; keys and personal data are scrubbed before analysis |
| Two-stage review | Collectors bring quotes, the judge scores eight criteria with evidence links |
| Tracked recommendations | Advice is not forgotten: the next review checks whether you applied it |
| Three report types | Personal - for yourself, group - for a team with dynamics across criteria, reviewer’s - for whoever does the checking |
| Privacy | Everything runs locally, logs are never sent anywhere |
The plugin analyzes session logs from any environment - the analysis itself is not tied to an IDE; I tested it with Claude Code, it should work in Codex too, but I have not tested it there yet and will keep polishing. Get it here: https://github.com/Serg1kk/retropair, demo site - https://www.retropair.dev
What I learned
I built this tool for myself: to solve a teacher’s problem of reviewing hundreds of sessions without drowning. Classic build for yourself. But it turned into a product beyond myself - because the pain turned out to be shared. A tech lead who wants to understand how their team actually works with agents. An engineer who wants to grow and see their own dynamics. A company adopting AI that cannot tell real usage from a checkbox in a report.
The second takeaway is about the choice that appears after any project like this. The tool works, the feedback is good, and you have two paths: let it gather dust on a shelf, or give it to the community. I chose the second, partly to finally learn what living in open source feels like. I joke that my project will get popular with ten stars. Or eleven.
Try it if you lead a team or just want to see your own work with an agent from the outside. Feedback is welcome - I keep polishing.
And a personal note to close. This was my last cohort in teaching for an indefinite while: two courses devoured the time and energy my main projects need. The courses live on: Level 1 and Level 2 are now taught by two other great instructors. Thanks to all the students of both cohorts: for two months you submitted your logs and answered questions honestly - and ended up with a tool anyone can now use for free.
FAQ
Where do my logs go?
Nowhere. The plugin runs locally: logs are collected and analyzed on your machine, inside your Claude Code. The sanitizer scrubs keys and personal data before the files even enter analysis. Today that is a deliberate principle: real development sessions are too sensitive a material to upload to someone else’s server.
Does it work without Claude Code?
The log analysis is not tied to an IDE: the plugin understands session logs from different environments, including SpecStory exports and terminal scrollback dumps. The plugin itself installs into Claude Code; it should work in Codex too, but I have not tested it there yet - still polishing.
How is this different from just reading my own logs?
You do not read - you get a review with an evidence base: every criterion score stands on a verbatim quote with the session and turn referenced. Plus dynamics: one review is a snapshot, several reviews in a row are a trajectory showing whether task framing is improving and whether verifying results has become a habit. And recommendations with verdicts: the next run checks whether you applied the previous advice.
Can I use this with a team?
Yes, that is what the group report is for: a “people × eight criteria” table, distributions and group dynamics across periods. You define the periods yourself - a week, a month, a sprint. At the course’s final call, a tech lead saw it as a ready-made instrument for measuring how a team is actually adopting agent work, instead of license-count reports.