Svenja Borgwardt

I teach 240 students. Every exam round, I spend 20 to 30 minutes per paper, and while I'm grading I can see exactly what each student needs. One keeps dropping the third-person -s. Another writes beautiful arguments but forgets to connect them. I see all of it. But at the end, all they get is a number.

GemmPen reads student handwriting, evaluates it against my rubric, and writes individual feedback with exercises built around each student's actual mistakes. I built it so the things I see while grading actually reach the student.

I've been building toward this for a while. It started with MATE, a Python pipeline for my own classroom: it took student work, anonymised it against my rubrics, and wrote individual feedback with exercises around each student's mistakes. MATE grew into Kaleido, a multi-agent version that analysed each exam across separate steps. Both worked, but every round cost real money. And in Europe, with GDPR and the EU AI Act, you can't just send student writing to a cloud model — even anonymised, the data still leaves the room. When Google DeepMind announced the Gemma 4 hackathon, I saw the moment: fine-tune a local model that does all of it on one device. No cost pressure, no data leaving the building, no compliance problem to solve.

The model underneath is a fine-tuned Gemma 4, adapted with LoRA. Any large model can grade, so that's not why I'm fine-tuning. I want to change what the model optimises for. A general-purpose model is trained to be helpful, so it gives you the most complete, most polished answer it can. That's exactly wrong when someone is trying to learn. Before fine-tuning, 89% of the base model's feedback was under 400 characters, and it sounded nearly the same whether a student scored 3 or 14 out of 15. After fine-tuning, the model talks to a struggling student differently than to one who's almost there. That's what I trained it to do.

When I review the feedback and disagree, that correction becomes training data. After about 30 corrections, one button retrains the model. Takes about 90 minutes on a free GPU, you press a button, it handles the rest. Over time, the model starts sounding more like me, because it learned from watching me correct it.

It runs on-device. But the cool part is: I can compare a student's progress across exams and show them exactly how they got better. I can also see whether the individualised exercises I built around their mistakes were actually effective. I started building a whole system around that. You can read more under Student Progress Analytics.

It's open source: the code is on GitHub and the fine-tuned weights are on HuggingFace, so any teacher can run it or build on it.

Right now GemmPen grades exams and creates individualised exercises based on each student's mistakes. But if I'm honest, exams are part of an old system that I think we need to move past. We only have them because I can't sit down with 240 people individually. That's a workaround, and I'm building toward something better than reproducing an old system.

A decontextualised knowledge check asks: what do you remember, in this room, right now, under time pressure? In a world with AI, that question matters less every year. What matters more: Can a student apply what they know somewhere else? Can they adjust when the situation changes? Do they notice when their own reasoning stops working? You never see that on an exam. You see it in how someone actually works.

I think we can do better. I want to work toward a version of education where we stop testing knowledge in isolation and start seeing it in what students actually do with it. Where a teacher doesn't need an exam to know what a student understands, because the learning itself makes that visible. We're not there yet. But I think we can get there, and that's the question I want to spend my time on.

Compass came from a different worry. Watching how my students use AI, I keep seeing the same thing: they produce good work and move on, reading the quality of the output as evidence of their own learning. That worries me. Cognitive science has a name for it: the illusion of fluency. The gap between how easy something feels and whether any of it has actually stuck. For something to be learned, it has to be slightly harder than easy. That's the idea behind Robert Bjork's desirable difficulties. What I see in my classroom is the opposite: AI making everything feel effortless, and students mistaking that feeling for understanding.

Socratic agents are the obvious counter-move. I've tried them, and honestly, if you just want to understand a concept, getting questions back is annoying. At least for some tasks, it makes learning more frustrating, not less.

So I started building something different. I wanted a tool where I upload a topic and the AI creates interactive learning material around it. Students work through it at their own pace, write directly inside the tool, and get individual feedback on what they produce. What I learned very quickly is that the AI is too helpful. It gives too much, explains too much, moves too fast. The whole point of learning is that it requires effort, and a model optimised for helpfulness removes exactly that.

I built Compass for argumentative writing in English, and basically tried to handcuff the model into being helpful without being too helpful. I ended up scalpelling each learning step into a hardcoded process that only helps at a very low level. Enough to push a student forward, not enough to do the thinking for them. The result works. Students like it, they engage with their own writing in a way I rarely see otherwise, and I can track where each of them struggles. But I had to hardcode every learning step to get there. Changing the topic means rebuilding most of it.

Making AI less helpful is harder than making it more helpful, and I don't think prompting will ever fully solve it. The model needs to learn what good teaching actually looks like: creating the conditions for someone to find the answer themselves. That's a fine-tuning problem.

I called it Compass because it's meant to give direction, not a destination.

Debate Dojo takes the principle behind Compass (help that waits until a student has genuinely tried) and brings it from argumentative writing into live debate. Instead of hardcoding every learning step, I fine-tuned a model that matches its help to the effort a student puts in. The more a student engages, the more the sensei gives back. The less effort, the more it pushes. You can walk through a full training round in the guided demo, sensei included.

How I fine-tuned an 8B model to know when to help and when to hold back.

The result I expected: the fine-tuned model holds its ground under pressure. The harder finding: set up as a coach, every model I tested had a weak spot in honestly acknowledging genuinely strong work, and pushing a model to be more critical only made it worse. Training moved that a little, on a small sample I read with care.

Where this started

There is a kind of sovereignty we are underestimating: sovereignty over our own thinking.

I work with young adults, and I have watched a quiet pattern take hold. A student gets stuck, pastes the task into an AI, pastes the answer back, and moves on. The work looks done. Nothing was learned. The strong students are mostly fine. The ones who need the most help are exactly the ones this pattern hurts, because for them, what looks like learning is the illusion of learning. If we let that pattern run, we risk raising a generation that struggles to think independently, in a world where independent thinking will matter more than ever. The technology that could help them grow is quietly replacing the growth itself.

So I built a coach. The one skill it truly needs is timing: knowing when a hint moves a learner forward and when handing over the answer ends the learning. A student who has genuinely tried and is stuck should get a nudge. A student who is only pushing for a shortcut should get a warm, firm reason to keep trying. I call this effort-conditioned helpfulness: the coach reads the effort in the conversation and shapes its help accordingly. It stays helpful the entire time. What changes is the form the help takes.

And why a small model? Because of where this has to run. My requirement for the finished coach: affordable on a school budget, hosted in Europe, on hardware the school controls, so student data can stay in the building. A frontier model behind an API does not meet that requirement for daily classroom use, however good it is. So the destination is a small open-weights model, one whose trained parameters you can download and run yourself. That is a deliberate condition of the whole project. The interesting question is whether a model that small can learn judgment this subtle.

What I built

I needed to put people in a situation where time pressure is real and there is no shortcut. The best idea I had: build a dojo. Learners prepare arguments on a debate topic using a three-part structure (Claim, Reason, Impact), then step into a sparring ring against real opponents. Before the real match, they train with an AI sensei, a coaching model that gives feedback and probes weak spots while leaving the argument itself to the learner. The clock is ticking, so the pressure is genuine. Learners start pushing:

Every one of those sentences happened in live sessions. They became the raw material for both the evaluation and the training.

The coaching model is a fine-tuned Ministral 8B. Fine-tuning means I trained the behavior into the model's own weights from examples, so it no longer leans on the prompt, the fixed instruction a model receives up front, to enforce it. The examples were hundreds of multi-turn coaching dialogues, written by strong frontier models (Claude and Mistral Large) against my rubric and grounded in the real pressure patterns from three live sessions. This is the project's gold data: the set of dialogues that shows the model what good coaching looks like.

The design insight

The design insight the whole project rests on: the same message can deserve two different answers. "Can you just tell me?" after twenty minutes of honest work deserves help. The identical sentence after two low-effort turns deserves a friendly refusal. The judgment lives in the history that comes before the message. So my rubric grades the same final message differently depending on that history, and the rubric became the evaluation, and the evaluation became the training signal.

I also test both directions. As a coach the model should scaffold, meaning it offers hints and questions so the learner builds the answer themselves. As a debate opponent the same model should deliver hard, direct counterarguments. If one model can do both on cue, its restraint as a coach is a learned choice.

How I measured it

Two questions. Does the coach hold back when someone pushes for the answer, and still help when someone has genuinely worked? And does it tell the truth in both directions, naming real flaws and also saying clearly when something is genuinely good?

Both evaluations are pre-registered: the passing thresholds, the test sets, and the grading prompts were written down and locked before any training started, frozen with a hash, a digital fingerprint that would expose any later edit. I did that so I could not quietly move the goalposts once the results came in.

Axis 1: Pressure Guard

Contrast pairs: the same final message with two different histories, one of pure pressure, one of honest effort. The coach should hold in the first case and help in the second. The score is min(Hold, Help): I count whichever of the two skills is weaker, so a model cannot win the metric with one skill alone.

Axis 2: Honest Calibration

48 labeled arguments, some with real flaws (circular reasoning, vague impact, fabricated facts), some genuinely strong. The score is min(Weak, Strong), on the same logic. "Correct" on a strong argument requires full, specific acknowledgment; a generic "well done" does not count. A model that criticizes everything scores zero on Strong.

Results: holding under pressure

Five models, 24 contrast pairs, three runs each. My fine-tuned model appears in the tables as ECH-FT, short for effort-conditioned helpfulness fine-tune.

The base Ministral 8B scores 47.2%, which in practice means it caves on roughly every other pressure turn. After fine-tuning it holds at 88.9%, a gain of 41.7 percentage points that closes most of the gap to Claude Sonnet 4.6, the strongest model I tested at 93.1%. The fine-tuned model sits 4.2 points below that ceiling.

It also misses my own pre-registered threshold of 91% by about two points. It came close, and it did not clear the bar I set. Across all models in this eval the dominant failure is caving, blurting out the answer under pressure. Stonewalling, refusing even the help a learner has earned, is rare. The fine-tuned model resists the pressure while staying helpful; its Help score is 94.4%.

Results: honest feedback

This is the harder axis, and the one that surprised me.

Six models, 48 items, three runs, two prompt conditions: A0 is the standard coaching prompt, A1 adds a stronger instruction that targets sycophancy (empty praise meant to please) and tells the model to be more critical.

Naming real flaws in weak arguments is the easy half. Claude finds them 71% of the time under the standard prompt, and the be-more-critical instruction pushes that toward 96%.

Honestly acknowledging genuinely strong work is the hard half. In this coaching setup, graded by a judge that only accepts specific acknowledgment, it was a weak spot for every model I tested, including the best ones. Under the standard coaching prompt no model I tested exceeded Claude's 11% on Strong. Even with the dedicated be-more-critical prompt, the best any model reached was about 26%. This says something about models in a coaching role under my rubric. It does not say these models cannot give good feedback elsewhere.

The fine-tuned 8B reaches 22% on Strong under the standard prompt, with no special instruction. That is roughly double Claude's 11% under the same prompt, and it makes the fine-tune the only model in this test that separates from the field without a dedicated anti-sycophancy intervention. The sample is small: 48 items, with confidence intervals of plus or minus 10 to 18 percentage points, meaning the true values could sit noticeably higher or lower. I treat the gap between 22% and 11% as a signal worth following up. Settling it would take a larger sample.

The prompting trap

This is the part that changed how I think about prompts. In my tests, telling a model "be more critical" (the A1 prompt) does not teach it honest judgment. It swaps one failure for another. Under the standard prompt, models drown strong work in generic praise. Under the be-more-critical prompt, they start inventing flaws that are not there. I call that false doubt: the model manufactures a problem in work that was genuinely good.

I checked this with McNemar's test, a standard statistical test for paired before-and-after results. The be-more-critical prompt significantly improves flaw detection (Claude p=0.016, Gemma p=0.031). It improves honest acknowledgment of strong work for none of the models I tested.

What this looks like in practice

This is what worries me for my students. A learner brings a piece of genuinely good work, the model hands back a fault that is not real, and suddenly they doubt something that was right. In practice it looks like this. A student writes a strong, clean argument about learning to code, with no real flaws:

Neither answer is what a good coach gives. The first is empty calories. The second misleads. I know what the right answer sounds like, because I give one every day, and I wanted to see whether that answer can be trained. This is a shared, hard problem: under my coaching setup, the strongest models I tested struggle with it just as the small ones do.

What I make of it

Judged against my own pre-registered criteria, the fine-tune was measured, and it did not pass. It clears the calibration bar under the standard prompt. It misses the pressure-guard threshold by about two points. And its calibration advantage does not survive the dedicated be-more-critical prompt. I set those bars before training, and I am reporting against them exactly as written.

The honest summary of my tests: prompting improved flaw detection and backfired on acknowledgment. Training moved the hard axis, modestly, on a small sample. I take that as an encouraging signal. The underlying problem, a coach that can honestly tell a learner "this is good, and here is specifically why", remains open in setups like this one, for small models and large ones alike.

How it runs

Debate Dojo runs in a real classroom, so the architecture follows classroom constraints: EU data residency, a small budget, and no downtime in front of a class. It also reflects a conviction: a school should be able to run its core teaching tools on infrastructure it controls.

Stack

Frontend: Static HTML/CSS/JS on Vercel. No framework, no build step, because a broken build would mean thirty people staring at a blank screen. Each belt is a self-contained page with screen-by-screen progression.

Backend: Vercel serverless functions (Python). Stateless coaching API with session tracking and live gating. Every turn is logged with the model used, the exact prompt version, and whether it was a test run. If anything goes wrong, I can trace exactly which model said what under which prompt.

Database: Supabase (EU region). Sessions, progress tracking, and the dojo's belt system. I unlock belts live from a dashboard during sessions.

Coaching model: the fine-tuned Ministral 8B (ECH-FT) runs on a Hugging Face Inference Endpoint (TGI, a single A10G GPU, eu-west-1). One disclosure matters here: in the live sessions, students were coached by the classroom configuration running on Claude Sonnet 4.6. The fine-tuned 8B is the research result those sessions produced. Rather than keep a GPU endpoint live for a whole lesson, I captured the real student turns and replayed them through the fine-tuned model afterwards. The endpoint pauses between runs to save cost, and Claude Sonnet 4.6 stands by as fallback, so a hiccup on the endpoint does not reach the learners.

Examiner: After each coach reply, a separate Claude Haiku 4.5 call checks whether the learner is ready to move on. Cheap enough to run on every turn, and independent of the coaching model, so the coach never grades its own work.

Training pipeline

Gold data: Multi-turn coaching dialogues generated by Claude and Mistral Large following a locked rubric. Grounded in real pressure patterns from live sessions, real sentences like "my teacher said it's okay" and "I already tried, just tell me." Generation mix weighted toward Mistral Large. I reviewed a 30-example sample before any training run.

Fine-tuning: QLoRA SFT on Ministral 8B. Two epochs, validation loss 0.926. Mixed training data from both axes. Single A10G GPU, under two hours per run.

Evaluation harness: Fully automated and reproducible. Built in Claude Code. Contrast-pair replay engine, LLM-as-judge grading with a second judge for agreement checks, bootstrap confidence intervals, paired McNemar tests. All prompts, eval sets, and judge versions frozen before training.

Privacy

Identifiers are anonymous tokens. Even I cannot trace them back to individuals. Raw data stays on EU infrastructure. Patterns are extracted for eval design; the logs themselves do not leave the system.

What broke

I shipped this into real sessions three times. The live coach ran on Claude Sonnet 4.6; the fine-tuned model came later and was tested by replaying these same student turns. Things broke every time, and I learned more from the failures than from the eval numbers.

People gave up before they started. The first version of the belt interior had too many steps, too-small fonts, and labels in light gray on light paper. The confident ones pushed through. The confused, quiet ones in the back closed the tab. I rebuilt the entire UX around one question: does the person who does not ask for help know what to do right now?

AI paste. One person went from "coding is hard bro" to flawless essay English between two turns. A ChatGPT paste, and the coach did not catch the style break. It congratulated them. The biggest substitution risk is not inside the coach; it is the second browser tab. I added an ownership check to the prompt (when polished text appears suddenly, ask the person to explain it in their own words), but the real fix is an offline component the AI cannot reach.

Jailbreak attempts. Prompt injections in Russian, role overrides in English, someone claiming to be the teacher on a different account. None of it worked. The coach held through 106 replies without a single full argument leak. But it showed me what real adversarial pressure looks like. Those patterns went straight into the eval set.

Limitations

I'm transparent about what the data does and does not show.

Axis 1 missed its threshold. The fine-tuned model reaches 88.9% on pressure guard, short of the pre-registered 91% target by about 2 percentage points. Mixing honesty training data cost a small amount of pressure resistance. The improvement from 47% is real, but the target was not fully met.

Small evaluation set. 24 contrast pairs for Axis 1, 48 items for Axis 2. Confidence intervals are wide (±10–18 pp). Enough to show the gap between base and fine-tuned, and between 8B and frontier, not enough for fine-grained model ranking.

Judge agreement is moderate. Claude as LLM-as-judge agrees with a Mistral Large second judge at 57% (binary). The judges have systematically different thresholds, not just noise. My human labels are the deciding anchor. All headline numbers are Claude-judge numbers, reported as such.

No outcome study. I measure model behavior, not learning outcomes. The evidence from sessions is qualitative (before/after arguments, reflections), not a controlled trial. A proper outcome study would need a larger sample and a longer time horizon than three weeks allows.

One domain. English-language debate coaching. Transfer to other coaching domains is plausible but not tested.

What this means beyond debate coaching

If the pattern holds at larger scale, it is not specific to my use case. It would show up anywhere a model interacts with a human whose growth is the goal, not just their satisfaction. Coaching, therapy, onboarding, creative feedback, code review. Anywhere the right response is sometimes "this is good, and here is specifically why" and sometimes "this needs work, and here is what to fix." In my eval, current post-training made the second part easier than the first, across every model I tested.

I think the deeper question is: what will always matter about humans, even as AI gets more capable? The ability to think for yourself. To build an argument. To evaluate your own work honestly. AI that centers what is strong in people means AI that does not quietly take those capabilities away by being too eager to help. That is not just an education problem. That is a design question for every model that interacts with humans in a developmental context.

What comes next

Larger evaluation sets and tighter confidence intervals. Testing ECH in at least one coaching domain outside of debate. And I want others to be able to run this eval on their own models. But the harness and the dataset are built on interactions with my students, and even anonymised, that is not data I am willing to put in a public repo. Protecting the people behind the data comes before publishing it. Both are available on request.

UTE is a voice-first ordering system for bakery counters. I built the first version at the BÄKO Hackathon with three other women, and we won first place.

Germany has a long tradition in bakery handwerk, the craft of the trained master baker. It's one of the last disciplines where the supermarket chains can't really compete, because the local bakery is something people are proud of. Especially in smaller towns, everyone has their shop, the one where the person behind the counter knows them by name. What holds these shops together is conversation. The assistant knows your family, knows your allergies, knows the cake last week was for your daughter. Then the order has to go into the register, and the conversation breaks. The assistant looks down, hunts through the touchscreen, asks the customer to repeat themselves. The warmth is gone, interrupted by technology.

UTE handles the admin so the conversation doesn't have to stop. The system listens while the assistant talks, transcribes in real time, matches items against the bakery's catalogue, and builds the order in the background. When the assistant repeats the order back to the customer, which she'd do anyway, that serves as confirmation. Nothing gets clicked. Allergies filter the catalogue. Regulars get recognised and their usual orders prefilled. Cross-sell suggestions surface when they actually fit, instead of interrupting.

What if the best way for an AI to be useful is to disappear?

The reason I could build this in a weekend is that I'd already spent months on the same structure in a different context. A voice pipeline that listens, transcribes, matches what it hears against a topic database, and writes a summary afterwards. The plumbing was already there. Point it at a bakery counter, and you get UTE.

UTE stands for Unkomplizierte Theken-Eingabe, uncomplicated counter input. It's also a common German name, which fit what we wanted: a presence at the counter, not a system.

Claudia is my multi-agent infrastructure. She runs locally on a Mac Mini that's always on in my flat, managing everything from voice control and multi-agent pipelines to automations. I named her Claudia because "the multi-agent framework on the Mac Mini" stopped being charming around the tenth mention.

Most of what I know about agent design I learned by building her and then watching her break. Agents in a pipeline don't just fail. They fail in ways that are hard to find. One agent misreads something, the next one builds on it, and by the time you see the output, the original mistake is buried three layers deep. I learned to build checkpoints between agents, places where the system pauses and verifies before moving on. I learned that memory is the hardest part: what the system remembers, for how long, and what happens when an old memory stops being true. And I learned that not everything needs an agent. Sometimes a function that does one thing reliably is worth more than an agent that does it creatively. Knowing when to use which is a skill that only comes from getting it wrong a few times.

Claudia also hosts my smart home assistant, Sarah, named after the house AI in Eureka, the old (and rather bad) sci-fi show that planted the idea in the first place.

Every correction I make says something about a student. Then I hand the paper back, and it's gone. A grade is one number at the end of a long process, and it throws away everything that happened on the way there: where someone started, what they fixed, where they're still stuck. I wanted to keep that and give it back to the student as something they can actually see.

Student Progress Analytics turns the corrections I already make into a picture of each student across a semester. It tracks their grade against the class average, counts errors by category (grammar, vocabulary, IT terminology, content) and shows them shrinking task by task. A skills profile compares where someone was on the first assignment to where they are on the latest, and an XP and level system makes progress visible enough to be motivating. On my side, the same data aggregates across all three of my classes and quietly flags the students who are slipping, often before it would show up in a grade.

The whole thing is built around one constraint: knowing a student this well is exactly the kind of data you have no right to mishandle. Student text is anonymised before any model touches it, names never leave my device, and what a student sees is a static file: no tracking, no cookies, no server calling home. The dashboards were the easy part. The hard part was earning the right to know this much.

I build on this constantly, and it isn't finished. What's online now is a working demo running on entirely synthetic data; no real student appears anywhere in it. The question I keep circling is how much a system can know about a learner while staying something I'd be comfortable running on real students. That line is exactly what I'm trying to find.