Chat Windows Don't Teach. So I Built an AI That Checks.

For most of a year I "learned" with a chat window. Explain the KV cache to me. Explain radix attention. Now explain it simpler. Every conversation felt like progress. Every one of them ended the same way. A week later I could not explain the thing to a colleague without opening the chat again.

That is not a memory problem. It is a design problem, and it turns out someone measured it.

Two trials, opposite results

In 2025 two groups ran randomised trials on one question. Does an AI help you learn? They got opposite answers, and the difference was not the model.

Kestin and colleagues at Harvard gave 194 physics students an AI tutor built around known teaching practices. It explained one step at a time, asked for an answer before it gave one, and kept the student's cognitive load in view. Median learning gains were more than double those of the same students in an active-learning class, in less time.

Bastani and colleagues gave nearly 1,000 high-school students in Turkey a ChatGPT-style interface during maths practice. Practice scores went up 48%. On the unassisted exam that followed, the same students scored 17% lower than classmates who had no AI at all. The students asked for solutions and copied them. The researchers checked, and the AI's own mistakes were not the cause.

Both trials used the same class of model. The design decided which result the students got. I had been running the Bastani experiment on myself.

I wrote up both trials, with the sources, in the guide on how to learn with AI, properly. The short version is four rules.

The four rules

1. The AI asks what you know before it teaches, and records the answers.
2. It teaches in prerequisite order, one smallest step at a time.
3. It tests you the same day you read, and again the next day, after sleep.
4. It turns only what passed into spaced-repetition cards, and reads the review history.

One prohibition holds under all four rules. During practice the AI gives hints and grades, never the answer.

None of this is exotic. It is what a good tutor does. A chat window does none of it, because a chat window forgets what you know and gives you what you ask for. Both properties are perfect for getting work done and fatal for learning.

What I built

The Learning Harness is an AI agent that follows the four rules. It runs in a terminal, and I am building a Mac app around it.

It starts by asking what I already understand and writes that down, in a database, not in a chat that scrolls away. From that record it lays out a chain. Every topic sits on one ordered path, and the next topic is the smallest step from what I already hold. The harness never teaches a topic before its prerequisites.

Each lesson is a page written for one learner. It uses the terminology of the field, has a diagram in every part, and works one example all the way through. The harness reviews and corrects the page before I read it.

Then comes the part a chat window never does. The same day, the harness gives me a quiz. On a later day, after a night of sleep, it asks for an explain-back. I explain the concept in my own words and the harness grades whether I hold it. If a concept fails either check, the harness teaches it again and tests it again. Only a concept that passed becomes an Anki card, and the harness reads my review history back, so the numbers measure retention instead of my confidence. The details are on the method page, and there is a separate guide on spaced repetition with AI if that is the part you care about.

What a day looks like

A day is thirty to forty-five minutes. It is one lesson, or one explain-back, or a review session, in whatever order the harness says is next. The start screen shows three streaks: sessions, reviews, lessons. There is no streak for the checks, because a check happens when it is due and not when I feel like it.

As of this week the harness has written 13 lessons for me across transformers, inference and a few non-technical topics. It has tested 32 concepts. I can explain 21 of them in my own words, which is the only number I trust. My Anki review streak is 32 days.

In a year of chat-window learning I could not have told you how many concepts I held, because nothing was checking.

What it is not

It is not a chatbot with a syllabus. If I ask it for the answer during a quiz, I get a hint. If I read a lesson and click "done", nothing happens until the quiz. It gives no credit for reading. That is the point.

It is also not finished. A handful of people are testing it with me. The guides on learningharness.org are public now, including one on upskilling in a technical field while working, which is the situation most people I talk to are in. If you want to be in the next group of testers, write to hello@learningharness.org or tell me on X.

The takeaway

Learning with AI works when the AI verifies you. The moment it hands you answers on request, it is answering you and not teaching you, and the exam a week later will show the difference. I stopped asking for answers. I built something that asks me instead.