Engineering the Harness

Last time I wrote about building my agents a wiki: an index that stays in context, a curator agent that turns my raw notes into something worth reading, and pages that stay out of the way until the agent needs them. Instructions that load on demand, same shape as Skills, except bigger and properly organised: cross-referenced, and not just processes. Deep knowledge. A glossary, so a term means the same thing on every page that uses it.

That post ended with the thing built. I’ve been running it for a while now and it’s genuinely awesome, and it keeps getting better.

I’ve borrowed the word harness for all this, and it’s stuck. The wiki is the knowledge. The harness is everything around it: the agents that curate it, the agent that reviews it, the rules about which page gets opened when, and the intern I imagine reading it every morning with no memory of yesterday. The wiki was the part I set out to build; the harness is what I’ve been working on ever since.

Where It Got To On Its Own

I started with one agent that curates: raw notes in, clean pages out. Then I added a second that reviews the wiki itself, hunting contradictions and gaps in what it knows.

That was already a great place to be, and it’s where I could easily have stopped. The pages were clean. Nothing obviously disagreed with anything else.

But tidy isn’t correct, and nothing in the system could tell me whether I could actually rely on it. The only way to find out is to ask it things and check what comes back.

So I Tested It

I’d ask questions I already knew the answers to. What’s the deployment process for this project, why did we choose that approach, what’s the constraint I keep forgetting. Not to learn anything. To find out what it didn’t know.

When it got one wrong I’d have it record the miss. Not the correct answer, but the shape of the failure. What it reached for, what it should have reached for, and why the page it read didn’t get it there. The curator would fold that back in on the next pass.

It worked better than I expected. Every wrong answer was a hole in the writing, and the holes were never where I thought they’d be. The pages I’d been proud of as often as the ones I’d rushed.

It’s Just Machine Learning

Somewhere around the tenth session I realised what I was doing. I know the answer. I get it to give me one. I rate the gap between the two.

That’s a loss function. How far its answer sits from mine, scored by me, fed back in.

The only difference is where the update lands. A real training run nudges weights; I can’t touch those. All I can change is words, but the words are what it reads before it answers, so a sharper sentence changes what comes out next time. Same loop, with the learning sitting in the prose instead of the parameters.

I’ve never been a teacher, but this is what grading work must feel like. You’re not handing over facts. You’re reading an answer, marking the exact point where the understanding ran out, then going back and writing something for that spot.

And that’s where it got interesting, because hardly any answers came back wrong. Most came back correct and thin. Right shape, missing the one detail that would have made them worth having: the exception, the constraint, the reason it’s done this way and not the obvious way. A wrong answer is easy, because you can see it. A correct answer that isn’t quite enough doesn’t announce itself, and getting that detail right turned out to be the hard part.

From Quizzes to Ratings

Quizzing has a ceiling. It eats my time, because sitting down to invent questions is work on top of the work. And it only ever finds what I think to ask about, which is the stuff I already have answers for.

Real work finds the rest. It’d open a PR without going anywhere near the page on formatting and commit structure. The knowledge was sitting right there, it just didn’t reach for it. No quiz would have caught that. I’d have asked about the guidance and got the guidance straight back.

So I built rating into the wiki. When it gets something wrong I give it a marker, xbad, and a reason, and it writes both down. That’s the whole interaction: a word and a sentence, in the middle of whatever I was already doing.

There’s an xgood too, and I use it less than I should. Recently it spelled out which wiki pages it had used to answer me, so I could see the trail it had taken. Worth reinforcing, and it cost me four characters to say so.

The markers sit there until the next curation, which runs daily. The agent picks them up and proposes changes to take the feedback on. I label, and the curation does the writing.

Three Loops

Which leaves three loops running at different speeds.

The short one is the agents working the wiki over on their own: the curator tightening pages, the review agent hunting contradictions and gaps. I’m still in it, because nothing gets changed until I’ve read the proposals and said yes. But I’m not the one doing the finding, and that’s what makes it cheap. It’s the loop that keeps the thing from silting up.

The medium one is testing. I ask it things I already know, mark what comes back, and get the answer on the spot: here’s what it doesn’t know, go and fix it. The random page review belongs here too: one page, pulled at random, read properly. That’s the uncomfortable one, because it puts my own process in front of me, written up by the curator from things I said weeks ago, and asks whether I really do what it says on the tin. Often I don’t. The page describes a version of me with more time and better intentions, and either the page changes or I do.

The long one is the ratings. A marker takes seconds to drop and then nothing appears to happen. It has to be picked up by a curation, turned into a proposal, argued into a page. And then that page has to actually get opened at the moment it matters. Information filters in slowly, and I usually can’t point at the change it caused.

But that’s the powerful one. It’s the only loop fed by real work, which means it’s the only one that finds the failures I’d never have thought to test for.

None of them is clever on its own. Together they’re the harness, and the harness is what improves.

Where It’s Got To

I barely read the wiki now. I couldn’t tell you what’s on most of the pages without opening them, and I don’t need to. Pages that have gone out of date float to the surface. The glossary gets stronger every pass, because every pass has a reason to touch it.

It doesn’t get much wrong any more.

That’s the bit I didn’t see coming. All the context I’d been carrying around is on disk now: the why behind a decision, the constraint that isn’t in the code, the thing that bit me last year. I don’t have to carry it.

It isn’t finished. Duplication still creeps in and the copies drift; I still think the answer is compiling the wiki against the glossary rather than curating it, and I still haven’t built that.

But the wiki was never really the point. It’s just where the knowledge sits. The harness is the thing I’ve been engineering, and it gets a bit better every time I tell it where it went wrong, a word and a sentence at a time.

I’ve stopped thinking of it as writing documentation. I’m training the harness.

— Brad

Disclaimer: This project is conducted in my personal time. All thoughts and opinions expressed here are solely my own and do not represent any current or former employers.