Too Many Notes is No Notes: The Design of note-supplement

Why “too many notes” and “no notes” are the same thing

Let me start with a rule I decided on early and haven’t changed since: in my note vault, there can only be one note per topic.

It’s not about being a neat freak. It’s because if the same thing is covered in five different places, I have to open five files when I look something up, and figure out which version actually counts. That organization work didn’t disappear, it was just left intact for my future self. And my future self is usually seeing patients in clinic or rushing to finish slides, with no time to do this. The chaos caused by too much information is the exact same pain as not having enough information.

This rule is easy to follow when you only have one source of material. The hard part is later, as sources pile up, and none of them will stop and wait for me:

• Textbooks → textbook-to-note, turning my own PDF textbooks into structured notes with citations (repo)
• Papers → paper-radar for tracking, paper-fetch for grabbing full texts, paper-review-and-digest for critical appraisal and content summary
• Lectures → haven’t cleaned this one up yet, currently the only branch I haven’t open-sourced

These tools all solve the same problem: turning outside stuff into readable material. They bumped up the material output, but that “one note per topic” rule didn’t get easier to keep because of it. It got harder.

So the whole architecture is two layers, with a very hard line between them:

1. Source library: Everything from the outside gets dumped in here, kept as-is, ready to trace back at any time. It can be messy, repetitive, or contradictory, because its only job is “preserving evidence”.
2. Topic notes: The part that belongs to me. There is always only one note per topic, kept in an isolated folder, written to the same template. It’s the only copy I trust, and the only one I’ll actually use.

Source material is not notes

Once this line blurs, your note vault degrades into a slightly prettier downloads folder.

And the act of merging material content back into topic notes is the border between these two layers. note-supplement, which this post is about, does exactly that.

That was also the part of the pipeline I got stuck on the longest. When writing a new note from scratch, if AI does a bad job you can tell at a glance: messy structure, no citations, empty content. But “supplementing new material into an already existing note” is different. I have thousands of notes, and every single one is the result of me seriously verifying things at some point in the past.

Why “update my note” is such a dangerous phrase

The intuitive approach is to throw new material at the AI and say “update this note for me.” I had to get burned a few times before I figured out where the problem lay:

The real risk of having AI update your notes isn’t that it misses new content, but that it very quietly changes the things you already got right in your old notes.

And you won’t notice, because the output still looks like a good note.

The costs of these two errors are completely asymmetrical:

• Missing new content: Recoverable. The next time you read the same material, or the same topic comes up again, you still have a chance to add it.
• Breaking old content: Irrecoverable. Because you have already verified that content, you won’t go verify it a second time. From then on, it stays in your note vault in its wrong form, carrying the trust you yourself gave it.

Once you see this asymmetry clearly, the design for the whole tool comes down to one sentence.

Design principle: additions are cheap, modifications are expensive

In every round, note-supplement checks the material against the existing note for three things:

• What is not in the note and can be added
• What is different between the two sides
• What is already in the note but not mentioned in the material (these are strictly left alone)

Then it categorizes every change first, before deciding who has the authority to execute it.

Auto-approved: additions with citations

Adding one more piece of cited content won’t hurt the note. So changes like “add factual bullet with citation” are written directly and logged in the changelog.

(I later removed one item from auto-approval: turning scattered bullets into a table. It looks like just formatting, but actually it’s stuffing every fact back into a column. That’s a rewrite, not an addition. And column misalignment is a silent failure—the output even looks tidier than the original. Now it has to ask before doing that.)

No “destructive” rulings: factual contradictions

This part is where I made the most changes, and it was the biggest design flaw in the first version of the tool.

The rules for the first version were written very stubbornly: send to review for any factual contradiction, no matter how strong the evidence looks on either side, strictly no auto-ruling. Sounds very responsible. After running it for a while, I realized two things:

First, a pending review list isn’t a safety mechanism, it’s a procrastination mechanism. Once the list gets long, you just click your way through it. The so-called “leave it for you to judge” practically equals “auto-apply, but with extra clicks.” And every item on the report is followed by a line saying “My recommendation is…” That line anchors you, and most of the time you just click along with it. I’m not the one judging; it’s judging, just with more polite formatting.

Second, not making a decision is itself a decision. During the time a conflict sits in the list, that potentially outdated old claim in the note is still the only version written there and trusted by me. The practical effect of “no ruling” is “the old one automatically wins”, and I never actively chose that policy.

So v1.1 took a different stance: it’s not about not ruling, it’s about not destroying. Contradictions are first categorized on two axes:

• Type: Is it a new edition of the same book or the same guideline replacing an old one (that’s not a real controversy)? Are two separately credible sources genuinely saying different things? Or does the note lack a citation entirely while the material has one?
• Risk: What happens if this value is wrong. Dosages, thresholds, contraindications, and red flags count as high risk; prevalence, naming history, and background knowledge count as low risk. This axis is an adjustable knob, with different scales for different fields. Medical notes are just at the strictest end of the dial.

Then it routes by category: new editions replacing old ones are updated directly, but the old value is kept in place with strikethrough, along with its original citation and date—not a single word disappears. Low-risk divergences are written directly into the note as an in-place block:

> [!conflict]- ⚖️ Unresolved: natural history of adhesive capsulitis (flagged on 2026-07-23)
> - **A (Existing)**: self-limiting, resolves in 12–18 months `(Citation, Year)`
> - **B (New material)**: a significant proportion of patients have residual symptoms after three years `(Citation, Year)`
> - Handle this the next time you read this note: delete this block, keep the version you adopt (or keep both with added conditions)

The original wording for both sides is there, the citations are there, the flagged date is there. Neither side gets deleted. It doesn’t interrupt me, but it doesn’t disappear either. The timing of the review shifts from “the moment of writing” to “the next time I read it." That’s when my attention is at its best and my context is most complete; looking at a list right after running a batch is exactly the worst time.

Only high-risk contradictions will stop and ask me on the spot, and at most five questions at a time. Anything beyond that is still written as the block above, and honestly reported by the numbers in the changelog. I’ll read five questions seriously; I won’t read thirty.

There’s also a patch I missed for a long time before realizing: if a note has a pending high-risk contradiction, any additions that rely on this disputed fact (in the same paragraph, or content where that value is a premise) are held back together, waiting to be written in until after I make my decision. Otherwise, by the time I open the note to make a ruling, the note has already been altered a round beforehand.

What about the concern from the first version, where the cost of a misjudgment is “I won’t go verify that point again in the future”? That concern wasn’t overturned, it was bypassed: none of the rulings delete anything. The old value is still there, still carrying its citation and date. If I ever discover I made the wrong call this time, I can still trace it back.

Always wait for human review: refactoring and opening new notes

Moving paragraphs, merging sections, and determining “this should be a new note” all require an understanding of the entire note vault. It’s not something single-note comparisons can decide.

scatter mode: one piece of material, a whole batch of notes

Supplementing single notes only solved half the problem. In practice, what’s more common is one piece of material affecting a dozen notes: a comprehensive review lecture, a guideline update. This is exactly the most common scenario under the “three sources flowing into one note vault” architecture.

--scatter mode extracts all concepts from the material, finds every related note, and then applies the tiering above, note by note:

The goal of tiering isn’t automation, it’s allocating attention. The mechanical eighty percent is done automatically, and you only look at the twenty percent that requires judgment. After each run it gives you a changelog: which notes were touched, what got added to each, how conflicts were resolved, what you vetoed.

Don’t trust the model when it says “I did it”

Once the material gets long, the language model’s attention dilutes, an entire paragraph gets skipped, and the output looks perfectly normal. I talked about the mechanism for this in the chat gets dumber the longer you talk post; the difference is back then we were talking about conversation quality, while here we’re hitting a wall where a chunk of content is actually missing. And if you ask it afterward, “Did you finish supplementing everything?”, it will say yes.

So the rule here is just as important as the earlier one: don’t trust the model’s self-reporting, only trust the things it leaves behind that you can audit.

Forced item-by-item closure coverage list

Every heading in the material has to be listed out and closed item by item: produced N items, already covered, or absolutely nothing. For any heading that yields “absolutely nothing”, it must go back and re-read it once before calling it a day. When a whole section under a heading produces nothing at all, it’s usually not because the section is actually empty, but because attention dropped.

And this list is part of the output, not its own scratchpad. Here’s the difference: self-checking on a scratchpad can’t be verified, but a delivered list can be audited.

Structure-blind reverse scan

The remedy in the first version was: take the draft items produced in the first pass as search terms, and go back to re-read the material. This approach is completely backward. Keywords derived from “things you already found” can only retrieve content you half-found; for a whole chunk of a topic that went completely unnoticed, you don’t even know what words to search for. What it missed is invisible to itself.

So the second round ignores chapter structure entirely, directly listing every item “present in the material, missing from the note.” If either lens catches it, it counts.

This also patched another hole: newly added content isn’t just compared against the section it’s going into, but against the entire note. An item written into “Treatment” could totally clash with a number in “Background.” If you only compare against the same section, you’ll never see it.

The parts actually guarded by code

Things involving judgment are handed off to the model, but if it can be checked with code, don’t check it with the model:

• scripts/gate_check.py: mechanical checks run before writing, including broken links, broken embeds, leftover citation placeholders, tables interrupted by prose, duplicate embeds. Purely syntax layer, requiring no judgment, so the model’s attention shouldn’t be wasted on it.
• benchmarks/: 14 sets of material/note pairs with deliberately planted defects, each set coming with a gold standard answer (how many items to add, which ones are conflicts, what the risk tier is). After changing the workflow, I can actually run it once to see if it regressed, instead of relying on a feeling that “it seems better.”

No overlap between the two layers

The model layer handles semantics, the code layer handles formatting and verifiability. This way the entire pipeline can actually be considered to have a harness, rather than just being a very long set of instructions.

A few other details I added only after actually failing

Ask first if the material itself is credible

Categorize the material before comparing: handouts I organized myself, textbook chapters, and peer-reviewed materials count as one category; AI-generated summaries and someone else’s notes from unknown origins count as another. The moment the latter shows up, all auto-writes for the whole round are downgraded to manual confirmation. This is a property of the document, not a property of item-by-item judgment. An untraceable piece of material doesn’t become traceable just because a few of its points sound reasonable.

And the bottom-line rule has no exceptions: every piece of content written into a note must come with a citation. Anything without a citation is marked as AI inference, and isn’t allowed to mix in.

The material and the target are two different universes

Unorganized raw material folders (inbox, lecture transcripts) can never become the targets for supplementation. Without this rule, AI will very happily go “supplement” your unprocessed junk, growing a second, crappier note vault right next to your real one. That’s exactly what I said at the start about “too many notes is no notes”, except this time you built it yourself.

Empty paragraph scanning

With permanent notes written to a template, there are always a few paragraphs that stay empty for years on end. So before comparing, the tool first lists the empty paragraphs in the target note, and suggests where to supplement them from (the evidence paragraph → literature search; the mechanism paragraph → corresponding textbook chapter), asking all at once.

How to use it

This is a skill file for Claude Code (essentially a markdown playbook, the workflow can be ported to any agent framework that can read it):

1. Copy skills/note-supplement/SKILL.md to your project and save it as .claude/commands/note-supplement.md
2. Swap out the three placeholders at the top: {NOTES_DIR} (your permanent notes folder), {SOURCES_DIRS} (material folders), {SEARCH} (your search tool)
3. Run /note-supplement <material file> in your note vault

Having a semantic search index makes “finding the right target note” a lot more accurate. I use my own open-sourced vault-search (you can also pair it with that graph RAG setup), but it runs fine with Grep at the start too.

Repo (MIT, README has English and Chinese versions): https://github.com/drpwchen/note-supplement

I’ve organized other tools along this pipeline in my My AI Toolbox post.

A little reflection

The goal of the entire pipeline has only been one thing from start to finish: information on the outside can be vast, but once it lands on my end it has to converge into a single copy. Those earlier tools handle turning sources into material, this one handles the convergence, and this convergence step is the hardest part, and the easiest place to mess up.

It was also while doing this step that my thoughts on “having AI do things for me” changed a bit.

I originally thought a good AI tool was about doing a little more: understanding a little more, deciding a little more, asking me a little less. Only after finishing this did I realize that on “things that have been accumulating for a long time,” the value of a good tool is actually knowing what it shouldn’t do, identifying the asymmetrical risks, letting cheap actions pass, and stopping the expensive actions to hand them back to humans.

But this revision dialed it back another layer. I originally thought “stopping and handing it back to humans” was the finish line for being responsible. I later realized that was just moving the responsibility to a place I wouldn’t seriously read. Throwing all the hard problems into a pending review list is formally very responsible, but practically it’s outsourcing judgment to fatigue. The real question to ask isn’t “Should this be decided by a human?”, but rather “Is this worth interrupting this person right now?”. I’ll read five questions seriously at a time; the rest, leave them in place in a way that won’t break anything, and we’ll talk about it the next time I read it.

This logic actually has no necessary connection to medical notes. Any system that “has already accumulated a batch of credible content, and needs to continuously pump new data in” applies here. The only difference is which notch that risk knob is turned to. I wrote about this separately in the repo’s docs/the-pattern.md.

If you’re also dealing with the problem of “old and new data fighting each other,” you’re very welcome to chat with me; if you think I got something wrong, you’re even more welcome to correct me.

Buy me a boba!

—