The paper learning station I shared earlier had “Quality” and “Content” buttons, but I didn’t release the backend implementation back then. It was basically a platform with a frontend web page, but you still had to organize the backend yourself. But for my own setup, of course the backend still had to be wired to AI to reduce the cognitive load. So here I’m sharing those two backend skills to help speed up everyone’s learning!
—
One handles “is this paper credible?”
The first is called paper-review. It does the kind of critical appraisal you see in a journal club.
Automatically picking the right tool by study design
It picks the recommended critical appraisal tool based on the paper’s category: a randomized controlled trial uses Cochrane RoB 2, an observational study uses ROBINS-I, a systematic review uses AMSTAR-2, and diagnostic accuracy uses QUADAS-2. It doesn’t just brute-force a universal checklist. This sounds obvious, but you know most “AI reads papers for you” tools actually just cram everything into one universal checklist.
Citation check: confirming the literature “actually exists” first
My favorite step is the citation check. It first goes to CrossRef to confirm that every cited paper “actually exists,” then compares “what the paper says this citation is about” with “what the citation is actually about.”
Why check if it exists first? Because without auditing, the accuracy of AI-generated citations is actually only 40 to 80 percent. Fake DOIs and attributing things to the wrong author happen way more than you’d think. So this step is a very cheap but highly useful idiot-proof measure.
Statistical significance does not equal clinical significance
There’s another point I insisted on: statistical significance does not equal clinical significance. So it takes the effect size and compares it against the scale’s minimal clinically important difference (MCID). If the p-value is less than 0.05 but it completely misses the MCID, it will just say so directly.
Overall, it draws on Dr. Hsieh-Ting Lin’s sharing and what I learned before in evidence-based medicine. You can also add whatever parts you think are important.
—
The point: I don’t let AI score based on “impressions”
This is the part I personally found the most interesting this time.
At first, I let Claude calculate the level of evidence (GRADE) and judge whether “the conclusion exceeds what the data can support” on its own. But I later felt this was wrong, because letting a language model give a score “based on overall vibes” means it can talk a great game, but you have no way to check how it actually got there.
So I just wrote two deterministic little scripts, separating “judgment” and “calculation”:
grade_judge.py: turning GRADE into arithmetic
The language model is only responsible for evaluating five domains (risk of bias, inconsistency, indirectness, imprecision, publication bias), giving “not serious / serious / very serious” for each. The actual final grade is calculated by the script adding them up, not stated by the model itself. The model’s self-reported grade is just for reference; if it doesn’t match what the script calculates, the script will jump out and warn you, “These two don’t match, go back and re-check a certain domain.”
It will even block GRADE rule errors for you. For example, once an observational study is downgraded in any domain, it is not allowed to be upgraded again. The model easily misremembers this rule; the script doesn’t.
argdown_lint.py: turning logical flaws in conclusions into something checkable
This script is even more fun. The language model labels the paper’s conclusion and each “finding used to support the conclusion” with a type (Is this direct evidence from a randomized controlled trial? Or just correlation? Or just a surrogate marker? Or just a secondary outcome?), and then the script judges whether this inferential leap is legal.
For example, “using improvements in surrogate markers to claim efficacy for real clinical endpoints,” “using correlation to claim causality,” “using a single study to claim ‘consistently shown’"—these are all the favorite hiding spots for paper spin. When the script catches it, it just highlights it in red and reports a flaw.
The division of labor is very clear: the language model does the semantic judgment it’s good at, and the script does the deterministic logic it’s good at. Only this way is the resulting conclusion not just a “feels rigorous” vibe, but something you can actually lay out and check step by step.
—
One handles “what this paper is about”
The second is called paper-digest. This one doesn’t judge quality at all, it just does one thing: lets me absorb the entire content into my brain at top speed.
It uses a three-tier structure: 30 seconds for a one-sentence takeaway, 5 minutes for the key points of each section, and if you want details, read down into the full content. And it swaps skeletons depending on the paper type—empirical studies, reviews, guidelines, technical papers each have their own way of being organized, not shoehorned into the same template.
Most importantly, it will force me to do active recall. After organizing, it generates a few self-test cards and pushes them to my own review site, using the same spaced repetition idea as Exam Practice Platform. Because I know my own bad habit very well: making beautiful notes and then never going back to remember them 😅. So I specifically designed this skill so that “organizing isn’t the end, recalling is.”
And it has a hard rule: if it can’t get the full text, it just stops. It’s not allowed to generate a seemingly complete summary just from the abstract. Because a “content summary” generated from an abstract is exactly what’s easiest to fool yourself with.
—
An honest reminder
These two things were originally tied deeply to my personal system (my Obsidian, my Zotero, my self-hosted review site). This time I extracted all these private settings into a config file so others can fill in their own, but it’s more like a case study of “how a clinician wrote the critical appraisal workflow in his head into an AI skill,” rather than a product you can just download and run blindly.
What you can actually take and use directly, and what is valuable to anyone, are those two deterministic scripts and that “picking tools by study design” look-up table.
It is not medical advice, and it cannot replace you reading papers yourself. It is a second reader that refuses to let “sounds reasonable but is actually wrong” critical appraisals pass.
If there are misunderstandings, you’re welcome to share or correct me. I’ve put the code and full English and Chinese instructions on GitHub:
👉 https://github.com/drpwchen/paper-review-and-digest
If this tool saved you time, you’re also welcome to buy me a boba tea so the servers can keep spinning 🧡
—