116,000 Papers on Leftover Tokens

TL;DR

• The Atlas holds 116,439 papers, and the pipeline that fills it runs off my laptop. Getting the engine here took many full token windows, because you reprocess the whole corpus every time you learn something. Priced as cloud spend, that bill never gets approved, which means it never gets built.
• Put the language work in the model and the judgment in code. Translation and summaries go to a frontier model. The “is this about microplastics” decision is four layers of deterministic rules, because it has to give the same answer every time.
• Cache in front of common reads. Materialize in front of common computations. The dividing line is how much work the query does, not how popular it is.
• Moving new rows into production is the dangerous part. One early push took the live site down for over an hour. The fix was small chunks, one transaction each, and a health check against the live site between them.
• A pipeline that only reports its successes is asking to be trusted. I built an admin page where every count is clickable, including the papers the filter threw away.

---

The Bill That Never Came

The Winnow Research Atlas is a public database of microplastics research. As of this week it holds 116,439 papers, 73,172 of which the system considers actually about microplastics. It’s free to use, and it runs on a pipeline I execute by hand.

That last part is the interesting one, so let me start there.

Every week or so I pull everything new from OpenAlex and PubMed, clean it, decide what’s relevant, translate what isn’t in English, summarize it, turn it into vectors, and push it to production.

The weekly delta is not the expensive part. A few thousand new papers is a modest bill at frontier rates, and if that were the whole story I’d have put it on a card and moved on. What’s expensive is getting to the point where the weekly run is boring, and that has taken many full token windows. Many.

Because the corpus is not a thing you process once. Every time the relevance rules got better, and they got better repeatedly, that’s a hundred thousand papers reclassified. Every time the summary prompt improved, that’s the corpus again. Add the runs that died halfway, the approaches that turned out to be wrong, and the ordinary cost of learning what this data even looks like, and the real figure isn’t a weekly line item. It’s the whole corpus, many times over. That is exactly the shape of spend you cannot put on a client’s card while you are still working out whether the thing works at all.

Then add the orchestration, a queue, a scheduler, monitoring, and the retries you need for when a scheduled job dies at 3am and nobody notices until Thursday.

Winnow Labs is a supplement company. They were not going to approve that line item, and they were right not to. So the honest version of this story is that the properly engineered cloud-native design would have produced a very nice architecture diagram and no Atlas.

Instead it exists, and it has cost close to nothing in cash. The compute was real, and there was a lot of it. It just came out of capacity I had already bought. The only stage that genuinely goes out to a metered API is the embedding step, and that runs about a penny a week.

Claude Will Talk You Into the Right Answer to a Different Problem

Ask any good engineer, human or otherwise, how to build a data pipeline and you’ll get the same answer. Make it a workflow. Put each stage behind an interface. Make it observable, idempotent, testable. Run it on a schedule in production so it can’t drift from the thing you tested. That advice is correct. I give it to clients. I have pushed Claude to hold me to it on other projects, and I’d push it on you.

It’s also the answer to a problem I don’t have.

Nobody presses a button and triggers an ingest. There is no user waiting. The dataset changes when I decide it changes, and if a run dies halfway through, the correct response is for me to look at it and run it again. Under those conditions the elaborate machine is pure overhead. What I need is a script.

But “a script” undersells what it turned into. The thing driving this pipeline is a file of instructions that Claude reads before it starts work. It knows the order of the stages. It knows which one everybody forgets. It knows the specific failure where a subagent finishes its batch and then quietly fails to write the results file, and it tells the model to go dig the results out of the transcript instead. I could not have designed that up front. The pipeline taught me, and the file is where I wrote it down.

It was sixteen hundred words in May. It’s over two thousand now. That growth is the whole argument: every few runs something surprises me, and the file is where it goes.

That’s the real trade. A workflow has to be code, so every piece of hard-won operational knowledge either gets forced into a shape the compiler accepts or ends up in a comment nobody reads. A file of instructions can hold exactly as much nuance as the job actually has. Less rigorous, more complete. For a process a human supervises anyway, more complete wins.

Thursdays

Here’s the other half of why this is free.

I get a token budget through my normal work, and it resets on a weekly cycle. Every so often that reset lands on a Thursday while my own working week doesn’t restart until Friday. For about a day, I’m holding capacity I have no other use for.

That’s when the Atlas run happens.

The tokens are not free in any absolute sense. I pay for them. But the marginal cost of spending them on Winnow’s corpus instead of letting them expire is zero, and zero is the number that made this project possible. I wrote a whole post about trying to measure what a token is actually worth, and this is the practical version of the same argument. The question is never what something costs. It’s what else you would have done with it. Ask it that way and a pile of work that looked unaffordable turns out to be sitting in the gap between one billing window and the next.

Put the Judgment in Code and the Language in the Model

This is the design decision I’d defend hardest, and it’s the one I had backwards in my own head until I went and re-read the code I wrote.

The pipeline uses a frontier model for two things. It translates titles that arrive in other languages, and it writes the plain-language summary sitting under each paper. Both are language problems. There’s no correct answer, only better and worse ones, and a model that genuinely understands what a study found will write a better sentence than one that merely knows how sentences go. So summaries get the deepest model I can point at, and I optimize for comprehension over polish every time.

The model does not decide what belongs in the Atlas.

That decision is four layers of deterministic rules in a single Elixir module. A definitive keyword list, including the word in a dozen languages. A regex for the spaced and hyphenated variants. A polymer or additive term appearing near environmental context. Bioplastics in environmental context. The last three layers carry explicit exclusions. The first one doesn’t, deliberately, because it only fires on words that cannot mean anything else.

The reason is a word. In materials science, “microplasticity” means small-scale plastic deformation in metals. A paper titled “Microplasticity and macroplasticity in copper single crystals” is real, it is genuinely about microplasticity, and it has nothing whatsoever to do with plastic in the ocean. Keyword search cannot tell the difference. Neither can a model, reliably, on a bad day.

But that isn’t the main argument. The main argument is that this decision has to be identical on Tuesday and on Friday. When the definition improves, and it has improved several times, I re-run all 116,439 papers against the new rules and every paper’s answer moves for the same reason. I can read the rules. Winnow’s scientist can read the rules. They’re published on the methodology page, so anyone using the Atlas for real work can check whether my definition of the field matches theirs.

Ask a model that question 116,439 times and you get 116,439 slightly different judgments and no way to audit any of them. Language is the model’s job. Policy isn’t.

Materialized Views and Caches Are Not the Same Tool

Now the boring part, which is the part I see people get wrong most often.

I use both. The rule I’ve landed on is that you cache in front of common reads, and you materialize in front of common computations.

A paper detail page is a read. Someone asks for paper 41,208, you go get it, and if a thousand people ask for that same paper you’d rather not fetch it a thousand times. That’s a cache. Cheap query, high traffic, short expiry, done.

Country rankings are not a read. To answer “how many papers came out of Germany,” Postgres has to walk from papers to authors to affiliations to institutions to countries, count them, count the high-quality subset separately, sum the citations, count distinct institutions, and then do all of that again for every other country. None of that gets faster by remembering the answer for the last person who asked, because the first person still waits. Caching an expensive query mostly relocates the pain onto whoever shows up right after it expires.

So the aggregations live in materialized views. There are seven of them, and the split between them turned out to be the interesting part.

Five are cheap rollups behind the leaderboards, one row per country or institution or author. Those refresh on every push. The other two are ambitious: one row per country holding the counts, the top ten institutions as JSON, the top ten authors as JSON, and the publication trend by year, so an entire detail page becomes a single primary key lookup instead of five separate queries.

Those two don’t refresh on the weekly push at all. They cost more to rebuild than the whole rest of the refresh step, and a detail page for one country is not worth that, so they’re built ahead of time and the page falls back to bounded live queries when the view doesn’t have the row. That’s the honest version of the tradeoff. Materializing is not free, it’s just prepaid, and prepaying for every country’s detail page when most of them are never viewed is the same waste in the other direction.

The dividing line is not how popular a query is. It’s how much work it does. When I’m unsure which tool applies, I ask whether the slow part is finding the data or computing over it. Finding is a cache problem. Computing is a view problem.

And then, because it’s both, the caches sit in front of the views and warm themselves every hour. The one exception is related papers, which is a vector similarity scan across the entire corpus. That one gets cached for a full day, because it’s the most expensive question the site can be asked and the answer barely moves.

The Dangerous Part Is Production

Everything up to here happens on my laptop against a local database. The last step is moving new rows into the production database, and that’s the step that has actually hurt.

In May, a push I’d written generated about 26,000 individual UPDATE statements and ran them in a single session. It saturated disk on the production database and took the Atlas paper listing down for over an hour. I killed the client. It didn’t help. The server kept executing for another twenty minutes and then rolled the whole thing back, so I got the outage and none of the data.

The rewrite isn’t clever, it’s just careful. The data goes up once, in bulk, into a staging table. Then it moves across in chunks, and each chunk is its own transaction, so if I kill it halfway through, everything already committed stays committed. Between chunks the script makes an HTTP request to the live Atlas and times it. Slower than a second and a half, it refuses to start. Crosses two seconds mid-run, it pauses.

The chunk size took a second try to get right. A thousand rows still wasn’t small enough, because the relevance column is itself indexed, so flipping it breaks Postgres’s in-place update path and a thousand-row chunk kept blowing through the statement timeout. It runs at two hundred now. That number isn’t a guess, it’s the largest one that landed reliably, and the comment in the script says so, because in six months I will not remember why it isn’t a rounder number.

That’s the whole fix. Small chunks, own transaction, ask the live site how it’s feeling before you make it feel worse. It’s the kind of thing you only write after you’ve had the bad afternoon.

Then the views refresh and the caches warm, which takes about two minutes, and the new papers are live.

Show People the Leftovers

Every stage of the pipeline gets a column, and every row is one day’s ingest from one source. On July 16 the run pulled 4,944 papers. 3,803 from OpenAlex, 82 from PubMed, and 1,059 that both databases had, matched and merged into single records. 852 were excluded by the relevance filter, and 4,092 went live.

The number I care about most on that page is the excluded column. 834 OpenAlex papers didn’t make it.

A pipeline that only reports what it accomplished is asking you to take its word for it. All the interesting failures are in the discard pile, both the copper crystal papers that belong there and the real microplastics research that doesn’t. So the counts are clickable. Winnow’s scientist can open the 834 and read exactly what my filter threw away, and when he tells me something in there should have made it, that’s a bug report I can act on. He knows the field. I know the code. That page is where those two things meet.

If you take one thing from this post, take that one. You are going to make judgment calls in code on behalf of somebody who understands the domain far better than you do. Give them a way to check your work that doesn’t require them to read your work.

Know Your Use Case

All of this follows from a single fact. The dataset changes only when I decide it changes.

That one constraint is what makes a hand-run script acceptable, makes an hour of cache staleness fine, makes materialized views obviously right, and makes a manual production push safer than an automated one. Change that fact and most of these answers flip. If Winnow needed papers indexed within an hour of publication, or if a hundred people could trigger an ingest, I would have built the elaborate machine and it would have been the right call.

I don’t think this is the permanent shape either. In a year or two the costs that make this uneconomical in the cloud will have fallen far enough that the automated version is the obvious one, and I’ll build it then. The conveyor belt delivers this too.

But it would have cost thousands of dollars today, and a thing that costs thousands of dollars before it has proven itself doesn’t get built. It gets discussed. So this design isn’t a compromise I’m apologizing for. It’s the reason there’s anything here to automate later. Find the way around the wall. You can move the wall once you’re standing on the other side of it.

What It’s Actually For

We’re pointing the same pipeline at probiotic strain research now. Same shape, bigger corpus. 217,246 papers mapped against 445 strains, with an enrichment layer that works out what mechanism a study is about and whether the subject was a human, an animal, or a petri dish. It goes public soon.

The point of that corpus is to answer questions that currently require a month-long literature review. Which strains have real randomized human trials behind them and which have one mouse study and a lot of marketing. Where the evidence for a given mechanism actually lives. Which combinations nobody has looked at yet.

That’s what I want to build with this technology. Intelligence pointed at connecting dots across a body of research too large for any person to hold at once, so that research gets directed better and products get made smarter. Not another system for watching people.

And the business footnote, since I’m a consultant and this was paid work. The altruistic version turned out to be the commercially correct one. Winnow gave away a genuinely useful research tool, and it has driven more organic traffic and more credibility than any amount of content marketing would have. They went from a supplement company with an opinion about microplastics to the place people go to look microplastics research up. That’s a completely different position in the conversation, and they got there by being useful first.

Go browse it at atlas.winnowlabs.com. The relevance filter I described is documented on the methodology page, and that July run has its own write-up. If you find a paper my rules got wrong, I would genuinely like to hear about it.

---

David Kerr is the founder of Kerrberry Systems. He builds custom software for businesses that want to own their systems, not rent someone else’s. Find him on LinkedIn or GitHub.