Robin B Thacker @RobinBThacker

@robjama Smart move. Going to set up something like this in Nanaimo.

@DegenApeDev West coast homie here...on V.I.

@bonkaton The word for that is 'frisson'. Funny how that works...ask a question to the universe and get the answer.

@rayansadri Just became your 1000th subscriber because of your post...way cool!

1/ Honest take: the agent isn't the bottleneck, verification is. Claude Code, Boltz-2, AF2 all produce plausible outputs faster than I can audit them. The interesting workflow question isn't "which agent", it's "what catches the confabulations." 2/ Built Verocite for this: peer-review harness that scores AI-generated claims on verification independence (sources + tests + reasoning / 3). 3/ Current runs: Mathys 2019 + Murphy/Skene 2023 AD snRNA-seq convergence audit. Two independent systems caught distinct violations the other missed — suggests multi-system audit is the move, not single-model verification. Happy to share the harness.

Building a verification-integrity gate that catches preservation failures in computational biology before they propagate into costly downstream R&D decisions. It's constraint ordering is substrate-independent so the same instrument that audits a docking pipeline also governs an LLM agent, a snRNA-seq analysis, or a clinical reasoning chain. One diagnostic instrument that is transferable across every layer of the R&D stack.

@Cappy_Nate Count me in. AI safety evals at inference.

I think one of the biggest challenges when it comes to going hard into using AI is loneliness I am learning all these awesome things and becoming super capable But the set of people that I can really talk to about it is very small Is anyone else having this experience?

@johnennis It was like that at first and fortunately once my inner circle understood that this is a new world, they began to understand.

@brian_armstrong And here is what everyone in this space is missing--verification infrastructure. Even AF2(3) has that as a major problem, and the longer it goes on, the worse the ground truth will become. That's what we are building, and so far we are making decent inroads.

Worth a closer read, though. If anyone is contemplating this, it's $220K annualized and prorated to 3 months. That's ~$55K for the term and full-time in SF. More importantly, what happens to the IP you develop during those 3 months? Standard tech employment contracts assign to the employer everything created during employment. For the garage scientists and renegade builders this targets, who already own novel work, that's the question to carefully consider before applying.

I use a proprietary constraint hierarchy that instruments where claims outrun verification. I took the liberty of applying it to your Mathys reanalysis as a case study. It was conducted as a multi-system convergence audit with two independent AI auditors under the same epistemic constraint. The report surfaced structural features in both papers that a standard review wouldn't flag. More than happy to share the full report to your Imperial email if you're interested.

Are there other builders working on AI verification / reliability? Hallucinations, false confidence, evidence-bounded RAG, evals. I’m also applying this to bio/chem workflows where mistakes are expensive. If you’re in that lane (research or product), I’d love to connect.

You just described the constraint we formalized and tested across five domains. The flywheel inverts when the system generates more than it can verify. In protein folding, we found that 1 in 3 high-confidence AlphaFold2 predictions on fold-switching proteins are confidently wrong. In drug discovery, analyzing 7 cofolding models totalling 16,163 predictions, the confidence tracks the global fold, not the local pose. It's measuring the thing that's almost always right, not the thing that matters. In cancer genomics, our most recent research shows that two frameworks, both claiming to measure chromosomal instability, yield orthogonal disagreement for the same patients. They don't agree on which patients are hard to classify. For LLM sessions, 57% confabulation rate on a real research question. Same question under our structural constraint: 0% without fine-tuning, nor retraining. In research documents, the same tool that scores Shannon 1948 as proof-heavy also finds an AI governance constitution to have 89% unverified preference selectors. This isn't just a bio problem. It's any system where inference runs without provenance. There is a uniform structural pattern in this, and it's visible if we look hard enough. We'll be building the open infrastructure to fix it as soon as we finish publishing the rest of our findings.

@karpathy I built a tool expressly for this problem. You keep circling it from different angles. Supply chain trust without behavioral verification; AI psychosis, plausible outputs without verification paths and now agents assembling production infrastructure where successful deployment feels like verification but isn't. Same gap every time: the system can't distinguish "it ran" from "it's right." I've been measuring this gap across substrates. Here's what the tool found: → It blind-distinguishes a Shannon-class foundational paper from a structurally compromised one using a clear metric on the reasoning deltas — no domain knowledge required, just the constraint applied to the claim structure. → Screened 27,098 AlphaFold2 predictions across 18 fold-switching proteins from the Porter benchmark. Found a 33.6% FalseVerify rate — one in three high-confidence predictions is simultaneously confident and wrong. Discovered an INVERTED confidence pattern invisible in benchmark data where the model is most confident where it's most wrong. Published on bioRxiv: "Confidence Without Verification: Screening pLDDT Unreliability in AlphaFold2 Fold-Switching Predictions." → Found the convergence of false-verify patterns across three independent AAV capsid engineering datasets — Ogden/Church (207K variants, Harvard), Bryant/Dyno (297K variants, Harvard/Google), and Eid/Fit4Function (100K variants, Broad Institute). Three different serotypes, three different labs, three different mutation strategies. The tool found the same structural deficiency in each: confidence on the measured trait (packaging) doesn't transfer to the unmeasured objective (tropism, immune evasion, multi-trait fitness). In Fit4Function it found 8,816 near-miss candidates the pipeline discards because of a single-assay bottleneck, with a data-driven rescue map showing which single-residue edits recover them. In Dyno it found 62.9% of model-endorsed variants fail the project's actual objective. In Ogden it caught a ratio artifact in the thermostability data that inverts the biological conclusion — then retracted its own initial finding when dual-system convergence showed the artifact. → It mapped 42 nodes from Fletcher's TIG distributed optimization protocol Rust source in a single session — having been built for protein structure data. Found strong computational integrity but absent algorithmic correctness verification. The protocol discards failure reasons and its innovators re-explore dead territory because the system can't remember why something failed. → Across 28 LLM architectures from tiny models to frontier, 420 evaluations: governed systems score full integrity, ungoverned systems score roughly a third of that. The delta is structural, not prompt-based. This is where the tool was developed and every architecture showed the same result. The tool adapted from LLM reasoning to protein folding to capsid engineering to Rust codebases to research documents without redesign. It doesn't need to understand the domain. It finds where claims exceed what verification supports. The empirical data behind all of these is done. The AlphaFold2 work is published on bioRxiv. The rest are in preparation for preprint. Your autonomous DevOps agent needs exactly this: a constraint layer that verifies substance, not just process, before deployment rather than after failure. If any of this is useful to a problem you're working on, DM me.

@Doodlegenics Crypto is Superman's dog spelled with a c instead of a K.

@StephNass Map 4 me.

Hi Farrel, I ran a blinded epistemic hierarchy analysis on your sandbagging dataset (top/bottom 15 consequences). Got Cohen's d=2.98, Spearman r=-0.81 discriminating strong vs weak triggers. The key discriminator your analysis missed: restrictive vs constructive action valence. Happy to share the full report. Would you be open to a conversation about running the instrument on your Inspect traces? Robin (@RobinBThacker)

@hthieblot Easier said than done, but you aren't wrong.

If you use pLDDT to decide what to trust, this matters. AlphaFold2 is confident about predictions it gets wrong 33.6% of the time on fold-switching proteins. The confidence metric tracks internal consistency, not correspondence to reality. We screened 27,098 predictions across 18 fold-switching proteins and found five distinct failure patterns. A blind screen of unvalidated E. coli candidates revealed a sixth one that doesn't appear in benchmark data at all. Confidence Without Verification: Screening pLDDT Unreliability in AlphaFold2 Fold-Switching Predictions biorxiv.org/content/10.648…

How many conservation laws are we still missing? The brain is a frontier that isn't close to being completely mapped. Life has no scientific origin. A 30 trillion-cell homeostatic system can recreate another and pass half its triadic knowledge to it, and most of that isn't even close to being fully understood. There's a lot of reality left to discover before we're anywhere near just 'creation of the new' left. Genuine awe is how we discover it.