Data For Science @data4sci

Take Control Of Your Data. Join our Data Science Briefing newsletter for the best in #DataScience and #MachineLearning data4sci.com/newsletter Manhattan, NY Joined February 2015

Tweets

8K
Followers

1K
Following

462
Likes

351

Data For Science @data4sci

3 days ago

6 days until "Prove Your Prompt Change Actually Helped" — free live class, Sept 9 @ 2PM EDT. Paired testing. Sample sizes. Real signal vs. noise. 30 minutes. Free.

1 0 0 55 0

View Details

Kevin R. MacCauley @KevinMacCauley

3 days ago

@data4sci calls LLM-as-judge without human calibration what it is. Opinion laundering. If you never checked the judge against human labels, you did not measure anything. You bought a more expensive shrug. Most internal dashboards have the same problem.

0 1 0 42 1

View Details

Data For Science @data4sci

a week ago

Hot take: Using LLM-as-judge without calibrating against human labels first isn't evaluation. It's opinion laundering. You're just replacing one shrug with a more expensive shrug. (Reply for how to actually fix this)

0 1 2 341 0

View Details

Data For Science @data4sci

2 weeks ago

Sep 12, 9:30 AM EDT. Live online. 3.5 hours. Evals, RAG, agents, LLMOps — the full production LLM stack. 2 weeks out. Grab a spot: eventbrite.co.uk/e/production-l…

0 0 0 35 0

View Details

Data For Science @data4sci

2 weeks ago

The setup isn't that hard once you have the pattern. 50–200 labeled examples + deterministic checks + LLM-as-judge + CI gate. The hard part is doing it before you're already in production firefighting mode. Start now. Or join us on Sep 12.

1 0 0 11 0

View Details

Data For Science @data4sci

2 weeks ago

2 weeks until my live LLM Engineering Masterclass. Here's what I think is the single most important thing we'll cover 🧵

1 0 1 54 0

View Details

Data For Science @data4sci

2 weeks ago

@svpino Not at that price point…

0 0 0 162 0

View Details

Data For Science @data4sci

2 weeks ago

4/ I'm teaching exactly this on Oct 16. 4 hours. You build it live. You leave with a running harness. Engineers who measure ship faster. Engineers who guess ship incidents. maven.com/bruno-goncalve…

0 0 1 9 0

View Details

Data For Science @data4sci

2 weeks ago

3/ What you actually need: - A judge that doesn't drift (calibrated, not vibes) - Error bars on every metric (bootstrap CIs catch regressions your eyes miss) - Paired tests that prove causality ("prompt A beat prompt B, p < 0.05") - A CLI gate that blocks the merge if eval fails

1 0 0 17 0

View Details

Data For Science @data4sci

2 weeks ago

1/ Most LLM teams don't have an eval harness. They have a spreadsheet of test prompts and a Slack channel full of screenshots. Here's what that actually costs you 🧵

1 0 0 62 0

View Details

Data For Science @data4sci

2 weeks ago

4/ Want to learn how to do this properly — including how to pick your sample size before you run? I'm teaching a free 30-min live class on Sept 9 @ 2PM EDT. "Prove Your Prompt Change Actually Helped" Register (free): maven.com/p/57edbc/prove…

0 0 0 6 0

View Details

Data For Science @data4sci

2 weeks ago

3/ The fix: run BOTH prompts on the EXACT same inputs. Compare the delta per row. This is called a paired test. It's been standard in clinical trials for 50+ years. With a paired design, you often need 5–10x fewer examples to reach significance. Faster, cheaper, trustworthy.

1 0 0 21 0

View Details

Data For Science @data4sci

2 weeks ago

2/ When you run Prompt A on Batch 1 and Prompt B on Batch 2, you're not isolating the prompt. You're mixing prompt variance WITH input variance. Some inputs are just harder. That difference swamps your prompt signal. You're not measuring what you think you're measuring.

1 0 0 42 0

View Details

Data For Science @data4sci

2 weeks ago

1/ Hot take: most "prompt A/B tests" aren't actually A/B tests. They're just vibes with a spreadsheet. Here's what's going wrong — and how to fix it. 🧵

1 1 0 103 0

View Details

Data For Science @data4sci

3 weeks ago

Engineers who measure ship faster. Engineers who guess ship incidents. Oct 16 · 4 hrs live · Build a production eval harness on Inspect-AI: calibrated judges, error bars, CI gates. $500 · Maven Guarantee included. maven.com/bruno-goncalve…

0 0 0 51 0

View Details

Data For Science @data4sci

3 weeks ago

Step 4: Wire it into CI. Every PR triggers the eval suite. Score drops >X% = blocked deploy. Score history tracked per model version + prompt version. Now you have engineering, not vibes. Full live workshop Sep 12 👇 eventbrite.co.uk/e/production-l…

0 0 0 53 0

View Details

Data For Science @data4sci

3 weeks ago

Step 3: Add LLM-as-judge for semantic quality. Write a rubric prompt that scores: faithfulness, relevance, completeness. Run it on your golden set. Validate judge agreement against human labels (>80% target). Track score distributions across deploys, not just single examples.