Hamel Husain

When I see this it confuses me. ssh into a computer for god sakes, or use codex remote feature w/ mac mini. Once you have been stuck in "i need to keep my laptop open" more than 3 times, its crazy to keep this insanity going.

Last open lesson of the year! How to crack the AI Evals Interview! AI Evals are now a part of an increasing number of AI job postings, and you're now more likely to encounter them in interviews. I've helped design quite a few evals interviews, so I know what kinds of traps exist, along with nuanced things companies tend to look for. I'll discuss this, as well as how to structure a take home or report to help you think through the problem clearly. Tuesday Oct 6, 8:30am PT. This one assumes some basic evals knowledge. Read this first if you're new to the topic to get the most out of the session: https://hamel.dev/evals-faq Sign up here: https://maven.com/p/07e7c7

I cannot understand this demo video at all. Sadly, it’s visual slop 😭 Not trying to disrespect anyone maybe the feature is cool it’s just what happened to demos humans can understand

Marketers: The next wave of founders will be marketers, distribution is the new moat. PMs: Taste is the moat, and its easier than ever to build the wrong thing. Now is the age of the PM. Eng: You need systems thinking and technical chops to solve hard problems and wield agents effectively at scale. Engineers have more leverage than ever. I suspect good people in each discipline will remain valuable and there isn't one role poised to become unicorns en masse. I do think its good to be skilled at more than 1 discipline, but I am not sure any one of these disciplines has a leg up over the others in that regard

Q: How often should I run my evals? A: Weigh the cost and how saturated the eval is against the business value of catching the error. Read more: https://hamel.dev/blog/posts/evals-faq/how-often-should-i-run-my-evals.html Read all the FAQs here: https://hamel.dev/blog/posts/evals-faq/

New blog post 🎉: How To Use Decision Models For Evals In this post, we show you how to choose a decision model for your task, tune its threshold (don't assume the 0.5 default is good!), and interpret its scores. Lots of cool interactive charts here. https://x.com/i/article/2112552453422104576

You have to respect Schmidhuber’s commitment to the bit 🤣 he was just trolling us all along (which is probably the right way to use this site tbh)

Saw this job posting for an "anti-slop" professional writer but the post itself written with a slop cannon. This is peak AI psychosis, someone is so enamored with their own writing they can't even see it

Q: Are similarity metrics useful for evaluating LLM outputs? A: Similar wording does not tell you whether an answer works for your application. Check specific failures. Similarity metrics can help with retrieval and output diversity. https://hamel.dev/blog/posts/evals-faq/are-similarity-metrics-bertscore-rouge-etc-useful-for-evaluating-llm-outputs.html

Q: Should I use "ready-to-use" evaluation metrics? A: No. It's a bad idea in most situations. https://hamel.dev/blog/posts/evals-faq/should-i-use-ready-to-use-evaluation-metrics.html

Should you rely on public benchmark scores when choosing a model? @VikParuchuri argues "no" in this talk. To illustrate this, he shows specific problems found in popular OCR benchmarks: ParseBench: • A test expects 310 gigawatts of solar-panel capacity when the correct value is 303. Additionally, 73% of its chart tests ask for values that aren't printed on the chart. (4:51) • A test requires "Parish Priest:" to be bold even though those words aren't bold in the source document. (15:33) • The scorer fails to recognize heading tags placed on separate lines. (16:15) • dots.mocr ranks last among six models even though it beats the first-ranked model on content and tables. This is an example of how overall benchmark score can hide important differences in model capabilities. (14:48) olmOCR-bench: • A model correctly preserves a line break in a document and gets penalized for it. (10:30) • A test expects "Republican Party" when a document says "Republican party". Copying the capitalization correctly makes the model fail. (10:49) • A 700-line postprocessor that rewrites model output adds 1 to 2 points, more than many gaps between models. (12:12) OmniDocBench: • This benchmark uses edit distance, which penalizes a correct table written in HTML more heavily than a wrong number in Markdown. So the format matters more than the answer! (6:58) This is just the tip of the iceberg, the video contains more examples. Chapters: 0:00 Choosing a model when every vendor claims to be #1 1:54 How OCR benchmark scores improve without better OCR 5:30 Metric design and tests that reject correct answers 11:33 Postprocessing, saturation, and misleading model rankings 12:40 How to slice benchmarks by your task and check for bias 16:53 How to build fair tests with verified answers and consistent scoring 19:05 How to test OCR models on your own documents 24:10 Benchmark marketing, private evals, and how to choose a model

👀