Hamel Husain

When I see this it confuses me. ssh into a computer for god sakes, or use codex remote feature w/ mac mini. Once you have been stuck in "i need to keep my laptop open" more than 3 times, its crazy to keep this insanity going.

Last open lesson of the year! How to crack the AI Evals Interview! AI Evals are now a part of an increasing number of AI job postings, and you're now more likely to encounter them in interviews. I've helped design quite a few evals interviews, so I know what kinds of traps exist, along with nuanced things companies tend to look for. I'll discuss this, as well as how to structure a take home or report to help you think through the problem clearly. Tuesday Oct 6, 8:30am PT. This one assumes some basic evals knowledge. Read this first if you're new to the topic to get the most out of the session: https://hamel.dev/evals-faq Sign up here: https://maven.com/p/07e7c7

I cannot understand this demo video at all. Sadly, it’s visual slop 😭 Not trying to disrespect anyone maybe the feature is cool it’s just what happened to demos humans can understand

Marketers: The next wave of founders will be marketers, distribution is the new moat. PMs: Taste is the moat, and its easier than ever to build the wrong thing. Now is the age of the PM. Eng: You need systems thinking and technical chops to solve hard problems and wield agents effectively at scale. Engineers have more leverage than ever. I suspect good people in each discipline will remain valuable and there isn't one role poised to become unicorns en masse. I do think its good to be skilled at more than 1 discipline, but I am not sure any one of these disciplines has a leg up over the others in that regard

Q: How often should I run my evals? A: Weigh the cost and how saturated the eval is against the business value of catching the error. Read more: https://hamel.dev/blog/posts/evals-faq/how-often-should-i-run-my-evals.html Read all the FAQs here: https://hamel.dev/blog/posts/evals-faq/

New blog post 🎉: How To Use Decision Models For Evals In this post, we show you how to choose a decision model for your task, tune its threshold (don't assume the 0.5 default is good!), and interpret its scores. Lots of cool interactive charts here. https://x.com/i/article/2112552453422104576

You have to respect Schmidhuber’s commitment to the bit 🤣 he was just trolling us all along (which is probably the right way to use this site tbh)

Saw this job posting for an "anti-slop" professional writer but the post itself written with a slop cannon. This is peak AI psychosis, someone is so enamored with their own writing they can't even see it

Q: Are similarity metrics useful for evaluating LLM outputs? A: Similar wording does not tell you whether an answer works for your application. Check specific failures. Similarity metrics can help with retrieval and output diversity. https://hamel.dev/blog/posts/evals-faq/are-similarity-metrics-bertscore-rouge-etc-useful-for-evaluating-llm-outputs.html

Q: Should I use "ready-to-use" evaluation metrics? A: No. It's a bad idea in most situations. https://hamel.dev/blog/posts/evals-faq/should-i-use-ready-to-use-evaluation-metrics.html

👀

Quick quality life of upgrade if you have lots of tasks in Codex Tell codex to rename all the tasks in your project according to this article You’ll be able to find stuff again and understand what’s happening!