Nathan Lambert

This is an excellent report on why full RSI / an intelligence explosion is fighting diminishing returns on many fronts, and not yet showing signs of happening. Really recommend reading. I wish I wrote it. I agree with it, but it could end up being wrong! https://www.noahpinion.blog/p/wheres-the-intelligence-explosion

Sigh. I wrote this post below, but then I didn't post it because I am tired of the consistent pushback I get from folks with the same safety worldview as Anthropic. I guess that means I should send it. Here goes! This post is pretty solipsistic and doesn't properly take recent events in cyber risks into it's discussion. It is really hard for me to watch how the US frontier labs like Anthropic don't have the ability to consider other approaches to safety and ways things could play out. It insinuates that Z ai (Chinese lab, builds GLM series) doesn't really care about safety and is reckless to take their business strategy. Saying things like "Given this evidence, we think it's likely both state and non-state actors will use models like GLM-5.3 to cause real-world harm." and "This is unlike any other similarly capable AI model, all of which were released with safeguards or through limited access programs." while closed models have been used on more of the documented cyber attacks is just bowing out of the interesting question. A plausible view is that open model weights and closed model apis (with some safe guards) are both far closer to being easy to mis-use, rather than API models being closer to safe. The trope "Open Dangerous, Closed Safe" may be closer to "Open Unsafe, Closed Unsafe" Closed models have stronger capabilities and stronger safeguards, but the stronger capabilities part could matter more in net harm if both the safeguards are porous. At the same time, as the authors do acknowledge (thanks - thats progress!), open models without extreme cyber guardrails are important to rapidly diffuse cyber readiness in the economy -- as programs like project glasswing are not perfect in getting all critical industry onboarded. There will be more issues like the time when Fable was released and Amazon found a workaround, which allowed them to access the full capabilities of the model served readily at an API. The blog overall is reasonable in it's narrow line, but it's a very effective tool in a complicated, rapidly evolving media ecosystem to reinforce a certain type of safety thinking.

Woah, huge bullish signal for open models that OpenAI feels a pressure to serve users with these models. It's not a world where OpenAI alone can build all the models it's customers need!

The problem for folks like this isn't that they're dumb or anything, Timnit has all the skills of top scientists, rather the information ecosystem and peer groups they live in that don't encourage interrogating thought. Echo chambers are a slow death of clear thinking.

ModelScope barely works outside of China and has few of the models people actually want to use. Unfortunately this slop flew too close to the sun and started impacting how people think about policy. Classic X / tpot.

It's a very hard time for people in AI who want to push on issues, criticize when due, update beliefs, and think in public. I appreciate the few independent voices out there not falling to straw man arguments, tropes, or clickbait delusions. Please keep going.

Current harnesses have a very rigid setup in how the LLM can manage its own context. We show that you can let LLMs edit the context directly - enabling better long-horizon memory, FLOP efficient adaption, & more. Don't fear compaction anymore!

Also iIf you don’t expect to be wrong sometimes in AI, you’re going to have a hard time making takes and predictions that matter.

Today we're unveiling Trillium Labs @trillium_labs, a new non-profit to foster the open science of frontier AI. We're building open post-training recipes and will expand into open infra to study RSI, reward-hacking, multi-agent systems, and whatever comes next. We're built around the theory of change that you need more eyes to solve hard technical problems. We have faith in the scientific methods and communities that humanity has built, and worry that AI is becoming too closed to utilize them. Trilliums are wildflowers that bloom briefly in the spring, before the forest canopies fill out. Though they are small, they lay the foundation for the cycles of growth and nourishment through the rest of the year. At Trillium Labs, the recipes will be the slow nutrients for the seasons and the model releases will be the blooms. Building an institution dedicated to this is needed because, much as nature’s trilliums are slow to expand and grow, the open-ecosystem needs time and dedicated resources to catch up. I co-founded with with a long-time friend and collaborator Tom Zick (@thesezickbeats). We're hiring (full time + student collabs/interns), we're fundraising, and we're looking for compute. Please get in touch if you're interested in helping out. Offices based in the Bay Area and Cambridge MA, remote okay. I’m in the Bay Area until for The Curve and COLM to connect with people who are interested. We’re thankful to have initial support from Halcyon Futures and Schmidt Sciences with more funding en route to enable our ambitions of scaling. Our advisors @Thom_Wolf, @HannaHajishirzi, @gneubig and @ctnzr have been instrumental to building the ecosystem that exists today, and I’m stoked to get to keep working with them.

We're hiring across many roles at Trillium Labs, @trillium_labs, to build the open scientific ecosystem for LLMs. Most important hires right now (ideally with start dates soon): * Full time post-training researchers * Engineers with experience with popular open-source libraries (megatron, MILES, sglang, Tinker, etc.) * Mission-aligned generalists, non-profit ops, etc. * Student researchers I'm also personally interviewing for MATS right now, which will be very aligned research. Please reply, email me, DM me, etc AND submit the form on our website. Excited to build this team. All early hires will work directly with me on all sorts of fun puzzles to rapidly scale our impact.

This weird qwen behavior is going to go viral because no one has read traces from mid-tier open models and realized they're almost all super janky.

About to go on and talk about how open science can still solve some of the hardest problems facing ai safety and diffusion. The most complex engineering problem humanity has undertaken yet.