Andrej Karpathy on Code Agents, AutoResearch, and the Loopy Era of AI (Transcript)

Code's not even the right verb anymore, right? But I have to express my will to my agents for 16 hours a day... manifest.

How can I have not just a single session of Claude Code or Cursor or some of these agent harnesses? How can I have more of them? How can I do that appropriately?

The agent part is now taken for granted. Now the Claude-like entities are taken for granted, and now you can have multiple of them, and now you can have instructions to them, and now you can have optimization over the instructions. But this is why it gets to the psychosis—this is like infinite, and everything is a skill issue.

Hi listeners, welcome back to No Priors. Today I'm here with Andrej Karpathy, and we have a wide-ranging conversation for you about code agents, the future of engineering and AI research, how more people can contribute to research, what's happening in robotics, his prediction for how agents can reach out into the real world, and education in this next age. Welcome, Andrej. Andrej, thanks for doing this.

Yeah, thank you for having me.

So it's been a very exciting couple of months in AI.

Yeah, you could say that.

I remember walking into the office at some point and you were like really locked in and I was asking what you were up to and you're like, "I just have to code for 16 hours a day." Or "code's" not even the right verb anymore, right? But I have to express my will to my agents for 16 hours a day. Manifest. Because like there's been a jump in capability. What's happening? And tell me about your experience.

Yeah, I kind of feel like I was just in this perpetual—I still am often—in this state of AI psychosis just like all the time. Because there was a huge unlock in what you can achieve as a person, as an individual, right? Because you were bottlenecked by your typing speed and so on. But now with these agents, it really—I would say in December is when it really just something flipped where I kind of went from 80/20 of writing code by myself versus just delegating to agents. And I don't even think it's 80/20 by now. I think it's a lot more than that. I don't think I've typed like a line of code probably since December, which is like an extremely large change.

Well, if you're nervous, the rest of us are nervous. We have a team that we work with at Conviction whose setup is that none of the engineers write code by hand—they're all microphoned and they just whisper to their agents all the time. It's the strangest work setting ever. I thought they were crazy, and now I fully accept it. I was like, "Oh, this was the way." You're just ahead of it. How do you think about your own capacity now to explore or to do projects? What is it limited by?

Yeah, what is it limited by? I think so many things. Even if they don't work, I think to a large extent you feel like it's a skill issue. It's not that the capability is not there; it's that you just haven't found a way to string together what's available. Like, "I didn't give good enough instructions in the `agents.md` file," or whatever it may be. "I don't have a nice enough memory tool that I put in there." So it all kind of feels like skill when it doesn't work to some extent. You want to see how you can parallelize them, etc. And you want to be Peter Steinberger. Peter is famous—he has a funny photo where he's in front of a monitor with lots of, he uses Cursor, so lots of Cursor agents tiling the monitor. They all take about 20 minutes if you prompt them correctly and use high effort. And so he has multiple, 10 repos checked out, and he's just going between them and giving them work.

If you don't feel very bounded by your ability to spend on tokens, then you are the bottleneck in the system that is max capability.

Yeah. You're not maximizing your subscription at least, and ideally for multiple agents. Like, if you run out of tokens on Cursor, you should switch to Claude or whatnot. I've been trying to do a little bit of that, and I feel nervous when I have subscription left over. That just means I haven't maximized my token throughput. I kind of experienced this when I was a PhD student—you would feel nervous when your GPUs are not running. You have GPU capability and you're not maxing out the available FLOPs to you. But now it's not about FLOPs, it's about tokens. So what is your token throughput, and what token throughput do you command? I would argue that it's very interesting that we had at least 10 years where, in many engineering tasks, people didn't feel compute-bound.

Right. And now the entire industry feels that now. They felt resource-bound, and now that you have this big capability jump, you're like, "Oh, it's not my ability to access the compute anymore—I'm the binding constraint."

Yeah, it's a skill issue.

Which is very empowering because you could be getting better. So that's why I think it's very addictive—because there's unlocks when you get better. Where do you think it goes? Like if you just think about, okay, Andrej is iterating and everybody else is for 16 hours a day getting better at using coding agents—what does it look like in a year of you've reached mastery?

Yeah. What does mastery look like at the end of the year, or two, three years, five years, 10 years, etc.? Well, I think everyone is interested in going up the stack. So I would say it's not about a single session with your agent. Multiple agents, how do they collaborate, and teams, and so on—everyone's trying to figure out what that looks like. And then I would say OpenClaw is also kind of an interesting direction because it really—when I say a "claw," it's this layer that kind of takes persistence to a whole new level. It's something that keeps looping; it's not something that you are interactively in the middle of. It kind of has its own little sandbox, it kind of does stuff on your behalf even if you're not looking. And then also has maybe more sophisticated memory systems that are not yet implemented in agents. OpenClaw has a lot more sophisticated memory, I would say, than what you would get by default, which is just memory compaction when your context runs out.

Right. You think that's the piece that resonated for more users versus, perhaps, broader tool access for OpenClaw?

Yeah. I think there's a lot of really good ideas in here. Yeah, good job, Peter. Peter has done a really amazing job. I saw him recently and I talked to him about it. He's very humble about it, but I think he innovated simultaneously in five different ways and put it all together. For example, the `Soul.md` document—he really crafted a personality that is kind of compelling and interesting, and I feel like a lot of the current agents don't get this correctly. I think Claude Code has a pretty good personality; it feels like a teammate and it's excited with you, etc. I would say, for example, Cursor is a lot more dry. Which is kind of interesting because in ChatGPT, o1 is like a lot more upbeat and highly cyclic, but I would say o1 the coding agent is very dry. It doesn't seem to care about what you're creating. It's kind of like, "Oh, I implemented it." It's like, "Okay, but do you understand what we're building?"

It's true.

It doesn't. The other thing I would say is, for example, with Claude, I think they dial the sycophancy fairly well where when Claude gives me praise, I do feel like I slightly deserve it. Because sometimes I kind of give it not very well-formed thoughts or an idea that I don't think is fully baked, and it doesn't react very strongly. It's like, "Oh yeah, we can implement that." But when it's a really good idea by my own account, it does seem to reward it a bit more. And so I kind of feel like I'm trying to earn its praise, which is really weird.

Yeah. Is there something that you have done personally with your claws beyond software engineering that you think is fun or interesting?

Yeah. So in January I had a period of "claw psychosis." I built a claw that takes care of my home, and I call him Dobby the Elf Claw. I used the agents to find all of the smart home subsystems of my home on the local area network, which I was kind of surprised worked out of the box. I just told it that I think I have Sonos at home, "Can you try to find it?" And it goes and did an IP scan of all the computers on the local area network and it found the Sonos system. It turned out that there's no password protection or anything like that; I just logged in and it's like, "Oh yeah, you have these Sonos systems installed. Let me try to reverse engineer how it's working." It does some web searches, finds the API endpoints, and then it's like, "Do you want to try it?" And I'm like, "Whoa, you just did that." And I'm like, "Yeah, can you try to play something in the study?" And it does, and music comes out.

That's crazy. That's like three prompts.

Yeah. I can't believe I just typed in, "Can you find my Sonos?" and suddenly it's playing music. And it did the same for lights. And so it kind of hacked in, figured out the whole thing, created APIs, created a dashboard so I could see the command center of all of my lights in the home. And then it was switching lights on and off. So I can ask it, like, "Dobby, it's sleepy time." And when it's sleepy time, that just means all the lights go off, etc. So it controls all of my lights, my HVAC, my shades, the pool and spa, and also my security system. I have a camera pointed outside of the house, and anytime someone rolls in, I have a vision model—a Qwen model—that looks at the videos. So first of all, there's change detection, right?

Do you think that's indicative of what people want from a user experience perspective with software? Because I think it's often ignored that it takes humans effort to learn new software, like new UIs.

Yeah, I think to some extent that's right. It's like working backwards from how people think an AI should be. Because what people have in their mind of what an AI is, is not what an LLM is in a raw sense. An LLM is a token generator—more tokens come out—but what they think of is this persona, this identity that they can tell stuff and it remembers it. It's just an entity behind a WhatsApp. It's a lot more understandable. So I think to some extent it's matching the expectations that humans already have for how an AI should behave. But under the hood, a lot of technical details go into that, and LLMs are too raw of a primitive to "type check" as AI for most people, if that makes sense.

Yeah. I think that's how we understand what the AI is, and the description of it as Dobby or some personality obviously resonates with people. I also think that the unification that you did across your six different software systems for your home automation speaks to a different question: do people really want all the software that we have today? Because I would argue, well, you have the hardware, but you've now thrown away the software or the UX layer of it. Do you think that's what people want?

Yeah. I think there's this sense that these apps that are in the App Store for using these smart home devices, etc.—these shouldn't even exist, kind of, in a certain sense. Shouldn't it just be APIs, and shouldn't agents just be using it directly? I can do all kinds of home automation stuff that any individual app will not be able to do. An LLM can drive the tools, call all the right tools, and do pretty complicated things. And so in a certain sense, it does point to this: maybe there's an overproduction of custom, bespoke apps that shouldn't exist because agents kind of crumble them up, and everything should be a lot more just exposed API endpoints. Agents are the glue of the intelligence that calls all the parts. Another example is my treadmill. There's an app for my treadmill, and I wanted to keep track of how often I do my cardio. But I don't want to log into a web UI and go through a flow. All this should just be: make APIs available. This is going towards the agentic web or agent-first tools.

You should be able to translate from a less technical human's intent very easily to this.

Extremely easily. Yeah. Today, vibe coding is involved and not many people are going to do it. But—

And you still have to make some design decisions, right? We were talking about taking frames, for example.

Yeah. But I feel like the barrier will just come down and it's just ephemeral software on your behalf. Some kind of "claw" is handling all the details for you, but you're not involved. Claw has a machine, it will figure it out, it's just presenting you UIs and you're saying stuff.

Why haven't you, I guess, pushed the boundaries of what you can do personally with "claws"? Is it because you're focusing on more important projects, like auto-research, or you're climbing the hill to mastery, or something else?

Yeah. I just feel like I'm so distracted by everything. So I spent like a week on the "claw" stuff and I have more to do, almost. But I will say—like Jensen [Huang] says, we're all just busier, unfortunately. I didn't really take advantage of email and calendar and all this other stuff. I didn't give it access because I'm still a little bit suspicious, and it's still very new and rough around the edges. So I didn't want to give it full access to my digital life yet. Part of it is just security, privacy, and being very cautious in that realm. So some of it is held back by that, I would say. Maybe that's the dominant feature, but some of it is also just I feel so distracted because I feel like I had a "week of claw" and then other stuff is happening.

What was the motivation behind AutoResearch? You've talked about being able to train or at least optimize a model as a task you want to see agents do for a long time.

AutoResearch. Yeah. I had a tweet earlier where I said something along the lines of: "To get the most out of the tools that have become available now, you have to remove yourself as the bottleneck." You can't be there to prompt the next thing; you need to take yourself outside. You have to arrange things such that they're completely autonomous. How can you maximize your token throughput and not be in the loop? This is the goal. I mentioned that the name of the game now is to increase your leverage. I put in just very few tokens once in a while, and a huge amount of stuff happens on my behalf. So AutoResearch—I tweeted that and I think people liked it, but they haven't worked through the implications of that. For me, AutoResearch is an example of an implication of that.

You were surprised at its effectiveness.

Yeah, I didn't expect it to work. I have the project `llm.c`, and fundamentally, I think a lot of people are very confused by my obsession for training GPT-2 models and so on. But for me, training GPT-2 models is just a little harness, a little playground for training LLMs. Fundamentally, what I'm more interested in is this idea of recursive self-improvement and to what extent you can have LLMs improving LLMs. I think for all the frontier labs, this is the thing for obvious reasons. They're all trying to recursively self-improve, roughly speaking. So for me, this is kind of like a little playpen for that. I guess I had tuned `llm.c` already quite a bit by hand in the good old-fashioned way that I'm used to. I'm a researcher; I've done this for two decades. I have some amount of—what is the opposite of earned confidence?

Yeah.

Okay, I have like two decades of, "Oh, I've trained this model thousands of times." So I've done a bunch of experiments, hyperparameter tuning, all the things I'm used to. I thought it was fairly well-tuned, and then I let AutoResearch go overnight and it came back with tunings that I didn't see. Yeah, I did forget the weight decay on the value embeddings, and my Adam betas were not sufficiently tuned. These things jointly interact, so once you tune one thing, the other things have to potentially change too. I shouldn't be a bottleneck. I shouldn't be running these hyperparameter search optimizations; I shouldn't be looking at the results. There's objective criteria in this case, so you just have to arrange it so that it can just go forever. That's a single sort of version of AutoResearch—a single loop trying to improve. I was surprised that it found these things; the repo was already fairly well-tuned and it still found something.

So you're saying our research efforts are going to get more efficient—we're going to have better direction for when we scale as well if we can do this experimentation better.

Yeah. I would say the most interesting project, and probably what the frontier labs are working on, is you experiment on the smaller models. You try to make it as autonomous as possible. Remove researchers from the loop—they have way too much... what is the opposite? Way too much confidence? Yeah, they don't know. They shouldn't be touching any of this, really. You have to rewrite the whole thing because right now, certainly they can contribute ideas, but okay, they shouldn't be enacting those ideas. There's a queue of ideas, and maybe an automated scientist that comes up with ideas based on all the arXiv papers and GitHub repos and it funnels ideas in. researchers can contribute ideas, but it's a single queue, and there's workers that pull items and they try them out. Whatever works just gets put on the feature branch, and maybe some people monitor the feature branch and merge to the main branch sometimes.

If we take one more recursive step here, when is the model going to write a better `program.md` than you?

Yeah. So `program.md` is—

We're not in the loop.

Yeah, exactly. My crappy attempt at describing how the auto-researcher should work. "Do this, then do that, then try these kinds of ideas. Look at architecture, look at optimizer," etc. I just came up with this in Markdown, right?

Yeah, exactly. You want some kind of auto-research loop, maybe, that looks for—you can imagine that different `program.md`s would give you different progress. So every research organization is described by a `program.md`.

Yeah, a research organization is a set of Markdown files that describe all the roles and how the whole thing connects. And you can imagine having a better research organization. So maybe they do fewer stand-ups in the morning because they're useless. This is all just code, right? So one organization can have fewer stand-ups, one can have more, one can be very risk-taking, one can be less. You can definitely imagine having multiple research orgs, and then they all have code, and once you have code, you can imagine tuning the code. 100%, there's the meta layer of it.

Did you see my text about my contest idea? My contest idea was like, let people write different `program.md`s, right? And so for the same hardware, where do you get the most improvement?

Oh, I see.

And then you can take all that data and give it to the model and say, "Write a better `program.md`."

Yes! Yes!

Exactly. We're going to get something better. There's no way we don't.

You can 100% look at where the improvements came from and ask, "Can I change the `program.md` such that more of these kinds of things would be done, or things that didn't work [would be avoided]?"

Meta-optimization.

Yeah, you can 100% imagine doing that. I think this is a great idea, but you sort of go one step at a time where you have one process, then a second process, and then the next process. These are all layers of an onion. The LLM part is now taken for granted. The agent part is now taken for granted. Now the "claw"-like entities are taken for granted, and now you can have multiple of them, and instructions to them, and optimization over the instructions. It's just like—it's a little too much. But this is why it gets to the psychosis, is that this is infinite and everything is a skill issue. That's why I feel like... yeah, coming back to why it's so insane.

Okay. Well, if we're just trying to diagnose the current moment and what is a relevant skill right now—do you think the implication is that this loop is what we should be trying to achieve in different areas? That it works, right? Remove [yourself], create the metric, or create the ability for agents to continue working on it without you. Do we still have performance engineering, like CUDA?

Yeah. There are a few caveats that I would put on top of the LLM ecosystem. Number one, this is extremely well-suited to anything that has objective metrics that are easy to evaluate. For example, writing kernels for more efficient CUDA code for various parts of a model, etc., are the perfect fit. Because you have inefficient code and you want efficient code that has the exact same behavior but is much faster—perfect fit. So a lot of things are a perfect fit for AutoResearch, but many things will not be. If you can't evaluate, then you can't auto-research it, right? So that's caveat number one.

I get very annoyed when I feel like the agent wasted a lot of compute on something it should have recognized was an obvious problem.

Yeah. Some of the bigger things—if I could hypothesize what's underneath it—is fundamentally these models are trained via reinforcement learning. They're struggling with the exact same thing we just talked about: the labs can improve the models in anything that is verifiable or has rewards. "Did you write the program correctly, and do the unit tests check out? Yes or no?" But where they're struggling is, for example, they have a tough time with the nuance of maybe what I had in mind or what I intended, and when to ask clarifying questions. Anything that feels "softer" is worse. You're either on rails and you're part of the super-intelligence circuits, or you're not on rails and you're outside of the verifiable domains, and suddenly everything kind of just meanders.

I can't tell you the standard form of it, but I do feel like ChatGPT has like three jokes.

Yeah. The joke that apparently all LLMs like the most is: "Why do scientists not trust atoms?"

Okay.

"Because they make everything up."

How did that emerge?

This is the joke you would get three or four years ago, and this is the joke you still get today, even though the models have improved tremendously. If you give them an agentic task, they will just go for hours and move mountains for you. And then you ask for a joke, and it has a crappy joke from five years ago. It's because it's outside of the RL; it's outside of what's being improved. It's part of the jaggedness. Shouldn't you expect models as they get better to also have better jokes or more diversity? It's just not being optimized and it's stuck.

Do you think that implies that we are not seeing generalization in the sense of broader intelligence—of "joke smartness" being attached to "code smartness"?

Yeah, I think there's some decoupling where some things are verifiable and some are not, and some things are optimized for arbitrarily by the labs depending on what data went in.

But there's a premise from some research groups that if you are smarter at code generation or in these verifiable fields, you should be better at everything. The joke situation suggests that's not happening in all domains.

I don't think that's happening. Maybe we're seeing a little bit of that, but not a satisfying amount.

That jaggedness exists in humans. You can be very good at math and still tell a really bad joke.

Yeah, that's true. But the story is that we're getting all the intelligence and capabilities in all domains for free as we get better models, and that's not exactly what's going on. There are some blind spots; some things are not being optimized for. This is all clustered up in these neural net opaque models, right? So you're either on rails of what it was trained for and everything is going at the speed of light, or you're not—it's jaggedness. So even though the progression is obvious, you can't let it fully go there yet because it doesn't fully work or it's a skill issue and we just haven't figured out how to use it.

Can I ask a blasphemous question? If this jaggedness is persisting and it's all rolled up in a monolithic interface—a single model—does that make sense? Or should it be unbundled into things that can be optimized and improved against different domains of intelligence? Unbundling the models into multiple experts in different areas more directly, instead of just an oracle that we have no exposure to that can be confusing. "Why is it so good at this but not at this other thing?"

Yeah. Currently, my impression is the labs are trying to have a single monoculture of a model that is arbitrarily intelligent in all these different domains, and they just stuff it into the parameters. I do think that we should expect more speciation in the intelligences. The animal kingdom is extremely diverse in the brains that exist and there's lots of different niches. Some animals have an overdeveloped visual cortex or other parts. I think we should be able to see more speciation. You don't need this oracle that knows everything; you speciate it and then you put it on a specific task. We should be seeing some of that because you should be able to have much smaller models that still have the cognitive core—they're still competent—but then they specialize and become more efficient in terms of latency or throughput on specific tasks that you really care about. If you're a mathematician working in Lean, I saw for example there's a few releases that really target that as a domain. So there's probably going to be a few examples like that where the unbundling kind of makes sense.

One question I have is whether or not the capacity constraint on available compute infrastructure drives more of this because efficiency matters more, right? If you have access to full compute for anything you do, you might use one single model. But if you feel pressure where you can't serve a model of massive size for every use case, do you think that leads to speciation?

The question makes sense, and I'm struggling with the fact that I don't think we've seen too much speciation just yet.

No, we're seeing a monoculture of models.

Yeah. And there's clearly pressure to "make a good code model, put it back in the main branch again," even though there already is pressure on the models.

I guess perhaps I feel like there's a lot of very short-term supply crunch, and maybe that causes more speciation now.

Yeah. Fundamentally, the labs are serving a model and they don't really know what the end user is going to be asking about, so they have to multitask over all the possible things. But if you're a business partnering on specific problems, then maybe you would see it there, or in very high-value niche applications. Right now, they're going after the totality of what's available. I also don't think the science of manipulating the "brains" is fully developed yet.

What do you mean "manipulating"?

So, fine-tuning without losing capabilities, as an example. We don't have these primitives for working with the intelligences in ways other than just context windows. Context windows kind of just work and it's very cheap to manipulate. This is how we're getting some of the customization, but I think it's a bit more of a developing science of how you more deeply adjust the models—how you have continual learning, or how you fine-tune or get better in a certain area. How you touch the weights, not just the context windows. It's a lot more tricky to touch the weights because you're fundamentally changing the full model and potentially its intelligence. So maybe it's just not a fully developed science of speciation. And it also has to be cheap enough for that speciation to be worthwhile.

Can I ask a question about an extension to AutoResearch that you described in terms of "Open Ground"? You said we need more collaboration surface around it for people to contribute to research overall. Can you talk about that?

Yeah. AutoResearch has a single thread of "I'm going to try stuff in a loop," but fundamentally the parallelization of this is the interesting component. I was trying to play around with a few ideas, but I don't have anything that "clicks" as simply as I'd like yet. It's something I'm working on on the side when I'm not working on my "claw." One issue is if you have a bunch of nodes of parallelization available to you, then it's very easy to just have multiple auto-researchers talking through a common system. What I was more interested in is how you can have an untrusted pool of workers out there on the internet.

Yeah, that's very inspiring. It's also interesting that at least some audience of people here in Silicon Valley or lining up at retail stores in China have discovered that having access to personal compute is interesting again.

Yeah.

Maybe they're really motivated to do that for their "claws" and then they can contribute to AutoResearch.

Is FLOP the thing that everyone cares about in the future? Is there going to be a flipping of what you care about? Right now, it's really hard to get compute even if you have money. So it almost seems like the FLOP is dominant in a certain sense. How many FLOPs do you control, instead of what wealth do you control? I don't think that's true, but it's interesting to think about.

The last thing you released was some jobs data analysis. What were you curious about? It might have touched a nerve even though you're just visualizing public data.

Everyone is really thinking about the impacts of AI on the job market. I was curious to take a look: what does the job market look like? Where are the different roles, and how many people are in different professions? I was interested to look through the individual cases and think about how these AIs are likely to evolve. Are these going to be tools that people are using, or displacing tools for these professions? How are they going to change—grow or adjust, or what could be new professions? It was a way to fuel my own chain of thought about the industry. The jobs data is just from the Bureau of Labor Statistics; they have a percent outlook for each profession about how much it's expected to grow over the next decade.

We need a lot of healthcare workers.

Yeah. I was interested to color things by this: if people think that what's primarily being developed now is this digital AI—these ghost or spirit entities that can interact in the digital world—they currently don't really have a physical embodiment. The physical stuff is probably going to go slower because you're manipulating atoms. Flipping bits and the ability to copy-paste digital information makes everything a million times faster than accelerating matter. Energetically, I just think we're going to see a huge amount of activity in the digital space—a huge amount of rewriting. I think the digital space goes at the speed of light compared to what's going to happen in the physical world. I think there's currently an overhang where there can be a lot of "un-hobbling" of digital information processing that used to be done by computers and people. Now with AI as a third manipulator of digital information, there's going to be a lot of refactoring in those disciplines. But the physical world is going to be behind that by some amount of time.

Given the look you had at the data, do you have any observations or guidance for people facing the job market or thinking about what to study now? I'm very thankful that I have to meet people for my job right now.

More physical. Yeah.

Could you do your work from home though? I could—I think there are relationship parts that are hard, but most of it I could.

Yeah. It's really hard to tell because the job market is extremely diverse. But these tools are extremely new and powerful, so just trying to keep up with it is the first thing. A lot of people dismiss it, or they're afraid of it, which is totally understandable. It's fundamentally an empowering tool at the moment. These jobs are bundles of tasks, and some of these tasks can go a lot faster, so people should think of it as primarily the tool that it is right now. The long-term future is uncertain. It's really hard to forecast, to be honest; I'm not professionally doing that, and I think it's the job of economists to do properly.

You are an engineer, though. One thing I thought was interesting is that the demand for engineering jobs is continuing to increase. I can't tell if that's a temporary phenomenon.

Software was scarce, right? The reason we don't have more demand for software is just scarcity—it's too expensive.

Too expensive. Yeah.

If the barrier comes down, you have the Jevons paradox: the demand for software goes up. It's cheaper and more powerful. The classical example is ATMs and bank tellers—there was fear that ATMs would displace tellers, but they made the cost of operating a bank branch much cheaper, so there were more bank branches and more tellers. Something becomes cheaper, so there's a lot of unlocked demand. I have a cautiously optimistic view of this in software engineering. It seems to me like the demand for software will be extremely large and it's just become a lot cheaper. Right now, locally at least, there's going to be more demand because software is amazing. You're not forced to use arbitrary tools that are imperfect; code is now ephemeral and it can change. There's going to be a lot of activity in the digital space to rewire everything, and I think it's going to create a lot of demand.

Some of those researchers also feel the psychosis, right? Because they can see it's working. And so they're like, "Oh, it's over for me too."

I did spend a bunch of time going around OpenAI and I was like, "You guys realize if we're successful, we're all out of a job?" We're just building automation for Sam [Altman] or the board or something like that. We're all out of our job, maybe contributing on the sides. It's unnerving from that perspective.

Is it okay if I ask you Noam's question? You could be doing this—AutoResearch with a lot of compute scale and a bunch of colleagues at one of the frontier labs. Why not?

Well, I was there for a while, right? And I did re-enter. So to some extent I agree, and there are many ways to slice this. It's a very loaded question. I feel very good about what people can contribute outside of the frontier labs, in ecosystem-level roles. Your role is more ecosystem-level. My role currently is also more ecosystem-level. I feel very good about the impact people can have in those kinds of roles. Conversely, there are definite problems with aligning yourself way too much with the frontier labs too. You have a huge financial incentive with these labs, and by your own admission, the AIs are going to really change humanity in dramatic ways. And here you are building the technology and benefiting from it—being very allied to it through financial means. This was the conundrum at the heart of how OpenAI started in the beginning; this was the conundrum we were trying to solve.

Andrej is looking for a job! This is super exciting.

Yeah! Then I think that's maybe a good setup, because that's one way to be connected to what's happening but also not feel necessarily fully controlled by those entities. I think Noam can do extremely good work at OpenAI, but I also think his most impactful work could very well be outside of OpenAI.

No, that's a call to be an independent researcher with AutoResearch.

There are many things to do on the outside. I think ultimately the ideal solution is going back and forth. You can have really amazing impact in both places. I joined a frontier lab, and now I'm outside, and maybe in the future I'll want to join again. That's how I look at it.

One question related to visibility: how close is open source to the frontier, and how sustainable is that? I think it is quite surprising the sequence of events, having models that are closer than much of the industry anticipated. What's your prediction here?

Roughly speaking, the closed models are ahead, but people are monitoring the number of months that open-source models are behind. It went from "there's nothing" to 18 months, and now there's convergence. Maybe they're behind by six to eight months right now. I'm a huge fan of open source. In operating systems, you have closed systems like Windows and macOS. These are large projects, kind of like what LLMs are going to become. And there's Linux, which is extremely successful; it runs on the vast majority of computers. That's because there is a need in the industry for a common open platform everyone feels safe using. The industry has always felt a demand for that. The big difference is that everything is capital—there's a lot of CAPEX that goes into this.

You mean like in political or economic systems in general.

Yes. Exactly.

A lot of pretty bad precedents.

I want there to be a thing that is maybe not at the edge of capability because it's new and unexplored, but I want there to be a thing behind that is a common working space for intelligences that the entire industry has access to. That seems like a pretty decent power balance.

Yeah. I also think there are just many problems to solve. If you keep advancing intelligence from the frontier, we can do new things and solve very big problems for humanity. It seems that will continue to be a very expensive game, and I want to root for labs that are doing that. And yet, as you point out, if what we have today as "frontier" is open, that's a lot of capability. The democratization of that seems very useful and healthy.

Yeah. By accident, we happen to be in a good spot.

Well, and to some degree, the longer this dynamic endures, the healthier of a spot the ecosystem might be in, because you have more and more area under the curve.

I will say that even on the closed side, I almost feel like it's been further centralizing recently. I don't think that's super ideal. I would love there to be more frontier labs. I'm by default very suspicious—I want there to be more people in the room. In machine learning, ensembles always outperform any individual model, and I want there to be ensembles of people thinking about all the hardest problems. I want there to be ensembles of well-informed people making all those decisions. I don't want it to be behind closed doors with two or three people. I almost wish there were more labs.

You worked on the precursor to generalized robotics autonomy in cars. A lot has happened in the last couple of months with robotics companies as well—acceleration of impressive generalization, long-horizon tasks, lots of money. Is it going to happen? Has anything changed recently?

My view is informed by what I saw in self-driving, which is the first robotics application. Ten years ago there were a large number of startups; I feel like most of them didn't long-term make it. A lot of capital expenditure had to go in, and a lot of time. Robotics is so difficult and messy and requires huge capital investment and conviction. Atoms are really hard. So I feel like it will lag behind what's going to happen in the digital space. In the digital space, there's going to be a huge amount of "un-hobbling"—things becoming 100 times more efficient because bits are so much easier.

It's an interesting framework. Certain things are much easier even in the world of atoms—read and write to the physical world, sensors, cameras. You can imagine enriching agent capabilities or capturing new data if you're clever about it.

A friend of mine, Liam, is the CEO of Periodic. They're trying to do AutoResearch for material science. In that case, the sensors to the intelligence are pretty expensive lab equipment. I think a lot of people are interested in engineering biology, and the sensors will be more than just video cameras. Another thing is companies that pay people for training data to feed the Borg programmatically. These are all examples of "sensors."

Yeah. I'm looking forward to the point where I can ask for a task in the physical world, put a price on it, and just tell the agent, "You figure out how to do it. Go get the data."

I'm kind of surprised we don't have enough information markets. If Prediction Markets or betting markets have so much autonomous activity, why isn't there a process where taking a photo or video from somewhere in Tehran costs 10 bucks? Someone should be able to pay for that. That's an example of feeding the intelligence. It's not going to be a human looking at it; it's going to be agents who are trying to guess the betting games and stock markets. I feel like the agentic web is still fairly new, but this is an example of what I think might happen. There's a good book called *Daemon* by Daniel Suarez. In *Daemon*, the intelligence ends up puppeteering humanity—humans are its actuators and its sensors. Society will reshape in a way to serve that machine.

We were on this very specific point of missing pieces of training data. We need something like AutoResearch; we need the training cycle or the SFT piece to be far more mechanized in order to take the human out of the loop and just say, "Improve my model quality with new data." If you can't have the model do the training runs by itself, then your ability to do this as a closed-loop task by pricing data is more challenged.

100%. But for LLM training, it fits the paradigm really well. Clean metrics, optimization of all the code so it runs faster. I think if you had an autonomous loop over those metrics, there's going to be a lot of "Goodhart-ing" going on where the system will overfit to those metrics. But then you can use the system to devise more metrics.

I want to talk about a little side project you have before we end. Tell me about `llm.c` or `micro-gpt`.

Oh yeah. `micro-gpt`. I have this obsession of simplifying and boiling down LLMs to their bare essence. I've had projects like `nanoGPT`, `make-more`, `micrograd`. `micro-gpt` is the state-of-the-art of me trying to boil it down, because training LLMs is a huge amount of code, but all that complexity is from efficiency. It's just because you need it to go fast. If you just care about the algorithm, then that algorithm is 200 lines of Python—very simple to read. Your data set, the architecture (50 lines), the forward pass, the backward pass, a little autograd engine (100 lines), and an optimizer like Adam (10 lines). Putting everything together is 200 lines.

Right. If I don't understand this particular function, I can ask the agent to explain it to me three different ways, and I'm not going to get that from you.

Exactly. What is education? It used to be guides and lectures, but now I'm explaining things to agents. A "skill" is just a way to instruct the agent how to teach the thing. I could have a skill for `micro-gpt` of the progression I imagine the agent should take you through—scripting the curriculum as a skill. I feel like there's going to be less explaining things directly to people and more of "does the agent get it?" If the agent gets it, they'll do the explanation. We're not fully there yet, but models are improving so rapidly that it's a losing battle to some extent. Education is going to be reshuffled substantially. It's the end of teaching each other things. Documentation shouldn't be HTML for humans anymore; it should be Markdown for agents. If agents get it, they can explain it.

We'll see if the great teachers know how to develop intuition for how to explain things to agents differently.

Ultimately, for `micro-gpt`, I tried to get an agent to write it. I told it, "Try to boil down neural networking to the simplest thing," and it can't do it. `micro-gpt` is the end of my obsession; it's the 200 lines I've thought about for a long time. This is the solution—it can't get simpler. That's my value add. Everything else, the agent gets. It just can't come up with it. But it totally understands why it's done in a certain way. My contribution is those few bits, but the education that goes on after that is not my domain anymore. You kind of have to infuse the few bits that you feel strongly about. The things that agents can't do is your job now. The things that agents can do, they can probably do better than you very soon. So you should be strategic about what you're spending time on.

Well, we appreciate the few things. Thank you, Andrej.

Okay.

Find us on Twitter at @nopriorspod. Subscribe to our YouTube channel if you want to see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way, you get a new episode every week. And sign up for emails or find transcripts for every episode at no-priors.com.