The best AI agents cost less than you think (Transcript)

We're talking about tens, hundreds of millions of dollars, even in a single enterprise. I think math is going to win here.

Today I'm talking to E No Reyes, co-founder and CTO of Factory, the company that's turning signal into deployed code inside some of the world's biggest engineering works.

A great model independent harness is making the models better. And so if your harness is able to find great workflows for solving problems, then you can bring that to every model that you use.

He explains why some of the labs might be trying to stop this from happening.

I think the model providers might know this, and so they're going down the route of regulatory capture in order to prevent the inevitable, which is that the shape of intelligence will be uncovered, and then everyone will have access to it.

We get into how he thinks engineering as a role is changing.

We have an engineer, his name is Alvin, who spends almost all of his time effectively building like lint and type check rules to prevent our agents from making increasingly like complex mistakes. It makes me very bullish on like humans in the loop for a very long time.

And throughout our conversation, E No shares some of the ambitious projects Factory is working on.

There's an even more like general meta workflow that an agent can use to discover the right workflow to solve almost any knowledge work problem.

Welcome to Max Agency, the podcast that goes deep into how the best agents are being built by builders like you. One term that I've heard recently that I don't fully understand, so I'm hoping you'll help me understand, is this idea of like a software factory. And you guys are named Factory, and I think you I've seen you guys tweeting about this concept. So, what is it and how do you guys approach it?

Yeah, totally. And I think this term, it's funny cuz when we started, the goal really was to say what would it look like to have this sort of 24/7 autonomous factory producing the software that you wanted. And and really what has become more concrete over the last 3 and 1/2 years is like there might be a data model, there might be a process, there might be, in fact, a methodology around building software in this sort of like factory-like way. We like to say that it starts with this process of signals. Right, like all software today is implicitly created by humans who absorb all these signals from the world. Like that could be internal conversations, customer reports, bugs. And they take those signals and they prioritize them, they triage them. They'll put them in places like Jira or Confluence or Linear. They take those and then they make plans and then those plans become code changes and they get validated, they get reviewed, tested. Ultimately, that deployed software that they produce goes and generates even more signals. And so this feedback loop has been implicitly orchestrated by us for quite a while. And when you think about what it would look like for AI to allow a signal to flow all the way to deployed change without a human intervening, I think that process begins to open up like the idea of, well, if that flows, first of all, how do you get it to flow? What are What are the automations involved? What do the humans do, right? Probably maintaining and evolving this software factory. And so that sort of like idea of signal to deployment is really what we're trying to help instrument for a given company.

When you talked about those signals, my mind went immediately to like the role of product. Like seeing these signals, thinking about things, deciding what to build. I've also seen some companies trying to think about like create there's all these coding agents out here. Like what does a version of that for like a product person look like? How do you think about the relationship between like product and engineering? And like are humans still involved in product? Is that where they're in the loop? Is there a role for like coding agents but for product? Or is that the the same as a coding agent or a software factory? Yeah. I guess are these just different things in the software factory?

No, it's it's a good question cuz I think that the And this has happened a couple of times. Like I think when we move to waterfall, when we move to Agile, like sort of new roles evolve and maybe I don't know if there's going to be like a Scrum Master equivalent for the software factory.

I was going to ask like what are the roles in the software factory?

Yeah, but I think that what seems very clear is that a lot of companies are sort of unbundling the traditional product manager role and but importantly that doesn't mean that product as a goal, a role, and a function is going away. In fact, I think quite the opposite. Like what I've seen is that in the world where planning was super critical and you didn't have like direct access to the signals the moment they're happening, human beings had to encode this prioritization and this almost like this process of deciding what to build in their heads. And there's like a certain type of person who's really good at encoding that and then building plans around it. And I think what the software factory sort of changes is it condenses the time from like the signal to the change. And what that means is that you almost need people who are just as good at deciding sort of like what that should be, what that prioritization should be, but they also need to be in charge of stewarding it further than just like okay, now this is in the hands of like the engineer of the change. So for us internally like we have this concept of what we call product owners. These are traditionally engineering backgrounds, but they would easily, you know, be product managers if they wanted to be. And so I think this role of like someone who is deeply aware of how to prioritize and build a product in a customer empathetic way, like that to me is not going away. In fact, I think it might be easier for product managers to learn how to become technical than for folks who do not have that sort of like product taste to who might be technical to like become product owners.

And why do you need to do both now?

Well, I think that there almost is no difference between deciding what should be built and that thing being built. And and that sort of like squashing of the role means that you can have one person who is both deciding what should be built and observing or monitoring its creation. But, there is probably another sort of deeply technical role that is more about as you do this process, underneath the hood, the architecture can easily become like sprawling, right? A product can be very elegant, but underneath the hood it can be unbelievably messy. And so, I think that there's also this interesting thing where we see a bifurcation. Where we're bringing on a lot of people who are very product-oriented, and then we're bringing in a lot of people who are extremely performance and architecture-oriented. I don't really know exactly if those are titles that are worth differentiating as much as like traits of people who make great products.

I think I saw one tweet that I liked, which was like there'll be two types of of engineers, like product engineers and then like infrastructure engineers, which I think aligns with that kind of like performance and architecture style thing.

Totally. in like factories, there are actual like process engineers, and they sort of look at the loops of the factory, and they sort of observe where the deltas, the issues. They do a lot of analytics on the process control, and then they go and they make refinements. I think that like that role is what this like performance and infra engineer We have an engineer, his name is Alvin, who spends almost all of his time effectively building like lint and type check rules that are increasingly elaborate and complex to prevent our agents from making increasingly like complex mistakes. and I think that the idea that you can like lint the architecture of a product, I think in the past is not necessarily worth the time, because you can just tell people, but I don't think the agents have that sort of like global local context really nailed, and I don't know if they will in the next like year or two.

So, you guys are building, I presume, a software factory. What does that mean? Like, what practically is involved in that?

Yeah, I think that it starts with being able to like represent this data, right? Like, you have to have some way of saying, I know what a signal is and where those come from. So,

What are these, yeah?

Yeah, for us, like, we have every single customer Slack channel instrumented with triage agents that listen for bugs, feedback, et cetera. We have all of our like GitHubs GitHub issues, our Twitter feeds instrumented. We have internal Slack conversations for the product teams instrumented and all of our telemetry. So, we use like Sentry and all these other tools. All of these get consolidated into a central pipeline where they become signals, and signals then are translated by agents based off of a prioritization document at the product level. So, each of our products has effectively this guidance that says, for our CLI, like, we are building, you know, a couple of opinions, and we've built those opinions over time about we get a lot of requests about billing and about how, for example, bringing additional plans. And then there's stuff on the CLI that's just opinionated where we're like, we are choosing to do this and that. and so, over time we build up this like It's kind of funny. It does become a little bit of a like a master architecture opinion doc of like what you are building. And then agents refine over time by they either make mistakes and then you correct it in the doc or they don't, and you keep going. and then those then get handed off, and this is where like humans are intervening the most. Like, we do not just let the system create changes and merge them. A lot of the time what humans are doing is they're observing these individual changes, and they're saying that is a change, but we don't want to make that change. Or that's a great way of putting it. However, the plan or the spec for that change looks nothing like what I want it to be. And so that's really where our humans get heavily involved in the process. And they're using obviously droid or agent along the way, so they're back and forthing in order to access this. We have agents then that run, they do QA, they do code review, they do security analysis. And ultimately when it gets deployed, all that stuff gets fixed in and like our change log gets automated. So it's this very like interesting. It is clearly human in the loop, but a lot gets automated.

For that like document or set of documents that is describing like what changes you do or don't want or like what are How much of that is like human written versus written by agents?

We started, it was very agent written in the beginning. And I am not totally convinced that it needs to be human written, but for now what we have seen is that there's sort of like an accumulation of individual pieces that can be synthesized in very creative ways that humans spot just much quicker than agents do. So it's not that they can't do it. In fact, we've seen some like amazingly creative like synthesis of you know, you see like 10 rules and then an agent will be like really you're just saying this and we're like, wow, we are just saying that. And but often times that is humans who spot those little like oh, should we just say like for example, no more changes to the rendering. Like the rendering, we know it's a little broken, but we just don't want to change anymore while we think about a better solution. Those sorts of things just obviously need humans today. I It makes me very bullish on like humans in the loop for a very long time.

How big is that document or set of documents?

It's not that large. It's probably less than like 2, 000 lines. It's like fits, fits easily in the context window. I bet you could probably bring it down even more.

You mentioned kind of like setting up all these different signals. When you go work with a customer, how like manual is that? How long tail are the signals that you look for? Is it really bespoke to each application or is there a set of like just off-the-shelf things you can just snap your finger and it's up and running?

Yeah, I think that a lot of what we try to do with when we're working with folks on this is rather than trying to say like there's an analogy that I like to give which is this concept of you know like Epcot, like Disney's Epcot? So, when Disney was creating Epcot, his thing was, I want to create a city of the future as a model for all the other cities to build towards. And so, Epcot was it's flying cars, it's centralized communal transport, you know, air vacuum things. and now it's a theme park because you know, it's just not possible to say, look, this is a city of the future. Now, everybody else please copy this. But, what's interesting is there was definitely quite a lot of inspiration that like actual city designers will often reference. And there's a lot of shared in spell but also just from Epcot directly. Our view is that building like Epcots inside of companies does not really work. You can't say, here's one example of something that's ultra futuristic. Now, everyone else just try and copy that. Instead, we feel that there is an incremental journey to this and it doesn't even start with the signals. It just starts with tooling up some basic automations for SDLC functions that you already have like code review security, automated documentation. Those are very easy to swap in and it requires almost no change from the developer. And if you aggregate like four to five of these, you're like 75% of the way there, right? So, a lot of what we do is more incremental transformation on those known SDLC steps. I think we're still a little bit away from very large orgs just suddenly just transitioning the initial like signal and triage process.

Super interesting. For all these different functions, like the harness the agent underneath, is it similar, different?

Yeah, it's it's the same harness underneath the hood.

What's different?

The workflow. And I think that this is the thing that I skills was like such a funny concept because when I was at Hugging Face, we had this thing called like Hugging GPT, which was effectively a giant. Yeah, you remember this? This is those were the days. I think that's like close to when we first met. And a Hugging Face Slack channel. But Hugging GPT was not that different from skills. It's like, look, you've got this list of things you can call on and the workflow is encoded. At the end of the day like code review versus incident response is a workflow where you say, I know how my harness works really well. And so that harness has a couple of primitives. I know it will run in a loop. I know it'll call tools in a clever way. I know that it can invoke skills. And I also I have a little bit more nuance on weird ways that prompting can affect my specific harness, which you kind of want to avoid in general. You don't want people to have to learn an arcane language, but it it always happens. And so what I think we're trying to do is we're trying to encode as much of the canonical workflow of for example code review while also retaining the fact that other people will want to change that workflow a little bit. So a lot of it is just getting creative and thinking, what if what parts of this workflow are consistent across companies versus what are different? And if you just want to wipe that whole thing and replace code review with your own workflow, you totally could.

And so what are the pieces that define a workflow? Is it prompt, skills, tools?

Internal to the product, it's quite literally just a baked-in skill. And so you can even say something like, hey, let me run we would call it Droid Exec, that's our headless CLI, but you run and just invoke the skill. And then it invokes the skill and it goes through the workflow. The cool thing though is that as you advance your harness, you get to take advantage of different things within this workflow that are quite subtle, right? So, for example, sub-agents. One thing that people some harnesses do have is like variability in the parameters of the sub-agent you spawn. So, I don't just want to spawn a sub-agent, I want to spawn a model that has these properties and that has this reasoning level. And so suddenly, if you want to implement a creator verifier pattern of like, you know, one sub-agent creates, the other verifies with a wall of context between them, if your harness makes it really easy to do that, then you no longer have to build such an elaborate like extra workflow. You can just say to it, just do this thing and the harness like knows, {quote} {unquote}. So, I think that's sort of the interesting thing is you start to find all these like primitives to play with. At the end of the day, it is kind of like engineering. And so, a lot of people don't want to worry about it. and so, baking it all in I think makes it easier for customers. Early on, one of the ways that I described the kind of like coding a software development landscape was that there was like coding agents and then agents that did like everything but the code. And you'd kind of have like documentation writing and there's a bunch of companies that just did kind of like testing.

Yep. I think recently it seems like a lot of that is collapsing, but I'm curious for your view of like, do you see kind of like different focuses within this or is it really just like everyone's going to do everything within the SDLC? And for you guys, how much of a focus is like the core coding agent Droid compared to some of the other things around it?

it definitely does all start at with an amazing like agent core. And we would say that like a model independent agent core is super critical to getting the best performance and cost out of the the tasks that you care about. Once you have that like really fluent agent core, one reason that we like to own that instead of like maybe just using like an open source one is that the tie-in between, I sort of alluded to this, the tie-in between the workflows, your control over the agent loop, and the and the models underneath of it. There is a relationship between these things and you can tweak all three of these parameters a little bit. And our thesis has always been that at the more that you can share in the agent, the better. So, it we started I think one of the first times that we chatted in as well. It was like we had like an agent for coding, an agent for docs, an agent for code review. We did want to have a unified at the time I think we called it cognitive architecture. But that's not how the models work. They just weren't good at that. Now, I think they are good at that. And so, we should try to consolidate as much as possible and then layer on that workflow on top. And I would even argue that it's not just like SDLC, it is very likely from my perspective that the best general agent at effectively all knowledge computer work is effectively what we call coding agents today.

Let's talk about the model agnostic part. So, you guys do not build a model. You work across all different models. Model providers are building coding harnesses of their own. How do you think about competing with them and how do you think about how a developer should choose when to use which ones?

Yeah, I think this is like an interesting question and pretty nuanced cuz there's like an economics game theory sort of play, which is like today, what should developers do? And then there's a longer term, how does this play out? And so, what I would argue is that right now, the clear incentive for the average individual developer, so you're not a part of like even if you work at a company, like you are just buying a subscription for yourself, you can rank order who is going to give me the most money. Like there the subsidies are pretty obvious. And I don't necessarily that's good, bad, I don't really know. It's morally neutral. It's just economics. I don't think that it's reasonable to try and say that like you should not take free money. And that's our part of our hypothesis is we do not necessarily feel that we want to currently sell in a market where there's such large subsidies. it's why we focused on enterprise. I think over a longer period of time, what's interesting is that eventually these prices will have to match the reality of the actual underlying cost. And when they do, I think the true distribution of models and like the Pareto frontier of cost and quality is much closer to a blended situation where certain models at a given price quality point are just the obvious best and it's not one model across the entire like cost quality Pareto frontier. Now, take it to enterprise or like businesses that do have to pay the true price of the of the inference. And that's the current situation. This is why in the enterprise costs have become insane is because in reality frontier models are extremely expensive. And if you can achieve similar performance with a cheaper model, you should. The the problem is that people have not been able to achieve the same performance because a great model independent harness is hard to build. And so a lot of what we're trying to do is we're trying to say, "Look, not only is this going to be potentially even better across the board because the models are spiky in different ways and you can access all the spikes but it will also likely be cost effective." So, in my mind it's kind of a no-brainer. It's like it not only is it more performant, but it is also cheaper. However, that is something that gets sort of changed by economics. and there is an open question of like how long could the subsidies ride? in the enterprise the answer is no more. And so, that has changed.

A lot to dig into. Maybe focusing just on performance and obviously building open source model agnostic harness company myself, I'm biased here, but I'll try to play

Yeah. Yeah. Totally.

Can't you just argue that the Frontier labs are going to fine-tune their harnesses on their models and that's going to be the best way to run them?

Yeah, and I think that this is something that I've sort of I've talked about a bit before where my view from how we run these evals alongside our work with some of these labs is that clearly the harness informs the model in a very interesting way. And what by that is that when you post-train these models, you are building in a gravity towards a workflow that solves the problem very effectively. And when I say workflow, what is just a series of reasoning steps and tool calls. And so what that means is that there is a really what you're doing is there's a workflow that exists that is good. And you make the model more inclined to find that workflow when you do that. There are certain patterns that you can follow in the harness. For example, like how you choose to inject system prompts can make the models feel like it's in one harness or another. And all this stuff is like very public and out there. And so what I would sort of argue is that finding these workflows is the responsibility of the harness. And so if your harness is able to find great workflows for solving problems, then you can bring that to every model that you use. So as an example, code review with I'll use current models with GPT 5. 5 and 5. 6, it's amazing. Code review is really cheap. It's like a dollar 70 roughly in our harness. I think in with a Codex harness, it might be like $2 raw API price. And then for Opus 4. 8, you're looking at $5 to $6 for that same code review. And GPT 5. 5 meaningfully outperforms it. What's interesting is that when you use our harness with Opus, we're following a little bit more of a Codexy code review workflow. We get that price down to like $3-ish and it's much closer. It's still not as good as GPT 5. 5, but it's much closer. I argue that it's just the workflow at play. So, my view is that a great model independent harness is bringing the best of all of these workflows together and making the models better. And the model labs do have an inherent disadvantage in that they're sort of they're on blinders. Like they can only operate on their model. They can't pull other models in to test is the workflow better with a different path. That nuance I think has really changed a lot of the people especially once you join factory and you see how like the sausage is made. I think a lot of people realize that there's not really an obvious advantage to having both the model and the harness.

Presumably they're training on some workflows and it would be the best way to do those workflows. So, is that true? Is that not true? Like what are the are there workflows that they are just like best and are those workflows different for OpenAI and Anthropic?

Yeah, and this is I think one of those things where in order to be super to give like a super empirical answer, I'd have to like go in and look at the harnesses sorry the eval harnesses that they're using. But here's my here's my like gut intuition. The vast majority of the pre-training data is coming from like a shared corpus. The synthetic pre-training is then built off that shared pre-corpuses just data distribution. The actual individual task set or the RL environments that they're building are being sold by an increasingly limited number of companies and the overlap in sort of the breadth of what tasks matter is the union is increasing very rapidly. So, in other words, almost everyone now is going to have a very elaborate harness eval harness for code review. And I think that probably there is like in the latent space of all code review like a very solid workflow that is just like the best workflow.

And should you think like OpenAI just has a better eval harness for code review and that's why their harness is better at it?

If had to guess, they got closer to like the canonical like global minimum of loss on like code review at this point in time. And what that means is that the workflow they've uncovered, you know, these agents they the reasoning traces are not exactly human interpretable, but if you follow the flow of what they do it's like you watch a human who like reads over some things and then does work versus doesn't read over and does the same work. It's like there's certain things that are human interpretable, and I would imagine that what they're finding with all of these things is that a really great model does like a very clever workflow very consistently and builds the nuance of the task into their workflow. And so, we have the benefit as in fact everyone out there has the benefit of like observing this. I have like a spicy take about like models being There's a thing called like the platonic representation hypothesis.

I haven't heard of that. You're going to have to explain it to me, but I love spicy takes, so let's go down this path.

Yeah, and so what it says is when you train multiple models with different architectures on the same data distribution, if they converge towards a very shared representation, but just with differing levels of compute. In other words, all learned algorithms might be very similar in the sort of mechanism that they're unfolding, which is matching the data distribution, and it how your model architecture or what you've built primarily impacts how efficient you are at finding that data distribution. And so, the sort of end result would be if you take model training all the way to its limit, we're building all towards like a platonic representation of all of human knowledge. And that and that already exists out in the ether, and we're sort of discovering it rather than creating something. And this has another interesting thing of meaning that like distillation is effectively unpreventable because once you found the shape of intelligence what it looks like. And so distillation might just be this thing that is not possible to prevent. And that's what's leading and I think the model providers might know this. And so they're going down the route of regulatory capture in order to prevent the inevitable, which is that the shape of intelligence will be uncovered and then everyone will have access to it.

Okay, so this is something I think about as well. Like are we building general-purpose models? Are we building models that are tuned for a particular harness? And I would argue I am curious for your take on this, but like I would argue a year and a half ago, however long, before they started like really post-training on these harnesses. The models were maybe like more similar than they are today. And it seems like today there's some differences in terms of how they work and like what tools they work best with. But it kind of sounds like what you're saying is like over time you expect those differences to go away as we discover like what this platonic ideal is. So like yeah, I'm I'm curious whether you agree with what I said or disagree or I'm getting anything wrong.

Yeah, I think what's funny about this is I see the difference as like and you remember this cuz I remember we once related to this. It's like you used to change like the a comma in the description of like the edit file tool and it would change your performance on a benchmark by like 10%. I think opening I famously had like a sentence you could add to Codex that increased its score on SweepBench by like 30%. And it was like one sentence and it was like like don't stop until you're done. I think that you see a lot less of that and a lot more like obvious spikiness on tasks where like some models just can't do it and others are really good at doing it. And I think that the way I would sort of also analogize this is we went from a world where like you had people who just like some were trained to use keyboards and others were trained to use mouse mice and like others were trained with touchpads. And if you gave the one a keyboard, they just like wouldn't be able to do it. And now you have a world where like all of them are more than competent at using all of those tools. However, like one's a bio PhD and the other's like a math PhD. And so I think that like that it's almost like the tool use. And this is the explicit goal of all of the model. Like they want generalizability. I think that they're going to struggle to fight against generalizability because that is effectively the goal of these are like general learning algorithms. So the more they also converge, like I know that our harness is definitely in the post-training data sets of nearly all of the frontier models. So like the more they generalize across different open harnesses, their own harness at least the way that they communicate this to us is that they want that generalizability. They want more people's unique harnesses to like also perform well. The question is does your harness unlock the workflow that they have decided really matters? like on for example in like bio or in math or in science.

Do you use different prompts or tools for different models or families of models in Android?

We still do. And what we haven't done in a little bit is we haven't rigorously evaluated if bringing them back to a union is ideal yet. right now the way our harness like auto optimizes is we explicitly tell it to find the tool descriptions that map to the models. One thing though that that is I think the noise is gone. Like if we like change a couple paragraphs or we change a sentence we're talking about incremental gains on we do think we have a good representative benchmark. It could be that we're like optimizing a local like it might not matter. and I do think though that different tools for different models does still matter.

Yeah, that's what I was going to say. So so we have deep agents. We get we have what we call like harness profiles which are things for different models.

Yeah.

And I think like the biggest thing that we found kind of to your point like it's not like small things that matter. I think it's like the edit tool is like pretty different from Anthropic and OpenAI. Like one's like I'm going to mess it up, but like one's like patches and one's like

Yeah, apply patch, like edit string, string replace, yeah.

And so it like and that's kind of what I mean by like a year and a half ago, I guess like a year and a half ago, like the smaller things mattered more. But it wasn't like one model was like really good at one of those and really good and and like those aren't like I don't think like those edit strings are like really specific workflows. Those are like tools that you would use, but they're still kind of different between the models.

Well what's interesting is like it kind of is a workflow though because like the way that they error corrects, like edit string sort of error corrects in a different way. And one of those like workflows is better. Like the the

You think one of the string replacement is better?

I think I think that I think Codex's editing is better. Why? On our benchmarks of like the algorithm, like it just performs better on editing that fact is shoutout to Aider, that original like coding CLI, that benchmark is no longer it's totally saturated now. It's like super trained on, I think. But a variant of that we have, we don't run it very often anymore, but at least as a couple months ago, like I think that there is just a better way to do it. But that being said, once you start to build up a corpus of this stuff and also the traces are consistent and then you start scaling that up, like you have to say, well, what if we just scrapped you don't scrap it, but rather you have to then start building up the new algorithm in your post-training from there. So I think it's like just not worth the cost-benefit analysis of making the model good at a ever so slightly better workflow. And so that's I think a great example of finding like of probably a very close like global like minimum. Like they just like know that both these are decently good enough that it doesn't matter. And can one do the other? Yeah, totally. So going to that like yeah, can one do the other? Like, it sounds like there are some differences in terms of like how you guys have your harness now. Maybe you should reevaluate that, maybe not. But, like let's say Anthropic does not adjust theirs and like, " what? We're going to keep it here." Open AI is like, "We're going to keep it here." Do you think you'll continue to need that in a year, in 2 years, in 3 years?

I think that the nature of our own intelligence shows us that we might need to keep that for a very long time. Like, imagine just like being like left-handed right? Like, we just like encode these preferences that are like pretty arbitrary and it's really hard for us to like switch. And some people can like with a ton of effort choose to switch, but I I that's sort of like how I would describe some of these like where like the workflow is it's almost the same, right? And so, I would argue that this is a great example of like what this convergence will look like in practice. It's like it won't be that they do the exact same thing. It'll be that they found two solutions that work so well at solving the problem or multitude, right? There might be like eight solutions that all work and maybe you could argue that one's like 1% or 2% better. I think that this is why at the limit like the models themselves are not the driver of the differentiation cuz you eventually just sort of find these like sort of optimal strategies across a bunch of different tasks. And then if you're able to find that a little bit earlier, maybe there's some alpha, but not for long. So, the model's probably not the driver of the differentiation, but couldn't you argue like you need to use it in that harness to achieve that like near like even if they're close like they'll just you need to use different harnesses for different models.

Only if there is no ability to make a harness that takes all of those into account, right? So, like I would sort of argue that like a lot of what we do is we just sit on top of these models and we look for those local like optima and then we and we compare a lot, too. So, like what's interesting is in our cybersecurity benchmark that we have. It's like a cyber gym plus-plus sort of situation. What we found is that GLM 5. 2 is honestly, it really looks like it might just be better at this task than the other frontier models. However, the harness is seriously holding it back because we're still observing in hill climbing these dumb decisions that it makes that when we sort of investigate, we read the reasoning traces, we see the mod like clearly it is post-trained on the harness in a very different way. That's not enough to overcome us just saying don't do that. So, when we say don't do that, it doesn't. And then it goes and it's way better at solving the problem. And so there And this is like I think this interesting thing where a lot of that is pretty intuitive. Like you read the you read the reasoning trace or well, an agent reads the trace. It sort of builds a hypothesis. It edits the system. And then it goes in and it says like let me run that again. And it you sort of do this for enough and you start to get this feeling that like there's probably these like workflows that exist that no one has uncovered yet that change the impact of the models by like such a significant amount. and they like no one's uncovered them yet, right? They just they exist in the latent space but haven't been found. So, yeah, I think I think that is where a lot of alpha is to be found. Not all of it. I think also like obviously the difference between like Opus and Fable is huge. So, like clearly model scale as well. So, hopefully we have people who keep investing in like all of the above. but I don't think that you like need to invest in the entire stack to see alpha or outperformance.

It sounds like one of the things that you're talking about and correct me if I'm wrong, but it sounds like one of the things you're talking about is running some sort of process to hill climb on like what the peculiars of the harness should be with relative to a particular model. what it like what do those peculiarities end up looking like? Is it Is it the prompt, the tool descriptions, the the tool arguments? And what does that hill climbing process look like for you guys?

This is the origin of the concept of signals for us is we were seeing so much of these we built the system called signals and this is like a much older version than what we have today. But what it did is it look it had a very lightweight model on top of the actual agent just sort of look at the agent's activity and then in online like on in real environments sort of self-assess did I complete that successfully? What issues did the user encounter? Did I make mistakes that the user had to correct me on? And we would then aggregate and anonymize the underlying activity but send up like I consistently was told to use this tool and I was not. And it's like that sort of insight, right? And so you aggregate enough of these and you then you build this list of like, "Wow, this open model just refuses to use rip grep. Why is that?" And then you sort of look and you see, "Oh, like the way that we've been prompting this is via system environment injection." And then you notice this trend, "Wow, every single one of these like system injections is being ignored by this open model. Clearly that open model was not post-trained with environment injections." So then you have multiple options, right? You one is you can like try and force that way into the harness. But at this point what you've uncovered is this is effectively a model problem. So we'll share that and we'll share that with the company and we'll say, "Hey, by the way, check this out. Like look at all of these things that work super well for these other models that does not work for your model." Then the next iteration comes. So that's part of it. The other part is sometimes you notice subtleties in the harness. and this is always the first time you roll out a model it ends up being that like whatever inference providers you're using have just like not to blame them but like it's a very hard thing to do. But like the thinking traces are no longer consistent. They're not being passed in order like that often times you find API errors and that is the vast majority of for a lot of companies when they first rolled out either inference or models. We see that like the vast majority of what we're doing is just finding bugs in the API. And so that's like the honest answer is a lot of it is it is I feel that all of this is engineering. It's like good old-fashioned engineering. It's not magic or like arcane whispering. It's just straight-up like looking at problem, finding bug, and then pointing out who should fix it. I think that's underestimated that requires like a team of people and you have to be thinking about it all the time. So, my main thing would be I don't think that it's like only one people one company can build a great harness or model-independent harness. In fact, I think like you guys build an awesome one. I think ours is great. The Amp folks make a great one. Cursor's is awesome. Like so many companies make great model-independent harnesses. But, I do think that you almost certainly need to be focused on building a harness if that like as a full-time for your company. Cuz otherwise, it's going to just be this painful process that you only discover on like the sixth model you integrate and that it's really hard.

Speaking of some of the API errors, do you think the model APIs are going to get more similar or more different over time? And how much of the differences do you guys lean into? Do you take advantage of all the different things that Anthropic does? I think they have some auto compaction, I think Yeah. tools they can call that run server-side. How much do you lean into those server specific ones?

all of the server-side operations that all of the API providers are doing are like better served implementing them ourselves. the primary reason being that we get it for everyone. So, like for example, we I I think might be the only model and maybe you guys have it. I don't know for sure. But, like as far as I'm aware, like we were one of the first at least to have tool search in like Kimmy and GLM and like MiniMax. And now I think it's a lot more common to have like tool search, but there's a lot of things like that where if we just choose to implement it on our side, we get a lot of value. I also think that like one of the things that we do that's a little like kind of like pattern breaking is that our full agent, the entire loop, the whole harness is inside that binary that we give. So like if you really wanted to, like you could kind of see how the sausage is made by just decompiling that binary. maybe don't do it, but like at the end of the day, it's out there and you could if you wanted to. So I think that like a lot of the stuff doesn't need to be black boxed from the user. Like you can just run it on the client. but if you did want to sell your API and prevent people from leaving you would put this on the server side. So it's like a reasonable business decision to do.

Yeah we do similar thing to you where we just we implement these mostly inside the harness, but Yeah. it's a question I come back to. And the And the root of this is like cuz I think if you're talking about kind of like just matching the prompts, matching the tool arcs,

that's easy. I think you can pretty easily do that. 100%.

Is there anything that like a model provider could do to make their kind of like harness really legitimately only work with their model and not easily be copied by who That That's kind of the root of the question.

We have like thought about this a ton. I'm sure you have as well. The answer is is one never say never. So like I don't know. It could There There is totally a scheme I could imagine. I think that almost every var- variant of this has so many properties that I think make it not an like an obvious thing to do. The first is I think that because the API itself needs to be available because that's their core business, right? Like they sell an API, a platform. As long as you have that publicly available, I think that you can always build things on top of that. I think that there are certain decisions that would severely limit the availability of their API. Like I think as a as an example of this is obviously there's some decisions that have been made to like degrade the model given operations on a certain type of like workflow. So, for example, if you tend to work on something that maybe they view as competitive, they being anyone who chooses to implement this sort of thing, then maybe they degrade the model silently. That happens on the server side. You can imagine for a second that they might also then say, "Unless it's got a tag for our harness, right?" I think that because these are also client-side bundles, you would see that. So, like you would know that if you included a tag that said wear this harness, it would go. Now, you can on your server-side say, "Well, I'm also searching for the prompt." And so, I only this system prompt does it. I think as you start to go down this route, you realize that you have to be actively combative to other harnesses. And I think that's sort of a jump that is a decision as a business that signals that platform business is no longer your core competency or you don't want it to be your core competency anymore. For so many reasons, that will lead to a funnel of money moving away from them, I think, to others. and I also think that to the market, right? they're valued as platform businesses, right? Especially in this IPO roll-up, it would be interesting to see what happens as their thing that makes up 70 to 80% of their revenue has a forward future projection of like maybe like $0, right? And instead, they think applications will be the only thing they want to serve. They would then have to make up the delta in revenue immediately. and so then you would they would have to imagine that they can do that for some reason because maybe their models jump far enough ahead that like people will, regardless of what they want, obviously pick that. To me, this scenario sounds like hard takeoff. So, I do think that like that could happen. Like, and if that does happen, I think the implications for the rest of the world are like so much more important than for like our company. Like, I think that like hard takeoff is like so interesting and important for so many reasons that I'm like kind of like, "You know what? If that happens,

What's got bigger to

play a different game." Like a different thing is going to happen and we can always like war game that .

What about some of like the statefulness stuff? Like that's kind of what I come back to. Like in theory, if like you could derive like just so much value from some stateful property of the API

Yeah.

that I guess you could use that without

the like Like memory maybe or like

Exactly. Yeah. That I think is the only like Yeah, that would be the only thing that would be like I think big enough to kind of like maybe carry some weight. But memory so early on maybe this is transi- like how do you guys think about memory?

Yeah, memory is hard. Memory is a very interesting thing. I think that there are several different directions that memory goes down. And one of the things that we've continued to say is like we want to make sure that we're building things that are like maximally useful for our customers. And so I think that there is it's like very worthwhile to do research on this. So we are we've thought about so many different ways of implementing this. I think the big first fork is do you think this needs to be a learned memory via gradient descent? Like is this a learned process? Does this need to change the weights somewhere? Is this like a Laura fine-tune? I think Ramp just did some really cool work on sort of like learning a Laura head on top of a model or they called it something else, but it's equivalent sort of system. If that is the case, then it leads to some very interesting things. Like for example, you could then you don't give people the weights. So you host these like Laura heads for every customer that get learned over time on preferences. I think that might make them the market for open models that do this like significantly larger because not your intelligence there's probably a flashy way to say that, but what . Like I think that honestly that decision would might shift people even more towards open models. So I think that's also an interesting thing to go. The other is if you think that memory is effectively a like us like a system of record like files context graph objects durable objects or something. We think there's a ton of things that like state outside of the agent super important and you could call it memory but I can always come up with better names like preferences style guides skills all of these to me feel like they're more useful than saying memory.

A few things you said that I agree with I think like in terms of enterprises or anyone wanting to own like their memory what in whatever form I think that's like spot on I think like that's absolutely what you want to own. I also think like it will probably in the long run I feel like it will probably be a combination of these things anyways I don't know how we'll get there. One thing though that you guys did which is maybe kind of like adjacent to memory auto Wiki can you talk about auto Wiki and how do you think that fits into the whole software factory?

So there's been so many variants of this right like when we three years ago when the first droid we ever made was review droid right and what we ended up finding and this is like a bit of like a history lesson for us we ended up finding that the review droid was just not consistently gathering the context that it needed about like not just the current decisions but historical decisions and so then we built documentation droid and what's funny is that the actual high level workflow is exactly the same as what we built three and a half years ago. Now what has changed is first of all there have been several products that have come out that are like sort of also along these lines and I think that each of them had like really interesting things that they did around like how you want to represent this like should this and also I think it helped us answer the question who is this for right like is this for the agent or is it for the person I think that it's clear to us at least that this is for the person and the people I don't know if agents need a Wiki anymore I think that they're good enough at finding this context. And so what we did is we said, what would it mean to optimize effectively like really useful dense representations of knowledge that a person can glance at and they can learn much more than even asking an agent. And so that made us turn towards like what are the structures of knowledge that are useful. So I think a lot of the value that we're bringing with Auto Wiki, the workflow's cool and I think that like you could like if you just kind of want to say like go and create like a wiki for me like Karpathy's living wiki, all these sorts of things are all great ideas. I don't think you need us to do that. I think what we provide are a couple of canonical structures where

we're like For the wiki itself.

the wiki itself. Like this type of code base, you should just think of your code base like this. Or like you are saying a lot of different things but really you're just talking about a microservice monorepo. And so like I think a lot of that is just saying like here are some canonical structures. And then we did one kind of spicy thing which is we made it create a video, like a video overview. So like it uses like a skill to design like a motion graphic text-to-speech sort of like video walk-through of the repo. and like all of our new hires like watch those videos for us. they're like pretty good. I'm I'm like very it's very amusing feature.

are they?

Like 4 to 6 minutes. Okay. I I'm super amused by them. Oh and then we have a thing called lore. And lore is so fun to read. It's just the history of the code base.

Okay.

And sort of like annotated by an agent.

part of like Auto Wiki when you use it.

Every Auto Wiki gets lore and when you read it, if you've been working on your code base for a couple years it is like a trip down memory lane. It's like pretty fun to observe. And it also pulls insights about I think the people. It's we try to make it pretty celebratory because you sort of see like these things where it's like oh, Stepan made the first commit of this thing that we call agent core and like this thing is now the centerpiece of your company. And like these are the types of things that are like very fun to see. you guys also have like a I think you guys just released something like a couple days ago in this kind of camp.

Yeah, we're starting to play around a lot more with this idea and I'd be curious for your take here, but it kind of goes back to memory. We kind of see like two different things in memory. One, I think the first person who I talked about this was someone at a financial institution they were talking about their traders on their desk. And like they said, "I want to like take all the knowledge that's in this person's head and that's expressed through their like trades that they make and slacks that they make or whatever and I want to like clone their brain so that if they leave and go somewhere else I can have that."

Yeah, and so I I kind of view some of these wiki ideas as cloning a brain of like the of some third-party like data that already exists. And then I think there's a separate form of memory which is like as the agent runs and as it interacts and and humans give feedback like how do you like how do you update based on that? And maybe that's the same data structure that it goes back to, maybe it's a separate one. That's like a separate process. But this sort of like first like clone your brain up front thing, a pattern that we've kind of like seen a few times. I think like auto wiki, deep wiki, this trader that I mentioned, Karpathy's LLM wiki. And so yeah, we released something we called open wiki same I just run an agent, create this wiki which is like a set of files. Some of the open questions in my mind and you already answered one of them which is like the structure of the wiki. Like how opinionated should it be and should it be different for different domains? Because I think we're really interested in enabling not just coding agents but everything. And this may be a little bit of time travel either future or past depending on when the episodes come out, but someone I just recorded a with another podcast they do the same exact thing and they also said they enforce a structure and it's and that's that's how they think about the value they provide. Probably similar to you guys. Like you guys have domain knowledge of different types of repos. And so like anyone can point a coding agent at a repo and be like create me a wiki, but you guys say, "Oh, it should be in this format with this structure." Yeah, it's funny you say that. I do think that this is just the single It's I think it's boring is why people don't think about this more often. Like the boring answer to like why application layer companies are continuing to grow and add value and people care about us is that we just simply take a bunch of opinions about a workflow and we just keep encoding it deeper and deeper and deeper. And I do think that is always going to be more valuable than just like not having the opinion. Like and I think that that's just intelligence in general. Like we see this modeled in consulting firms and in other services industries. Like you can hire generalists or you can hire specialists and I and I feel like that idea of we just give you the almost like a an opinion about it is always going to be valuable.

I think I think that's exactly right. there's an interesting I don't know if you've seen going to mess up the acronym. It's from Google. I think it might I think it's OKF open knowledge

format. Oh, yeah. Yeah, yeah.

Yeah, have you thought about adopting that for AutoWiki or conforming to that or leaning into that?

Yeah, I saw that. I think that's one of those interesting things. I think this intersects with our previous conversation about model preferences. Like my biggest thing would be is like that open knowledge format. What model is like sort of jumping to that? And is it in the data sets enough? And I think that if the goal of that project is to say, "Let's make sure every frontier model post trends on like an open knowledge format based structure to maintain its own curated memories palace." Like that to me is like great. Like then we should definitely adopt it. But my guess is that's like Gemini. Like Gemini is probably good at maintaining that format maybe like the other models are going to take a couple generations to even like learn about that format. And so, that would be my initial hesitation. However, very open-minded. I do think that the difference in structure though is like useful. I don't think that a code base that represents like a web app is even close to what FFmpeg looks like. And so, we should just like not model those as the same, I think.

I need I need to also look closer at Okay, I have to see if Like I think you can probably have some difference in structure and maybe more of some of the semantics around things. But, it also goes back to what you said earlier, which I thought was really interesting. Because when we're thinking about these wikis, because we're thinking of it as memory, we're very much thinking about them as power and the agents. Yeah, and so I thought it was really interesting that you thought it was more for humans, which I totally get. And I don't know if I don't know if that's something we're wrong on. I don't know if it's just different use cases need different things.

Well, I think that I think that if it's not a wrong or right because like if we were to build and we're very interested in building like the same structured state of knowledge outside of the agent as like for the agent. That's a really weird way of saying it, but effectively, we're also interested in that. To me, the Auto Wiki product though that we sell, that is primarily for humans. So, I think we'll call whatever that like is closer to what you're building something else. In fact, that may be like something that we call memory. Or I just think that word has caused so much trouble for so many people in the AI space.

Memory?

Yeah, and this is more just like I'm I've got my like enterprise selling brain on. I do think that if you say that word to a CIO or a VP of engineering, they will they have just heard that pitch like a hundred times. And so, I almost feel like that we downplay some of that. Even if we were to do it, we might even hide it so that you don't even necessarily know the memory's happening.

You guys do have something around like agent readiness report. And so, I'm curious like maybe that ties in some way cuz that is like presumably how you should I'd love for you to give a deeper overview, but presumably it's like how you structure your code base so that an agent can use it well. That's kind of the similar thing, right?

Like I think that how you structure is a huge part of agent readiness and really even more broadly what I would say is that agents succeed when they have non-human deterministic feedback. So like when they can consistently receive feedback from anything other than us, then they can keep going.

For software engineering, is that unit tests or are there other things?

So many things. In fact, there's like hundreds of these sources of deterministic feedback.

Like what are some examples that I'm not thinking of?

So not only are do you have So testing would be a whole category. Unit, end-to-end, integration, fuzzy, also like test coverage. So like whether or not you pass test coverage is itself a feedback item, right? So there's like testing dev environment reproducibility. So can you spin up the dev environment? When you run it, does it work? Is it flaky? So there's sorts of the flakiness detection, right? Like those sorts of things. Standardized metrics and logging. Do you have a rule that says if I try to make a metric that doesn't fit our format, will it fail? So that's like a form of linting, but it's a little bit more specific to how logs are done. and you can get really specific with all of these, like do you have AST parsers? Do you have opinions about the code base? Do are you linting file names? Do you have a structure? do you have code owners? So all of these things the presence or lack thereof, hurt or help agents. And so what we have seen is that if you really want to solve a problem with an agent, if you can define the problem clearly and define the outcome and measure whether that outcome is true, there are harnesses, like for example missions, that can just achieve that outcome as long as you're willing to pay the inference cost.

to ask about missions. Yeah, could you talk about missions and then what makes maybe like missions successful and what's going on under the hood?

Super related to agent readiness. In fact, we now have instant We did We used to not do this, but now if you try to run a mission and you haven't run agent readiness, we gently recommend that you do that first. We're not going to block you, but we gently recommend. missions is a attempt to say there is a like more even more I've used the word workflow a lot. There's an even more like general meta workflow that an agent can use to discover the right workflow to solve almost any knowledge work problem. And so that's like kind of a lofty goal, right? Like a universal meta harness almost. What we What we think is that this verification problem or like can you validate or verify the outcome is so fundamental to how agents like succeed or fail inside of almost all tasks that you should just build a system that says for whatever problem that they've just defined, one, define what that workflow should be and define it only via the like structure of what should be What is the outcome and how do you verify that outcome? So, for example, when you run a mission That's like a very like kind of like like meta way to describe it, but take a software engineering task. when you do like a spec driven agent, you create a spec and you say this is the outcome I care about and in the bottom you might even say like, you know, how we're going to validate it. You write tests.

ball.

This is way too loose. Like what we do with missions is we say, let's create an extremely structured data object that represents every possible like hypothesis that could be tested about the software. Like can I do this action? Is this flow supported? Does it do these? Are these things present? And then you also in that structured object say, and this is how I will know deterministically that this is true. Or if you can't deterministically validate, here's how an LLM will analyze the success or not.

Do the Do the deterministic things end up looking like tests, or I guess some maybe some lens and all that?

So, it could be a test. It could be like, I know that this flow works because a unit test passes. But, another one could be, I know that this flow works because when I open up the browser and I use computer control to click on the sign up button, it takes me less than four clicks to get to this page. And so, like that is obviously non-deterministic. However, it's very clear. And so, like a verification agent can go in and it can run and it can say, I either achieved that outcome or I didn't. And so, when you scale this out, you start to realize there is no system that can let a non-deterministic agent accomplish all of these goals in a very like consistent way. So, you have to add a little bit more of a push. Do not let the agent finish until it has confirmed every single one of these validations. As a result, this can take a very long time. And so, we sort of say, "Look, this is probably going to get faster. It's probably going to get better. But, for now, if you want to ensure an outcome, then you can line it all up such that the outcome is confirmed if you can verify it." So, that's why it ties then to we say, "Look, do you want to bring the performance of the system down to X milliseconds? Do you want to validate that this thing works?" As long as you can like ver- verify, then put it in a mission and it will succeed. And what we've seen is they can do some pretty amazing things. some problems are like unbounded or unconstrained, so you can't necessarily just solve everything. I don't necessarily know if missions couldn't If that like if this some more advanced version of this couldn't solve like most verifiable problems.

For some of the tests that you run, won't those need to be generated by the agent as it's running? And like couldn't the agent just write a silly test that always pass and be like, "Oh, satisfied the mission."

Totally. And so a lot of what ends up happening is you need to validate the validators. So you it's kind of turtles all the way down, right? So you there is implicitly some trust, but I think this is how like human organizations also work is like you kind of have we're fuzzy non-deterministic systems. What's nice about agents is that they are pretty good faith. They do genuinely want to solve the problems and I think that the reason I Anthropic's interpretability research on this was so fascinating of like panic and how like or at least the auto-encoded representation of panic inside of agents as they get closer to failing their goal, they become more likely to cheat, right? So we want to work around some of these weird quirks. We make sure that like agents are sort of given very specific instructions. And so there's a lot of like harnessing around this. Our goal is to remove as much of that harnessing as possible by getting increasingly eloquent workflows, like skills . And so at this point Droid will define its own skills for each worker. And then it will continuously refine those skills as each worker works. And because they're workers, they don't have context or access to what others have done. So each one is sort of like starting fresh and given a blank task with a set of instructions that are sort of like the result of many agents running on that task. And so you have a good chance that like any like P hacking or like any attempts to sort of like sketch through it only lasts within a single run. And then if you have like milestone based validation, you can get rid of a lot of the like variability within a given like individual run.

So just to make sure I understand, by worker you mean like one run of the agent up until it checks the criteria.

Yeah, at a very high level, it's like an orchestrator that defines skills that then workers pick up the skill and the and the task. They do the work and then a validator comes in and it validates the work. And then you can imagine running tons of those consecutively and then you sort of have like almost like a consolidation phase where you like look at all of the accomplished tasks and you say is this chunk of work sort of adequately completed? To us it feels like a lot of the problem this definitely uncovers how far we have to go with even the models themselves. Like the clearly a very diligent human would do much better at like every individual task but the sum of work I don't know if there's many humans that would as diligently complete the total sum of work in the time in anywhere near the time that these can.

In practice do you see that the validators are more often than not the deterministic things or the LLM as a judge type things?

The validators rely very heavily on the deterministic flows. So for example if it says like in order to do this you must pass this end-on-end test the validator will be very good at writing like a pretty fluent end-to-end test. I mean the intuition for this is like if you open up a coding agent in an empty repo with like pretty little context and you say to it like write a really good end-on-end test for this workflow. They tend to be like pretty good at that style of task. So that's what we're doing is we're just trying to imitating the fresh context clear repo clear instructions workflow but scaled across what is like a very large task that's been decomposed into its principal units.

Missions sound like they would cost a lot of money and I know in general like token usage and therefore cost associated with coding agents has risen dramatically even inside of LangChain over the past 6 months I'd say. What are you guys seeing and how do you guys think about dealing with that?

Yeah, tokenomics is the number one topic for I think everybody right now. Missions do cost a lot of money. I think that there's like a true ROI story for them. Like this scale of task that people are currently using or think order of magnitude like couple tens of millions of line code base migrated. So, I think it's the type of thing where the order of magnitude for a mission could be like $100. it could be $1, 000, right? And at the end of the day, that is a decision that you sort of have to make knowing it's going to be expensive. I think in many ways this is not that different from almost like instead of just like hiring a contractor, it's more like hiring like a firm, right? Where you're really going and you're getting like a quote. You're saying, let me shop this around, right? Like I want to understand is this going to be expensive?

Do you Do you provide those estimates and do you let people say, "Hey, do this mission until it cost me 50 bucks. That's That's the max I'm willing to pay."

Yeah, we have price controls and increasingly we're working on our prediction of the total cost. Because I think that in the end, like what we want to do is we almost want to be able to say to you, " what would be great is to insure it." Like we promise that you'll get this done for this much money. And we know what that price is and we sort of hold ourselves to it. I'd love to get to that. That's a tech problem. That's That is nothing other than a tech problem. And we're working on it. But I think that what a lot of people face is more like wasted work not in missions, but across the vast majority of enablement. Like when you roll agents out to an entire company, you've got 10, 000 people using these tools. It's so easy to waste money with them that I have found that the vast majority of wasted spend has nothing to do with like whether the agent is 5% more efficient. It's just sort of like cutting off access to expensive models when they shouldn't be used.

I was going to ask yeah, like how do you What do you see people doing? How do you help companies deal with that?

Auto model routing. the biggest save for most people is to understand what tasks need what models and to remove the humans need to learn an entirely new world in order to use these agents.

And I think you guys launched or announced model routing a monthish ago and I think some of the figures were it cut costs by 30% something like that more 50%

a parameter which is funny. Like we you can choose how much it saves you money and that is because what you're trading off is quality, right? So there's a

is this the admin that chooses that or

The admin does. Yeah, so it's it's pretty cool. Like we give you a slider on the burrito frontier, right? You can sort of say hey, I'm comfortable with this level of like aggressive cost savings and I know it's going to is that different from picking a cheaper model as the default? No, I'd argue it's better because the more that you're able to sort of slide up and down that true burrito frontier, the more effective you can find like a happy place and I think right now a lot of this is like kind of the equivalent of like the elbow curve estimation where data scientists kind of eyeball the part on the curve that looks good. What we are doing is we're trying to build what we call agent effectiveness tooling. So think of it like this. We know that there's a task distribution that's unique for all tasks that agents are being used for in every company. So if we can take that task distribution and learn a model that is extremely good at deciding when to use what model across that distribution, we can give you the correct answer on that curve. you want to preserve the best possible quality and the best possible cost and open models are getting so good that it is impossible to ignore. However, what we found is that if you show people the model, people's preferences immediately fly towards frontier models. So I think a lot of this is like psychological. Like if I do not know what model I'm using, I am likely to say, "Wow, this is a great model." If I know the model I'm using, my bias is towards the frontier models. So I think that like that alone explains a lot of what has happened over the last couple months is like marketing. Like some models are better at a lot of things and if you're better at the right things, people will get convinced that they're better at everything. And in reality, I think if the cost was like not that much, this probably would just be fine. But the cost is so sick We're talking about tens, hundreds of millions of dollars even in a single enterprise. I think math is going to win here.

One common critique of model routing that I hear is prompt caching. How do you guys deal with prompt caching and model routing?

Cash-aware model routing is itself like we have like an eval like routing bench. And so the big challenge is breaking the cash is often times worth it if that is you have to just be aware of it, right? So, what's cool is also within a given family of models, you can at least preserve a little bit more continuity. You can't preserve the cash, but it's like faster to switch because like this is just how our model switching works. We compact in order to switch models because the reasoning traces don't nicely pass over between the two. And so like if you're routing to a model in the same family, you can preserve like midway through instead of just compacting. But what's nice about the cash-aware sort of evals that you build is you need to make model routing not just And this is why I don't think you can do it in the gateway. Like I do think maybe if you really wanted to, you could route you could get some gains from like a gateway. But at the end of the day, routing is a harness problem because the harness needs to be aware of when to upgrade or downgrade, right? And so if the harness becomes aware of its own ability to upgrade its intelligence and aware of its own cash, then it knows, "Well, I'm about to finish this problem anyway. And so I might just finish it now while it's cheap." And if I can't finish it, then I upgrade. Versus I am so clearly far away from solving this problem, I'm just going to upgrade now because otherwise I'm going to waste tokens. That is instructions and guidance you can provide the like monitoring system that does the routing decision. so, tech problem, very solvable.

You mentioned open models earlier. What are you seeing there in terms of how viable they are, how much adoption they're getting, and where you think the future goes from there?

We're very bullish on open models. It seems like I wish there were more options in general. I think that my hope is geopolitics doesn't get in the way of the progression of open models. I think a great US-based open models is obviously a huge need. I would argue it's like a national security imperative. For us to be building better American US models. But that being said I think that my hope is that we're able to sort of transcend a little bit of that. And open models are to me a global thing. So, if we can get like really amazing globally just open models I think that is clearly much better for almost every single business in the world. There are like a very limited number of businesses that do not benefit from open models. And I think the vast majority do. So, I believe in like sort of like accelerating the economic prosperity of the entire planet. So, I think that even if the US doesn't get access to open models, I'm still in favor of them. Because at the end of the day you know, intelligence is probably going to be a commodity. Interestingly enough from our previous conversation. And so, if intelligence is going to be a commodity, my hope is that people have access to it.

There's another type of routing or there's another type of model usage that sometimes gets lumped in with routing and I think called like fusion. Like Open Router had their kind of like Open Router fusion. Sakana AI had Fugo, I want to say, or something like that. Have you guys thought of that? routing to multiple models at the same time and then consolidating the results back.

So, this is super interesting because we experimented with several different flows. I will say I think we're in our like this is a throwback the cognitive architectures era where the naming of this stuff is probably doing a little bit more confusing. At the end of the day in my mind these are all model routers and the interesting thing is there's so many levers to play with. For example, I mentioned upgrading like giving the model a tool to upgrade or downgrade its intelligence. Then there's sub agents. So sending off sub agents that are weaker models to then come back and synthesize. And then there's parallelization and synthesis like run to it once and pick the winner. We used to do this all the time back when you had to like really hard code that workflow. Super not obvious to us if there's like a strictly greater than but I do think that what's clear is all of these strategies are helpful. So I think that right now we're allowing users to kind of toggle a couple of these things. The parallelization has been a feature for a little bit. You can run multiple at once with different models. We is a very unpopular feature.

And that's parallelization of agents. So not just like a single call but agents.

Exactly. Like let's run two agents with effectively the exact same prompt but with different models and resynthesize. These are all very unpopular with our users.

Why?

I think that it's too complicated. And so I maybe if you like chose to hard code this and like make it like the only option, maybe people would like it but what we see is that it's way more expensive and unclear gains. That could be just us. I've no knowledge of if it's there's like a best answer.

You mentioned evals a few times. How many different benchmarks do you have?

We have we even have a benchmark called bench which is droid's ability to create more benchmarks. So we have a ton and I think a lot of them are small though. I think people assume that like to create a benchmark you have to build this like massive data set. and we have a couple that are objectively very large, but also we just run we have a very eval driven culture in general. Like we have these little like regression based evals that run like 10 tasks and they just make sure that prompt caching works or compaction works. We have And so I think the line for us between very integrated end-to-end test and like eval is very blurry. I consider it an eval when there's some degree of like rubric based grading or scoring that doesn't It's like not a binary yes no. Instead, there's like sort of a gradient of scores. The cool ones, legacy bench is cool. That's like COBOL basic Fortran migrations. bench is cool. Code review is a totally uncharted territory of benchmarking. It's insanely hard to benchmark. I think that code review benchmarks might actively be hurting code review products.

going to say there's at least one code review benchmark out there, I think, right? You just not trust it?

Everybody's code review We learned this recently. Every single company in this space's code review benchmark is a fork of that benchmark. And so that benchmark is effectively like no statement if it's like good or not. I think it's like was useful. It's probably not useful anymore and it was a great innovation at the time, so it's like not shade to the creators. But I do think that benchmark is massively influencing how code review is done across the board and I don't know if it's obvious that like it is aligned with a better experience for people who use these tools.

How often do your benchmarks change? Like when does a benchmark stop being good? Cuz I think it's totally reasonable what you said. Like yeah, it was a great innovation at the time. It probably provided a bunch of value.

Benchmarks have a very short shelf life for us. Like 3 to 6 months max. because we get better ones or we build like a V2. Like terminal bench for us, we still run not because we think it's making our agent better, because it's a very useful regression tester. Like you just like there's certain things that at this point we built a ton of tests for regressions, but like we do not even look at terminal bench anymore. It's just not even it's not useful. So, three to six months is a my kind of like shelf life for it, I'd

say. Do you use Harbor as a format for bench marking?

Love Harbor. That's That was great. I'm glad we were like standardized on that. Yeah. Not only us, but like most people as well. We just did an integration with them like literally just last week. Like deep agents in Harbor, trace things to LangSmith, LangSmith sandboxes. I think for like long-running agent evals, it definitely seems like the industry standard.

Yeah, completely agreed. Thanks for listening to Max Agency. If you liked this episode, leave a review and subscribe. Send feedback or questions to maxagency@langchain. dev. We want to hear from you.