Transcript EN 0:00 Yeah, welcome from the world of snot and coughs. 0:05 Yes, Mario and I, we are at the beginning of the Kids 0:11 Bring All Your Sickness at Home Wave. 0:13 Yeah, everybody sends their kids to school or kindergarten and then a week later everybody is sick. 0:20 So a small little anecdote here. 0:22 I started my career at Sentry pretty much exactly 0:28 coinciding with us getting our first kid. 0:31 And now David, a friend of mine, founder of Sentry, now he has his first kid. 0:36 And now he's learning all that stuff and then sharing on Twitter. 0:42 It's like, how do you manage to work with kids? 0:44 It works. 0:47 Yeah, first time. 0:50 It just requires adjustment. 0:52 Yeah. 0:54 And we are now also adjusting to the models changing. 1:00 What a segue. 1:01 And what a segue. 1:03 You're really good at this now. 1:07 So we decided we want to talk a little bit about how the new models behave and what 1:13 implications to have on harnesses and how we might or might not respond to that. 1:20 And what we think about it. 1:21 Right. 1:21 Right. 1:22 So the problem that Armin identified recently with the release of Opus 5.5 was that it stopped using the 1:28 edit tool and just basically uses bash for everything now. 1:32 For, yeah, well, Python, why a bash? 1:35 That might be token efficient. 1:39 It might have some correctness issues like you identified. 1:43 Like the code might not necessarily give the model signal that the edit didn't actually 1:49 do what it's supposed to do. 1:51 So that's not cool. 1:52 And from a UX perspective, you, as the user watching a transcript flow by, 1:59 don't get a nice diff anymore necessarily, but just a super complicated token efficient 2:05 bash call with Python intermixed. 2:08 And that's not great. 2:10 So the question is, what do we do, Armin? 2:12 What do we do? 2:13 So maybe, so I think that there are two 2:19 macro trends, 2:22 I would say, that affect us too. 2:24 But I think like at least for the token max of the world, like why do we even care what the transcript is one? 2:30 It's like you only look at the end result. 2:32 And then the second, I think, 2:35 thing worth discussing is not everything is a coding agent. 2:40 And some things want to have more control. 2:43 And like the less likely an agent is at using the things you want him to use, 2:48 the more control you're sort of giving away in one form or another. 2:54 So maybe we start with like, what's the point of the transcript? 2:56 Yeah, I mean, for me, 3:01 the transcript, the transcript with respect to seeing diffs and 3:07 so on is meaningless for me 99% of all the time. 3:11 If I have more than one agent open, I'm not looking at all the transcripts flowing by. 3:15 That's a waste of my time. 3:16 I'm basically looking at the result, the artifacts, the file changes they made after they think they're done. 3:22 And I do this in a proper IDE or editor with a nice diff tool where I have code 3:28 navigation, might even be able to debug things like VS Code. 3:32 And that seems to be a much better spend of my time than watching transcripts go by and seeing edit tool 3:38 calls and kind of interfering when I see something stupid being generated or edited. 3:46 I think it's the same for me. 3:47 I don't watch the transcript as it's doing except for a handful of cases, 3:52 one of which is if I ask it to, for instance, do something with a computer. 3:59 I'm not saying a computer use, but like, for instance, 4:03 configure CI or figure out some access to a 4:10 remote service, like apply Pulumi things. 4:11 Then I really care about what it's doing. 4:14 But for the most part, the transcript to me doesn't have, at least when it implements stuff, I don't really care about it. 4:21 I do, however, have the situation where I have a conversation with the agent about how to solve a particular problem. 4:28 And in order for it to have a better understanding of what we're doing, 4:34 it is encouraged to read files, understand the code base and so forth. 4:38 And then you have this intermittent read some stuff, maybe do a POC, and then we discuss. 4:46 And so most of the transcript now no longer collapses as nice in Pi as it used to because like all their, 4:52 I mean, it probably already had a lot of like bash commands in there that already didn't compress all that well. 4:57 But now the inputs to those tool calls are humongous because they are all the writes now and they don't collapse. 5:04 So the transcript now for me becomes harder to read when I do still want it just because all the 5:09 verbosity and the ugliness of the tool calls in there. 5:14 Yeah. 5:14 I can see that something that touches a production system or even a staging system where you want to actually see individual 5:21 commands to judge whether something's going to go to shit real soon. 5:26 That makes sense. 5:29 I don't know if there's a good solution to that. 5:33 Like we looked at what other harnesses like Codex or Claude Code are doing. 5:37 And I think you looked into Claude Code. 5:39 What's their solution here? 5:41 So Claude Code on all the new models, 5:45 which is Fable 5, 5.1 and Opus 5.5 uses, 5:49 I forgot the name, they had a really funny name for this, but it's basically a mode where two things are happening. 5:54 One is it basically pushes with a prompt instruction, the agents would just use bash. 5:59 So like they're reinforcing even the thing that sort of it does by hand, out of the box anyways. 6:04 But then what they are doing is they're snapshotting by writing into a separate Git work tree all the files that you're working on anyway. 6:11 So in addition to the Git, it's not the user work tree in the traditional sense, 6:17 but it writes into a separate index. 6:19 So it maintains for each repo you're working on a separate index just for one agentic run. 6:24 And then after and before each tool call, 6:27 it compares the state of that local checkout to the work tree it had before. 6:32 And then it synthesizes a diff into the UI as if an edit tool call happened. 6:38 So you can see the snapshot as it was taking place there. 6:43 And I think this is sort of, well, so I think that there are two kind of interesting things here. 6:50 One is that I actually sometimes do perform edits from the wrong working directory. 6:56 And the reason for this is, for instance, we have pi, we have radius. 6:59 Every once in a while, I have to work on two things and actually do let it commit to the other thing. 7:02 That will not work with the work tree approach anyway. 7:05 So like the moment you're outside of this place where you are, it doesn't work. 7:09 The other thing is it's actually not super fast either because like you might have a lot of files in there, 7:14 which are at least not sufficiently tracked by Git so that you can very trivially make that work. 7:20 So it's a little bit unclear, I think, if it's a good trade-off. 7:26 It also means in a way that what you generally consider to be the diff, which is against your commit, 7:33 and what then the coding agent thinks, which is against some sort of other index, 7:38 like it's another form of indirection. 7:40 I'm actually not convinced that that's a particularly necessary trade-off. 7:44 I think at that point, just commit to Git and like make the diff experience versus the Git checkout work really well. 7:54 So I don't know. 7:55 I actually, I think that there's another sort of like small wrinkle on it, 7:58 which is like if you actually do want to watch the transcript, 8:01 like let's just assume you have a use case where you do want to watch the transcript. 8:04 You can't read that stuff anyways as it's happening. 8:08 So you could only ever read it once it's done, which like that's a little bit weird anyways. 8:14 So in a way, if you do want to read the transcript, 8:18 you probably want a summary of what it's about to do. 8:22 I suppose. 8:24 I guess the biggest takeaway here is we need better diff tools. 8:27 Yes. 8:29 Yes. 8:30 And I also like, like having tried a lot of diff tools now and sort of diff tools that also are Gentic integrated, 8:37 like friend of ours, the Tunk, then for modems, which I think is pretty neat, 8:43 but it is also not quite what I want, I feel like. 8:50 So there's definitely, at least for the coding use case, I think like the diff needs to work better. 8:57 It needs to support also like multi-stage commits, like in a PR. 9:02 Maybe not even multi-stage commits, but before you even commit, 9:06 you want to take snapshots of your files at specific times. 9:12 I have this in my custom Pi review extension, 9:17 which is like a desktop UI using Daniel's Glimpse framework. 9:23 And I've used this for months, right? 9:26 And I just found I don't even need that anymore. 9:30 Like I don't need these snapshots before I do a commit. 9:34 I don't care so much anymore, 9:36 but mostly because I try to break down my stuff into very small commits in 9:42 terms of lines of code. 9:44 But I guess that's not a scalable solution because some workflows might just result in a large diff, 9:49 and you might want to have the snapshots. 9:51 Yeah, it's hard. 9:53 I mean, like obviously the goal should be make very small PRs. 9:56 And I think generally that has been true before and after agents. 10:00 There are situations where this doesn't fully work because in addition to 10:06 like the size of the PR, 10:09 there's also the confidence that you have in shipping it or the complexity of testing it. 10:15 So you might actually have a commit even with feature flags that you're not willing to merge yet 10:20 because you want to test it properly. 10:21 And then these have a tendency to getting a little bit larger. 10:26 And then you actually might want to do these checkpoints in between. 10:29 But then when you review it with the agent, it's actually versus the main branch. 10:36 Yeah, I think so. 10:37 When I started working on Pi, 10:39 my realization was that all the models back then were pretty much RL'd on the same trajectories apparently 10:43 because they assumed the base set of tools. 10:46 And with those, they could really work well, which was read, write, edit, bash. 10:49 And I guess the times are changing now. 10:52 And the RL is now geared towards you only need bash. 10:55 GPT models started doing that already last year, I think, 11:00 when they came out with the first five point whatever. 11:03 And now Claude does the same. 11:05 And it stands to reason that other models would do the same. 11:07 So along with these changes in RL, 11:09 the interesting part is that it has a large effect on UX of agentic engineering. 11:16 And we just need to adapt, man. 11:19 Let the RL flow over you. 11:22 Yeah, so I don't think like... 11:24 We have been pretty good, I think, at not fighting the RL. 11:28 Or at least like projecting where it should go. 11:30 But I don't think we have been particularly good at being creative about what the UX around it should be. 11:36 It's like the UX in Pi is basically the thinnest layer over what the models are doing. 11:40 Which probably has like a shelf life. 11:45 And we're sort of approaching. 11:47 One, I think there are like two interesting things here. 11:53 It's like if you had control over the LLM. 11:56 In the sense that you are in fact OpenAI, Anthropic, or you run your own LLM. 12:01 You increasingly are running more than one LLM simultaneously. 12:05 So you're running your large model that does some stuff. 12:07 And then you run a smaller model for either predictive decoding. 12:09 Or because you, for instance, want to already truncate the reasoning output. 12:14 It's like on their side, they're already doing this. 12:18 So basically like there's an almost natural idea now. 12:25 That in addition to the tool call, you might get some commentary around. 12:29 And it's very little today is being done. 12:31 A little bit is coming up with some of the ChatGPT models. 12:35 They have this commentary channel. 12:37 I think it's mainly used now for like this. 12:40 Ambigating the assistant message that comes at the end. 12:43 From the assistant message that comes in between. 12:46 But if you were to run your own inference stack. 12:49 Then theoretically you could actually. 12:51 So like this is a short summary of the description. 12:54 You could invest some extra tokens to summarize it. 12:56 But that's tricky as it streams in. 12:58 But then you could also run Chef on it. 13:00 To sort of classify what's going on. 13:02 So I don't know. 13:03 It's... 13:05 There's probably going to be some experimentation. 13:08 That people might want to run around. 13:10 How to visualize these tool calls in a transcript. 13:15 So yeah. 13:16 I don't know. 13:16 I think that the simplest trick at the moment. 13:19 That works across many models is just. 13:21 You have your tool definition. 13:24 And you add a description, summary. 13:26 Whatever fields that the model has to fill out. 13:29 Make it the first parameter. 13:31 First field within the tool calls argument. 13:33 Then you get it as the first thing usually. 13:38 We should experiment with that. 13:39 I think that could work as a band-aid. 13:42 Ultimately I think we will have to give up on visualizing individual tool calls. 13:45 So that just leads us to the other thing. 13:49 Which is like why was it useful in the first place to have these separate tool calls? 13:52 And a big reason was the approval gates. 13:56 Which Pi obviously never really had. 13:58 But for instance, in Claude it was used to... 14:03 You can have this allow edits by default. 14:05 Which was gated on it using the edit tool. 14:07 And then it would ask for everything else. 14:10 But obviously like the whole thing now moved into a world where either nobody gives a shit anymore. 14:14 Or alternatively they run a second LLM to check the approval. 14:22 To see like should you run this or not. 14:23 So this is this auto mode. 14:26 And what do you think about that? 14:29 Like the idea of running a second classifier or model to make 14:35 some determinations here. 14:37 So first of all the approval gates were always bullshit. 14:40 I don't know a single engineer who is actually using that. 14:44 It's like a checkbox that your harness needs to have to become enterprise ready or something. 14:49 But nobody is actually using that. 14:52 So those went away anyways. 14:54 Because even if like having a second model judge whether a tool call is good or not is 14:59 highly problematic. 15:02 Still in my opinion. 15:04 Because for that the model doesn't just have to look at the tool call usually. 15:08 It also has to look at the full context. 15:11 For it to actually make sense. 15:13 Like I can just have a haiku or a terra or lunar model. 15:18 Look at the single bash call and say yeah this is not dangerous. 15:21 Or this is not destructive in a way that we don't want it to be destructive. 15:25 I don't know man. 15:27 It's just put your fucking agent in a sandbox. 15:29 Call it a day. 15:31 Yeah. 15:32 So I think like we're definitely most likely now reaching the point where the coding agent 15:39 has fully subsumed every other agent format. 15:43 So if you do want to use it in a non-coding environment. 15:49 You're probably going to give it a programming language. 15:52 Which is Turing complete. 15:53 Where good luck. 15:55 So you gotta constrain everything around it. 16:02 Did you say something like there's a code mode implementation in one harness that actually gives the model also bash fully unsiccured? 16:09 I mean this is basically for the most part like code mode now is it exposes all the tools that the agent has. 16:15 Right. 16:15 So code mode in itself might be tightly sandboxed but then you can 16:21 call bash in codex via code mode. 16:25 So then you need to separately sandbox the whole thing anyways. 16:28 So you have like one sandbox and another sandbox which sort of makes sense I guess because like code mode theoretically runs where the harness 16:34 runs which is more trusted. 16:36 And then the bash runs where you're sort of exe.dev or oversell workers or like 16:42 cloudflare durable object sandbox kind of sits. 16:44 But it is all basically in a way really 16:50 like we have now fully committed to this code thing. 16:53 Right. 16:53 I think like at this point we just better get these sandboxes really tightly controlled but like within the 16:59 capabilities of that sandbox. 17:03 The harness basically at this point no longer plays a role. 17:05 Or at the very least it has so little control because the models are so 17:10 yeah programming language slash bash trained. 17:15 I think the question is just I think within Pi for the most part is like how do we make it so that people can 17:22 experiment with how they want to see the transcript? 17:25 Like do we have enough permittives right now to tinker with it? 17:29 And then for someone who builds an agent that is supposed to be self-running a durable object or something similar, 17:36 how should one structure that? 17:40 Do we have enough primitives within Pi to make it acceptable? 17:46 I think that is going to be a lot of interesting. 17:49 Like if someone has thoughts up there, I think we would love to hear. 17:53 But yeah, in a way it's the end of custom tools in many ways. 18:00 Yeah. 18:01 I do have to say that custom tools have historically been not very helpful to my workflows at least. 18:07 So I just recently built a new tool called Pi History Handoff, 18:14 which is sort of kind of like the new codex experimental feature. 18:18 Well, the LLM decides it wants to basically clear the context and just leave a summary of itself there. 18:25 It's a fancy form of compaction plus a tool to give you to give the LLM a way to search in the compacted history. 18:34 That's the only thing recently that I built that makes sense for my workflows. 18:38 But other than that, I don't really have many custom tools. 18:42 I think one thing is super interesting right now is like we're simultaneously seeing a full commitment to bash. 18:48 At the same time, we see a full resurgence of MCP. 18:53 And I still don't really fully know how to put these things together because I think MCP 18:59 solves the off thing, which is important. 19:02 It does set some constraints on the tool execution, 19:06 but then the way we wire it together is like a code mode. 19:11 So it is interesting. 19:14 I would say like where this all lands today. 19:16 Yeah. 19:17 So I think you looked into MCP a little bit more recently. 19:20 And my understanding is that the input schema must be defined 19:27 for an MCP tool, but the output schema can be, but it's optional. 19:34 And for the most part, like you actually need two channels, right? 19:37 You need one channel that goes to the LLM, which is likely unstructured to conserve some tokens. 19:42 And you would need a second channel that is structured so you can use an MCP tool as part of code mode, for example. 19:49 What's the state of that? 19:50 Like, do we actually have those two channels for the output or is that just one channel? 19:56 For the most part, there's just one channel. 19:59 I think it's what is interesting is like, so we, when we built radius, 20:02 we wanted to be so that you can talk to the agent and reconfigures radius. 20:07 And we tried a bunch of things. 20:08 And what we actually found to work the best today with these models is OpenAPI. 20:14 So we actually, so we added an endpoint where you can describe your task and what you get back is a rack filtered. 20:22 These are the endpoints that you might want to use plus the OpenAPI schema. 20:25 And you can actually expose that via MCP. 20:27 So like it is actually very possible to use this pattern against MCP. 20:32 The only problem is like MCP actually at that point really only does off. 20:37 It's like it's RPC plus off, which is a very, very narrow way of using MCP. 20:43 And if you were to build a server that works like this, you don't really, 20:46 you would only output structured output, which MCP is not. 20:51 Like if you, if you were to put this in a traditional MCP, where like you take the entire output and stash it into the context, 20:58 that doesn't really work. 21:00 So like there's an in-between field now where you could actually use MCP almost like an RPC 21:06 system, which only outputs structured data. 21:10 And then your code mode hopefully has enough utility in it to sort of do filtering 21:17 and post-processing, whatever. 21:19 But most MCP servers don't work like this today. 21:21 Like most MCP servers output like various compressed markdown. 21:25 Plain text, whatever, yeah. 21:26 So the protocol is actually really useful now. 21:29 The servers are like not sure if they should target one versus the other. 21:35 Yeah, it's interesting. 21:37 So I guess the takeaway for future MCP server authors and existing MCP server maintainers is 21:42 provide structured output because we're going to go full code mode. 21:47 I think so. 21:47 I think, I think that this is probably what's going to happen. 21:52 Yeah. 21:53 Yeah, that kind of makes sense. 21:53 I still would like, like in Pi, the difference between what the model sees and what harness or 22:00 consumer of events sees has mostly always been there. 22:06 Including for tools and for entries in the transcript. 22:11 Right. 22:12 And I think this. 22:12 And you have to say, like if you are bash only, there's only one output, right? 22:16 Sure. 22:16 Sure. 22:17 If you're bash only, you're kind of, you're kind of, yeah, full part. 22:20 But I think a future will be composed via MCP plus code mode. 22:25 That kind of makes sense. 22:26 I worry though that the models aren't as trained on that just yet. 22:31 I mean, I analyzed a lot of codex transcripts. 22:34 So when, when Astra released, I, I did a whole bunch of like work just in the codex harness 22:40 so that I can see how it behaves. 22:43 And, um, I can see that Astra loves to use code mode, even for situations where it wouldn't have to. 22:50 Like I've seen it uses code mode to apply patch, which is like, why would you do that? 22:55 It's just like a level of interaction that was not really necessary. 22:58 Um, but there are differences in how code mode works in the different models. 23:04 Right. 23:04 And so, um, and the, the surface, the surface area of code mode is obviously much bigger than the surface area of, 23:10 of just a regular tool code channel. 23:13 Um, so I think this requires a lot of testing and evaluation and 23:19 it's unclear where it's going. 23:22 Um, there's also some tension with local models because I guarantee you that local models will 23:27 suck at code mode. 23:29 So yeah. 23:32 I think the interesting use case why I really want to get code mode into PI is not even MCP. 23:37 It is actually Jev because for Jev to make sense, 23:41 you need step one, generate some code that then Jev can evaluate over and over. 23:47 So it's like you basically create like a state machine or like a workflow where on every iteration, 23:52 Jev makes the determination of like which path to go down to. 23:54 So, so it will become a really useful primitive, um, to have within PI harness. 24:03 I'm actually more convinced that it's useful for Jev and less convinced that it's useful for MCP right now, unless the MCP servers are changing. 24:10 Um, but we'll see. 24:12 Nice. 24:13 Well, interesting times. 24:15 Should we go back to our beds and die off man flu? 24:18 Like the babies we are? 24:19 I think so. 24:20 Okay. 24:20 All right. 24:22 Happy Monday. 24:23 Happy Monday. 24:23 Okay. 24:24 Happy Monday.
Download captions .vtt ↓