WEBVTT 00:00:00.000 --> 00:00:03.360 Yeah, welcome from the world of snot and coughs. 00:00:05.160 --> 00:00:11.640 Yes, Mario and I, we are at the beginning of the Kids 00:00:11.640 --> 00:00:13.220 Bring All Your Sickness at Home Wave. 00:00:13.220 --> 00:00:19.160 Yeah, everybody sends their kids to school or kindergarten and then a week later everybody is sick. 00:00:20.740 --> 00:00:22.600 So a small little anecdote here. 00:00:22.600 --> 00:00:28.640 I started my career at Sentry pretty much exactly 00:00:28.640 --> 00:00:31.120 coinciding with us getting our first kid. 00:00:31.920 --> 00:00:36.100 And now David, a friend of mine, founder of Sentry, now he has his first kid. 00:00:36.100 --> 00:00:42.560 And now he's learning all that stuff and then sharing on Twitter. 00:00:42.640 --> 00:00:43.980 It's like, how do you manage to work with kids? 00:00:44.240 --> 00:00:45.320 It works. 00:00:47.600 --> 00:00:49.680 Yeah, first time. 00:00:50.960 --> 00:00:52.140 It just requires adjustment. 00:00:52.600 --> 00:00:53.860 Yeah. 00:00:54.880 --> 00:00:59.940 And we are now also adjusting to the models changing. 00:01:00.320 --> 00:01:00.980 What a segue. 00:01:01.560 --> 00:01:02.340 And what a segue. 00:01:03.420 --> 00:01:04.820 You're really good at this now. 00:01:07.700 --> 00:01:13.920 So we decided we want to talk a little bit about how the new models behave and what 00:01:13.920 --> 00:01:18.660 implications to have on harnesses and how we might or might not respond to that. 00:01:20.320 --> 00:01:21.380 And what we think about it. 00:01:21.380 --> 00:01:21.880 Right. 00:01:21.880 --> 00:01:21.900 Right. 00:01:22.040 --> 00:01:28.340 So the problem that Armin identified recently with the release of Opus 5.5 was that it stopped using the 00:01:28.340 --> 00:01:31.280 edit tool and just basically uses bash for everything now. 00:01:32.720 --> 00:01:35.600 For, yeah, well, Python, why a bash? 00:01:35.600 --> 00:01:39.200 That might be token efficient. 00:01:39.200 --> 00:01:43.320 It might have some correctness issues like you identified. 00:01:43.320 --> 00:01:49.620 Like the code might not necessarily give the model signal that the edit didn't actually 00:01:49.620 --> 00:01:51.160 do what it's supposed to do. 00:01:51.160 --> 00:01:52.880 So that's not cool. 00:01:52.880 --> 00:01:59.920 And from a UX perspective, you, as the user watching a transcript flow by, 00:01:59.920 --> 00:02:05.840 don't get a nice diff anymore necessarily, but just a super complicated token efficient 00:02:05.840 --> 00:02:08.120 bash call with Python intermixed. 00:02:08.400 --> 00:02:09.700 And that's not great. 00:02:10.120 --> 00:02:12.560 So the question is, what do we do, Armin? 00:02:12.700 --> 00:02:13.280 What do we do? 00:02:13.280 --> 00:02:19.080 So maybe, so I think that there are two 00:02:19.080 --> 00:02:22.540 macro trends, 00:02:22.540 --> 00:02:24.600 I would say, that affect us too. 00:02:24.680 --> 00:02:30.140 But I think like at least for the token max of the world, like why do we even care what the transcript is one? 00:02:30.400 --> 00:02:32.400 It's like you only look at the end result. 00:02:32.400 --> 00:02:35.820 And then the second, I think, 00:02:35.820 --> 00:02:39.480 thing worth discussing is not everything is a coding agent. 00:02:40.680 --> 00:02:43.060 And some things want to have more control. 00:02:43.380 --> 00:02:48.000 And like the less likely an agent is at using the things you want him to use, 00:02:48.000 --> 00:02:52.200 the more control you're sort of giving away in one form or another. 00:02:54.300 --> 00:02:56.540 So maybe we start with like, what's the point of the transcript? 00:02:56.540 --> 00:03:01.000 Yeah, I mean, for me, 00:03:01.000 --> 00:03:07.480 the transcript, the transcript with respect to seeing diffs and 00:03:07.480 --> 00:03:11.240 so on is meaningless for me 99% of all the time. 00:03:11.340 --> 00:03:15.060 If I have more than one agent open, I'm not looking at all the transcripts flowing by. 00:03:15.300 --> 00:03:16.180 That's a waste of my time. 00:03:16.260 --> 00:03:22.060 I'm basically looking at the result, the artifacts, the file changes they made after they think they're done. 00:03:22.060 --> 00:03:28.200 And I do this in a proper IDE or editor with a nice diff tool where I have code 00:03:28.200 --> 00:03:31.860 navigation, might even be able to debug things like VS Code. 00:03:32.540 --> 00:03:38.940 And that seems to be a much better spend of my time than watching transcripts go by and seeing edit tool 00:03:38.940 --> 00:03:44.620 calls and kind of interfering when I see something stupid being generated or edited. 00:03:46.200 --> 00:03:47.400 I think it's the same for me. 00:03:47.400 --> 00:03:52.600 I don't watch the transcript as it's doing except for a handful of cases, 00:03:52.600 --> 00:03:58.840 one of which is if I ask it to, for instance, do something with a computer. 00:03:59.140 --> 00:04:03.720 I'm not saying a computer use, but like, for instance, 00:04:03.720 --> 00:04:10.040 configure CI or figure out some access to a 00:04:10.040 --> 00:04:11.780 remote service, like apply Pulumi things. 00:04:11.780 --> 00:04:13.220 Then I really care about what it's doing. 00:04:14.700 --> 00:04:21.340 But for the most part, the transcript to me doesn't have, at least when it implements stuff, I don't really care about it. 00:04:21.460 --> 00:04:28.700 I do, however, have the situation where I have a conversation with the agent about how to solve a particular problem. 00:04:28.980 --> 00:04:34.620 And in order for it to have a better understanding of what we're doing, 00:04:34.620 --> 00:04:38.580 it is encouraged to read files, understand the code base and so forth. 00:04:38.580 --> 00:04:45.400 And then you have this intermittent read some stuff, maybe do a POC, and then we discuss. 00:04:46.120 --> 00:04:52.160 And so most of the transcript now no longer collapses as nice in Pi as it used to because like all their, 00:04:52.160 --> 00:04:57.560 I mean, it probably already had a lot of like bash commands in there that already didn't compress all that well. 00:04:57.780 --> 00:05:04.200 But now the inputs to those tool calls are humongous because they are all the writes now and they don't collapse. 00:05:04.200 --> 00:05:09.740 So the transcript now for me becomes harder to read when I do still want it just because all the 00:05:09.740 --> 00:05:13.780 verbosity and the ugliness of the tool calls in there. 00:05:14.420 --> 00:05:14.540 Yeah. 00:05:14.920 --> 00:05:21.180 I can see that something that touches a production system or even a staging system where you want to actually see individual 00:05:21.180 --> 00:05:26.840 commands to judge whether something's going to go to shit real soon. 00:05:26.840 --> 00:05:28.440 That makes sense. 00:05:29.100 --> 00:05:32.920 I don't know if there's a good solution to that. 00:05:33.020 --> 00:05:37.500 Like we looked at what other harnesses like Codex or Claude Code are doing. 00:05:37.740 --> 00:05:39.600 And I think you looked into Claude Code. 00:05:39.780 --> 00:05:41.160 What's their solution here? 00:05:41.160 --> 00:05:45.240 So Claude Code on all the new models, 00:05:45.240 --> 00:05:49.120 which is Fable 5, 5.1 and Opus 5.5 uses, 00:05:49.120 --> 00:05:53.880 I forgot the name, they had a really funny name for this, but it's basically a mode where two things are happening. 00:05:54.060 --> 00:05:59.240 One is it basically pushes with a prompt instruction, the agents would just use bash. 00:05:59.240 --> 00:06:03.800 So like they're reinforcing even the thing that sort of it does by hand, out of the box anyways. 00:06:04.520 --> 00:06:11.380 But then what they are doing is they're snapshotting by writing into a separate Git work tree all the files that you're working on anyway. 00:06:11.580 --> 00:06:17.620 So in addition to the Git, it's not the user work tree in the traditional sense, 00:06:17.620 --> 00:06:19.140 but it writes into a separate index. 00:06:19.240 --> 00:06:24.000 So it maintains for each repo you're working on a separate index just for one agentic run. 00:06:24.000 --> 00:06:27.240 And then after and before each tool call, 00:06:27.240 --> 00:06:32.260 it compares the state of that local checkout to the work tree it had before. 00:06:32.420 --> 00:06:37.960 And then it synthesizes a diff into the UI as if an edit tool call happened. 00:06:38.760 --> 00:06:41.660 So you can see the snapshot as it was taking place there. 00:06:43.580 --> 00:06:50.740 And I think this is sort of, well, so I think that there are two kind of interesting things here. 00:06:50.740 --> 00:06:55.620 One is that I actually sometimes do perform edits from the wrong working directory. 00:06:56.340 --> 00:06:58.640 And the reason for this is, for instance, we have pi, we have radius. 00:06:59.000 --> 00:07:02.500 Every once in a while, I have to work on two things and actually do let it commit to the other thing. 00:07:02.900 --> 00:07:05.280 That will not work with the work tree approach anyway. 00:07:05.380 --> 00:07:09.140 So like the moment you're outside of this place where you are, it doesn't work. 00:07:09.460 --> 00:07:13.680 The other thing is it's actually not super fast either because like you might have a lot of files in there, 00:07:14.480 --> 00:07:20.440 which are at least not sufficiently tracked by Git so that you can very trivially make that work. 00:07:20.740 --> 00:07:24.300 So it's a little bit unclear, I think, if it's a good trade-off. 00:07:26.320 --> 00:07:33.140 It also means in a way that what you generally consider to be the diff, which is against your commit, 00:07:33.900 --> 00:07:37.620 and what then the coding agent thinks, which is against some sort of other index, 00:07:38.080 --> 00:07:39.840 like it's another form of indirection. 00:07:40.260 --> 00:07:44.860 I'm actually not convinced that that's a particularly necessary trade-off. 00:07:44.860 --> 00:07:52.140 I think at that point, just commit to Git and like make the diff experience versus the Git checkout work really well. 00:07:54.040 --> 00:07:54.780 So I don't know. 00:07:55.320 --> 00:07:57.960 I actually, I think that there's another sort of like small wrinkle on it, 00:07:58.000 --> 00:08:00.800 which is like if you actually do want to watch the transcript, 00:08:01.140 --> 00:08:04.440 like let's just assume you have a use case where you do want to watch the transcript. 00:08:04.780 --> 00:08:08.060 You can't read that stuff anyways as it's happening. 00:08:08.060 --> 00:08:13.420 So you could only ever read it once it's done, which like that's a little bit weird anyways. 00:08:14.360 --> 00:08:18.200 So in a way, if you do want to read the transcript, 00:08:18.340 --> 00:08:20.060 you probably want a summary of what it's about to do. 00:08:22.700 --> 00:08:23.440 I suppose. 00:08:24.060 --> 00:08:27.260 I guess the biggest takeaway here is we need better diff tools. 00:08:27.260 --> 00:08:27.700 Yes. 00:08:29.700 --> 00:08:30.020 Yes. 00:08:30.200 --> 00:08:37.620 And I also like, like having tried a lot of diff tools now and sort of diff tools that also are Gentic integrated, 00:08:37.760 --> 00:08:43.240 like friend of ours, the Tunk, then for modems, which I think is pretty neat, 00:08:43.620 --> 00:08:48.100 but it is also not quite what I want, I feel like. 00:08:50.420 --> 00:08:56.860 So there's definitely, at least for the coding use case, I think like the diff needs to work better. 00:08:57.260 --> 00:09:02.020 It needs to support also like multi-stage commits, like in a PR. 00:09:02.420 --> 00:09:06.480 Maybe not even multi-stage commits, but before you even commit, 00:09:06.740 --> 00:09:11.320 you want to take snapshots of your files at specific times. 00:09:12.200 --> 00:09:16.900 I have this in my custom Pi review extension, 00:09:17.360 --> 00:09:22.800 which is like a desktop UI using Daniel's Glimpse framework. 00:09:23.860 --> 00:09:26.400 And I've used this for months, right? 00:09:26.400 --> 00:09:30.480 And I just found I don't even need that anymore. 00:09:30.660 --> 00:09:33.780 Like I don't need these snapshots before I do a commit. 00:09:34.180 --> 00:09:35.840 I don't care so much anymore, 00:09:36.000 --> 00:09:42.220 but mostly because I try to break down my stuff into very small commits in 00:09:42.220 --> 00:09:43.680 terms of lines of code. 00:09:44.040 --> 00:09:49.800 But I guess that's not a scalable solution because some workflows might just result in a large diff, 00:09:49.880 --> 00:09:51.540 and you might want to have the snapshots. 00:09:51.540 --> 00:09:52.660 Yeah, it's hard. 00:09:53.120 --> 00:09:56.060 I mean, like obviously the goal should be make very small PRs. 00:09:56.380 --> 00:10:00.000 And I think generally that has been true before and after agents. 00:10:00.000 --> 00:10:06.420 There are situations where this doesn't fully work because in addition to 00:10:06.420 --> 00:10:08.840 like the size of the PR, 00:10:09.480 --> 00:10:15.100 there's also the confidence that you have in shipping it or the complexity of testing it. 00:10:15.180 --> 00:10:20.160 So you might actually have a commit even with feature flags that you're not willing to merge yet 00:10:20.160 --> 00:10:21.500 because you want to test it properly. 00:10:21.500 --> 00:10:24.400 And then these have a tendency to getting a little bit larger. 00:10:26.140 --> 00:10:28.700 And then you actually might want to do these checkpoints in between. 00:10:29.000 --> 00:10:32.380 But then when you review it with the agent, it's actually versus the main branch. 00:10:36.260 --> 00:10:37.460 Yeah, I think so. 00:10:37.880 --> 00:10:39.200 When I started working on Pi, 00:10:39.340 --> 00:10:43.940 my realization was that all the models back then were pretty much RL'd on the same trajectories apparently 00:10:43.940 --> 00:10:45.940 because they assumed the base set of tools. 00:10:46.120 --> 00:10:49.360 And with those, they could really work well, which was read, write, edit, bash. 00:10:49.360 --> 00:10:51.720 And I guess the times are changing now. 00:10:52.080 --> 00:10:55.480 And the RL is now geared towards you only need bash. 00:10:55.940 --> 00:10:59.680 GPT models started doing that already last year, I think, 00:11:00.100 --> 00:11:02.640 when they came out with the first five point whatever. 00:11:03.900 --> 00:11:05.040 And now Claude does the same. 00:11:05.120 --> 00:11:07.140 And it stands to reason that other models would do the same. 00:11:07.320 --> 00:11:09.480 So along with these changes in RL, 00:11:09.960 --> 00:11:14.920 the interesting part is that it has a large effect on UX of agentic engineering. 00:11:16.220 --> 00:11:18.980 And we just need to adapt, man. 00:11:19.360 --> 00:11:22.080 Let the RL flow over you. 00:11:22.940 --> 00:11:24.500 Yeah, so I don't think like... 00:11:24.500 --> 00:11:27.320 We have been pretty good, I think, at not fighting the RL. 00:11:28.420 --> 00:11:30.500 Or at least like projecting where it should go. 00:11:30.640 --> 00:11:35.460 But I don't think we have been particularly good at being creative about what the UX around it should be. 00:11:36.280 --> 00:11:40.500 It's like the UX in Pi is basically the thinnest layer over what the models are doing. 00:11:40.500 --> 00:11:45.420 Which probably has like a shelf life. 00:11:45.640 --> 00:11:47.100 And we're sort of approaching. 00:11:47.100 --> 00:11:53.320 One, I think there are like two interesting things here. 00:11:53.380 --> 00:11:55.860 It's like if you had control over the LLM. 00:11:56.240 --> 00:12:00.000 In the sense that you are in fact OpenAI, Anthropic, or you run your own LLM. 00:12:01.160 --> 00:12:05.040 You increasingly are running more than one LLM simultaneously. 00:12:05.040 --> 00:12:07.140 So you're running your large model that does some stuff. 00:12:07.240 --> 00:12:09.500 And then you run a smaller model for either predictive decoding. 00:12:09.500 --> 00:12:14.280 Or because you, for instance, want to already truncate the reasoning output. 00:12:14.620 --> 00:12:17.040 It's like on their side, they're already doing this. 00:12:18.500 --> 00:12:25.520 So basically like there's an almost natural idea now. 00:12:25.660 --> 00:12:28.920 That in addition to the tool call, you might get some commentary around. 00:12:29.640 --> 00:12:31.480 And it's very little today is being done. 00:12:31.720 --> 00:12:35.100 A little bit is coming up with some of the ChatGPT models. 00:12:35.100 --> 00:12:36.420 They have this commentary channel. 00:12:37.060 --> 00:12:40.440 I think it's mainly used now for like this. 00:12:40.880 --> 00:12:43.800 Ambigating the assistant message that comes at the end. 00:12:43.860 --> 00:12:45.220 From the assistant message that comes in between. 00:12:46.500 --> 00:12:49.720 But if you were to run your own inference stack. 00:12:49.840 --> 00:12:51.400 Then theoretically you could actually. 00:12:51.660 --> 00:12:54.020 So like this is a short summary of the description. 00:12:54.440 --> 00:12:56.400 You could invest some extra tokens to summarize it. 00:12:56.480 --> 00:12:57.980 But that's tricky as it streams in. 00:12:58.600 --> 00:13:00.080 But then you could also run Chef on it. 00:13:00.160 --> 00:13:01.860 To sort of classify what's going on. 00:13:02.480 --> 00:13:03.520 So I don't know. 00:13:03.520 --> 00:13:04.360 It's... 00:13:05.100 --> 00:13:08.180 There's probably going to be some experimentation. 00:13:08.180 --> 00:13:10.260 That people might want to run around. 00:13:10.780 --> 00:13:13.220 How to visualize these tool calls in a transcript. 00:13:15.260 --> 00:13:15.980 So yeah. 00:13:16.060 --> 00:13:16.420 I don't know. 00:13:16.920 --> 00:13:19.520 I think that the simplest trick at the moment. 00:13:19.620 --> 00:13:21.200 That works across many models is just. 00:13:21.660 --> 00:13:23.540 You have your tool definition. 00:13:24.060 --> 00:13:26.420 And you add a description, summary. 00:13:26.540 --> 00:13:28.720 Whatever fields that the model has to fill out. 00:13:29.460 --> 00:13:30.920 Make it the first parameter. 00:13:31.240 --> 00:13:33.820 First field within the tool calls argument. 00:13:33.820 --> 00:13:36.840 Then you get it as the first thing usually. 00:13:38.240 --> 00:13:39.880 We should experiment with that. 00:13:39.980 --> 00:13:41.580 I think that could work as a band-aid. 00:13:42.260 --> 00:13:45.520 Ultimately I think we will have to give up on visualizing individual tool calls. 00:13:45.520 --> 00:13:49.220 So that just leads us to the other thing. 00:13:49.220 --> 00:13:52.740 Which is like why was it useful in the first place to have these separate tool calls? 00:13:52.740 --> 00:13:56.240 And a big reason was the approval gates. 00:13:56.240 --> 00:13:57.980 Which Pi obviously never really had. 00:13:58.180 --> 00:14:03.240 But for instance, in Claude it was used to... 00:14:03.240 --> 00:14:05.740 You can have this allow edits by default. 00:14:05.900 --> 00:14:07.920 Which was gated on it using the edit tool. 00:14:07.920 --> 00:14:10.480 And then it would ask for everything else. 00:14:10.640 --> 00:14:14.400 But obviously like the whole thing now moved into a world where either nobody gives a shit anymore. 00:14:14.400 --> 00:14:19.020 Or alternatively they run a second LLM to check the approval. 00:14:22.020 --> 00:14:23.800 To see like should you run this or not. 00:14:23.860 --> 00:14:24.980 So this is this auto mode. 00:14:26.520 --> 00:14:29.000 And what do you think about that? 00:14:29.000 --> 00:14:35.240 Like the idea of running a second classifier or model to make 00:14:35.240 --> 00:14:36.920 some determinations here. 00:14:37.500 --> 00:14:39.840 So first of all the approval gates were always bullshit. 00:14:40.780 --> 00:14:44.660 I don't know a single engineer who is actually using that. 00:14:44.920 --> 00:14:49.000 It's like a checkbox that your harness needs to have to become enterprise ready or something. 00:14:49.100 --> 00:14:50.360 But nobody is actually using that. 00:14:52.020 --> 00:14:54.740 So those went away anyways. 00:14:54.740 --> 00:14:59.900 Because even if like having a second model judge whether a tool call is good or not is 00:14:59.900 --> 00:15:01.920 highly problematic. 00:15:02.700 --> 00:15:03.840 Still in my opinion. 00:15:04.780 --> 00:15:07.960 Because for that the model doesn't just have to look at the tool call usually. 00:15:08.160 --> 00:15:09.680 It also has to look at the full context. 00:15:11.260 --> 00:15:13.020 For it to actually make sense. 00:15:13.340 --> 00:15:17.800 Like I can just have a haiku or a terra or lunar model. 00:15:18.460 --> 00:15:21.400 Look at the single bash call and say yeah this is not dangerous. 00:15:21.400 --> 00:15:24.700 Or this is not destructive in a way that we don't want it to be destructive. 00:15:25.880 --> 00:15:26.880 I don't know man. 00:15:27.100 --> 00:15:29.360 It's just put your fucking agent in a sandbox. 00:15:29.900 --> 00:15:30.480 Call it a day. 00:15:31.960 --> 00:15:32.360 Yeah. 00:15:32.940 --> 00:15:39.300 So I think like we're definitely most likely now reaching the point where the coding agent 00:15:39.300 --> 00:15:43.220 has fully subsumed every other agent format. 00:15:43.220 --> 00:15:48.360 So if you do want to use it in a non-coding environment. 00:15:49.400 --> 00:15:52.440 You're probably going to give it a programming language. 00:15:52.700 --> 00:15:53.780 Which is Turing complete. 00:15:53.980 --> 00:15:54.600 Where good luck. 00:15:55.800 --> 00:16:00.560 So you gotta constrain everything around it. 00:16:02.440 --> 00:16:09.220 Did you say something like there's a code mode implementation in one harness that actually gives the model also bash fully unsiccured? 00:16:09.220 --> 00:16:15.540 I mean this is basically for the most part like code mode now is it exposes all the tools that the agent has. 00:16:15.660 --> 00:16:15.760 Right. 00:16:15.940 --> 00:16:21.860 So code mode in itself might be tightly sandboxed but then you can 00:16:21.860 --> 00:16:25.260 call bash in codex via code mode. 00:16:25.260 --> 00:16:28.240 So then you need to separately sandbox the whole thing anyways. 00:16:28.240 --> 00:16:34.540 So you have like one sandbox and another sandbox which sort of makes sense I guess because like code mode theoretically runs where the harness 00:16:34.540 --> 00:16:35.720 runs which is more trusted. 00:16:36.560 --> 00:16:42.920 And then the bash runs where you're sort of exe.dev or oversell workers or like 00:16:42.920 --> 00:16:44.880 cloudflare durable object sandbox kind of sits. 00:16:44.880 --> 00:16:50.140 But it is all basically in a way really 00:16:50.140 --> 00:16:53.340 like we have now fully committed to this code thing. 00:16:53.460 --> 00:16:53.600 Right. 00:16:53.700 --> 00:16:59.900 I think like at this point we just better get these sandboxes really tightly controlled but like within the 00:16:59.900 --> 00:17:02.000 capabilities of that sandbox. 00:17:03.280 --> 00:17:05.780 The harness basically at this point no longer plays a role. 00:17:05.780 --> 00:17:10.660 Or at the very least it has so little control because the models are so 00:17:10.660 --> 00:17:14.260 yeah programming language slash bash trained. 00:17:15.940 --> 00:17:22.160 I think the question is just I think within Pi for the most part is like how do we make it so that people can 00:17:22.160 --> 00:17:24.840 experiment with how they want to see the transcript? 00:17:25.020 --> 00:17:29.220 Like do we have enough permittives right now to tinker with it? 00:17:29.220 --> 00:17:36.140 And then for someone who builds an agent that is supposed to be self-running a durable object or something similar, 00:17:36.140 --> 00:17:40.160 how should one structure that? 00:17:40.380 --> 00:17:44.280 Do we have enough primitives within Pi to make it acceptable? 00:17:46.720 --> 00:17:49.440 I think that is going to be a lot of interesting. 00:17:49.700 --> 00:17:53.080 Like if someone has thoughts up there, I think we would love to hear. 00:17:53.080 --> 00:17:59.840 But yeah, in a way it's the end of custom tools in many ways. 00:18:00.440 --> 00:18:00.620 Yeah. 00:18:01.800 --> 00:18:07.660 I do have to say that custom tools have historically been not very helpful to my workflows at least. 00:18:07.940 --> 00:18:14.140 So I just recently built a new tool called Pi History Handoff, 00:18:14.140 --> 00:18:18.120 which is sort of kind of like the new codex experimental feature. 00:18:18.120 --> 00:18:25.300 Well, the LLM decides it wants to basically clear the context and just leave a summary of itself there. 00:18:25.480 --> 00:18:33.080 It's a fancy form of compaction plus a tool to give you to give the LLM a way to search in the compacted history. 00:18:34.340 --> 00:18:38.800 That's the only thing recently that I built that makes sense for my workflows. 00:18:38.960 --> 00:18:42.020 But other than that, I don't really have many custom tools. 00:18:42.020 --> 00:18:48.120 I think one thing is super interesting right now is like we're simultaneously seeing a full commitment to bash. 00:18:48.940 --> 00:18:51.580 At the same time, we see a full resurgence of MCP. 00:18:53.260 --> 00:18:59.540 And I still don't really fully know how to put these things together because I think MCP 00:18:59.540 --> 00:19:02.440 solves the off thing, which is important. 00:19:02.440 --> 00:19:06.660 It does set some constraints on the tool execution, 00:19:06.660 --> 00:19:10.080 but then the way we wire it together is like a code mode. 00:19:11.060 --> 00:19:14.040 So it is interesting. 00:19:14.220 --> 00:19:16.120 I would say like where this all lands today. 00:19:16.620 --> 00:19:16.760 Yeah. 00:19:17.360 --> 00:19:20.600 So I think you looked into MCP a little bit more recently. 00:19:20.600 --> 00:19:27.080 And my understanding is that the input schema must be defined 00:19:27.080 --> 00:19:33.920 for an MCP tool, but the output schema can be, but it's optional. 00:19:34.380 --> 00:19:37.620 And for the most part, like you actually need two channels, right? 00:19:37.660 --> 00:19:42.720 You need one channel that goes to the LLM, which is likely unstructured to conserve some tokens. 00:19:42.720 --> 00:19:49.200 And you would need a second channel that is structured so you can use an MCP tool as part of code mode, for example. 00:19:49.720 --> 00:19:50.580 What's the state of that? 00:19:50.900 --> 00:19:54.120 Like, do we actually have those two channels for the output or is that just one channel? 00:19:56.020 --> 00:19:58.080 For the most part, there's just one channel. 00:19:59.280 --> 00:20:02.920 I think it's what is interesting is like, so we, when we built radius, 00:20:02.920 --> 00:20:07.300 we wanted to be so that you can talk to the agent and reconfigures radius. 00:20:07.580 --> 00:20:08.840 And we tried a bunch of things. 00:20:08.840 --> 00:20:13.700 And what we actually found to work the best today with these models is OpenAPI. 00:20:14.480 --> 00:20:21.700 So we actually, so we added an endpoint where you can describe your task and what you get back is a rack filtered. 00:20:22.280 --> 00:20:24.860 These are the endpoints that you might want to use plus the OpenAPI schema. 00:20:25.340 --> 00:20:27.480 And you can actually expose that via MCP. 00:20:27.640 --> 00:20:31.240 So like it is actually very possible to use this pattern against MCP. 00:20:32.240 --> 00:20:37.420 The only problem is like MCP actually at that point really only does off. 00:20:37.420 --> 00:20:42.540 It's like it's RPC plus off, which is a very, very narrow way of using MCP. 00:20:43.140 --> 00:20:46.980 And if you were to build a server that works like this, you don't really, 00:20:46.980 --> 00:20:51.260 you would only output structured output, which MCP is not. 00:20:51.840 --> 00:20:58.280 Like if you, if you were to put this in a traditional MCP, where like you take the entire output and stash it into the context, 00:20:58.280 --> 00:21:00.680 that doesn't really work. 00:21:00.680 --> 00:21:06.780 So like there's an in-between field now where you could actually use MCP almost like an RPC 00:21:06.780 --> 00:21:10.080 system, which only outputs structured data. 00:21:10.920 --> 00:21:17.100 And then your code mode hopefully has enough utility in it to sort of do filtering 00:21:17.100 --> 00:21:18.320 and post-processing, whatever. 00:21:19.440 --> 00:21:21.580 But most MCP servers don't work like this today. 00:21:21.700 --> 00:21:24.680 Like most MCP servers output like various compressed markdown. 00:21:25.300 --> 00:21:26.260 Plain text, whatever, yeah. 00:21:26.260 --> 00:21:29.060 So the protocol is actually really useful now. 00:21:29.300 --> 00:21:33.420 The servers are like not sure if they should target one versus the other. 00:21:35.420 --> 00:21:36.700 Yeah, it's interesting. 00:21:37.120 --> 00:21:42.820 So I guess the takeaway for future MCP server authors and existing MCP server maintainers is 00:21:42.820 --> 00:21:46.300 provide structured output because we're going to go full code mode. 00:21:47.080 --> 00:21:47.660 I think so. 00:21:47.660 --> 00:21:50.540 I think, I think that this is probably what's going to happen. 00:21:52.640 --> 00:21:53.120 Yeah. 00:21:53.180 --> 00:21:53.860 Yeah, that kind of makes sense. 00:21:53.920 --> 00:22:00.180 I still would like, like in Pi, the difference between what the model sees and what harness or 00:22:00.180 --> 00:22:05.160 consumer of events sees has mostly always been there. 00:22:06.640 --> 00:22:11.240 Including for tools and for entries in the transcript. 00:22:11.240 --> 00:22:11.280 Right. 00:22:12.060 --> 00:22:12.440 And I think this. 00:22:12.440 --> 00:22:15.960 And you have to say, like if you are bash only, there's only one output, right? 00:22:16.080 --> 00:22:16.320 Sure. 00:22:16.720 --> 00:22:16.960 Sure. 00:22:17.040 --> 00:22:20.520 If you're bash only, you're kind of, you're kind of, yeah, full part. 00:22:20.680 --> 00:22:25.360 But I think a future will be composed via MCP plus code mode. 00:22:25.560 --> 00:22:26.620 That kind of makes sense. 00:22:26.980 --> 00:22:30.060 I worry though that the models aren't as trained on that just yet. 00:22:31.340 --> 00:22:34.240 I mean, I analyzed a lot of codex transcripts. 00:22:34.240 --> 00:22:40.680 So when, when Astra released, I, I did a whole bunch of like work just in the codex harness 00:22:40.680 --> 00:22:42.540 so that I can see how it behaves. 00:22:43.460 --> 00:22:50.460 And, um, I can see that Astra loves to use code mode, even for situations where it wouldn't have to. 00:22:50.520 --> 00:22:55.380 Like I've seen it uses code mode to apply patch, which is like, why would you do that? 00:22:55.500 --> 00:22:58.140 It's just like a level of interaction that was not really necessary. 00:22:58.140 --> 00:23:03.920 Um, but there are differences in how code mode works in the different models. 00:23:04.280 --> 00:23:04.440 Right. 00:23:04.580 --> 00:23:10.860 And so, um, and the, the surface, the surface area of code mode is obviously much bigger than the surface area of, 00:23:10.860 --> 00:23:12.600 of just a regular tool code channel. 00:23:13.700 --> 00:23:19.100 Um, so I think this requires a lot of testing and evaluation and 00:23:19.100 --> 00:23:22.000 it's unclear where it's going. 00:23:22.000 --> 00:23:27.840 Um, there's also some tension with local models because I guarantee you that local models will 00:23:27.840 --> 00:23:29.120 suck at code mode. 00:23:29.460 --> 00:23:31.560 So yeah. 00:23:32.320 --> 00:23:37.660 I think the interesting use case why I really want to get code mode into PI is not even MCP. 00:23:37.900 --> 00:23:41.880 It is actually Jev because for Jev to make sense, 00:23:41.880 --> 00:23:47.320 you need step one, generate some code that then Jev can evaluate over and over. 00:23:47.320 --> 00:23:52.040 So it's like you basically create like a state machine or like a workflow where on every iteration, 00:23:52.040 --> 00:23:54.820 Jev makes the determination of like which path to go down to. 00:23:54.820 --> 00:24:00.440 So, so it will become a really useful primitive, um, to have within PI harness. 00:24:03.080 --> 00:24:09.560 I'm actually more convinced that it's useful for Jev and less convinced that it's useful for MCP right now, unless the MCP servers are changing. 00:24:10.160 --> 00:24:11.460 Um, but we'll see. 00:24:12.140 --> 00:24:12.520 Nice. 00:24:13.680 --> 00:24:15.160 Well, interesting times. 00:24:15.160 --> 00:24:17.900 Should we go back to our beds and die off man flu? 00:24:18.720 --> 00:24:19.560 Like the babies we are? 00:24:19.560 --> 00:24:20.100 I think so. 00:24:20.380 --> 00:24:20.700 Okay. 00:24:20.960 --> 00:24:21.320 All right. 00:24:22.040 --> 00:24:22.860 Happy Monday. 00:24:23.220 --> 00:24:23.800 Happy Monday. 00:24:23.800 --> 00:24:23.960 Okay. 00:24:24.820 --> 00:24:25.820 Happy Monday.