Transcript EN 0:00 Why Durable? 0:02 So Pi, the coding agent, has historically been you are a single human, 0:08 you sit in front of a computer or you have a remote machine and you SSH into it. 0:12 And you start your Pi session, you do a thing and you stop the Pi session and then you usually don't care about the session anymore. 0:18 So it's more like a one-to-one relationship, the sessions are ephemeral, 0:22 more or less, and everything fits snugly into memory as well because Pi 0:28 historically loads the entire session into memory just to do a bunch of things. 0:36 It's not entirely necessary, it's just a historical artifact stemming from the fact that it's a one-on-one ephemeral session, 0:42 usually not super long, right? 0:44 However, does that fit all agentic applications? 0:49 I guess the answer is no. 0:52 If you want to have a Slack bot that stays with you for years and years and can answer multiple 0:58 people in parallel or have threads and answer there, 1:03 Pi is not your best option. 1:04 It can be done and we have like Piedolf, our Discord bot, 1:09 who's been asleep for a couple of months because it was not so useful. 1:14 But it's kind of not what Pi the coding agent was built for. 1:19 So earlier this summer we decided we needed a different kind of substrate, 1:24 a different kind of framework or library or whatever you want to call it, 1:28 that allows you to model this kind of agent or agentic application as well. 1:33 Long-running agents, multiplayer, there's multiple users or other agents can interact 1:39 with each other. 1:42 And not everything should be in RAM, in memory in total. 1:47 And it should be able to run anywhere. 1:49 Like Pi is obviously mostly bound to a computer-like system. 1:53 We also want to run on Vercel, on Cloudflare, 1:57 on E2B, on whatever you can come up with, 2:01 where the computing substrate is potentially different than a process in an operating system. 2:07 For example, Cloudflare, Durable Objects, it's an entirely different execution model than what you have in a process in Node 2:14 on your Mac or Windows machine. 2:16 So yeah, and that was how Durable was born. 2:19 And I think it's also kind of useful to just point out what maybe some of the 2:25 classical limitations are of Pi's existing session model. 2:29 And why it works quite well for the Pi coding agent, but like why it shows some limits. 2:34 And actually I think like to a large degree, those limitations are also shared with a lot of other agents. 2:41 So for a start, it writes JSON-L files. 2:45 And they're definitely useful. 2:46 There's nothing fundamentally wrong with them. 2:48 But they're append, right? 2:53 So it's like it's not very easy to rewrite those files. 2:58 And as you're putting larger and larger workloads into them, they're accumulating a lot of intermediary 3:04 results potentially. 3:08 That, yeah, are potentially wasteful. 3:12 But the second thing is also that if you are going to have sub-agents, 3:16 you would actually have to split that across dependent sessions. 3:22 So Pi's tree works really, 3:25 really well, particularly if you are a single writer, single user. 3:29 You can navigate to different parts of the tree, rewind the session, do some stuff. 3:32 But modeling sub-agents into a single session 3:36 would interleave lines, 3:41 would require some locking that doesn't exist today. 3:44 So like the entire tree system as it exists is very cool, 3:50 but has some limitations, let's say. 3:53 So a lot of the work towards durable started, 3:58 I think, in some way or another by figuring out how to make this stuff work with a database. 4:04 And really, I think there are multiple things. 4:08 Like durable means you should be able to stop and start it again. 4:12 So it's like it suspends and picks up work. 4:15 And that if you were to have sub-agents in it or you were to have multiple parallel things in it, 4:20 it would extend to all the branches which are active at any point in time. 4:25 And then the second part is that while 4:31 Pi is quite durable even today, 4:35 it is very easy to build stuff with it where you 4:41 cannot pick up the work. 4:43 And even the core agent look that we have today, if you were to continue, it could not continue on its own. 4:49 So like if you were to shut down and you bring it up, you would have to write continue or something into it to actually continue to work. 4:56 And there are a lot of features that you might want to put into Pi that require extra state to be 5:03 persisted along the way. 5:04 So the classic example would be if you interrupt a tool execute 5:10 in order to ask the user for approval, 5:14 which again, not really the most useful case right now because I think tool 5:21 approvals are largely gone. 5:22 But you could not really suspend the agent at this point in time and say like, 5:25 hey, we need to figure out if we can go ahead or not and then continue from there. 5:30 So it's like the system is quite limited in being able to suspend at points where we didn't intend 5:35 it unsuspending before. 5:38 Yeah, I think we should probably just go through what durable means because we throw it around. 5:44 But not a lot of people are actually very familiar with the concept. 5:49 So in old Pi, not old Pi, in Pi coding agent, 5:53 you start a session, you let the agent do something, your machine dies. 5:58 You spin up Pi again, the same session, and that Armin said you have to write continue. 6:04 If there was an extension that did something in the background, that extension needs to be written in such a way that it actually spins up the background work again. 6:11 And Pi coding agent didn't help the extension with that at all. 6:14 Like everybody had to reinvent how to do that. 6:17 If you had a tool that was being executed when the process crashed, 6:23 that tool is just aborted and not run again. 6:26 And the agent has to figure out if it should run it again and if so how and if the tool execution was a 6:32 destructive thing or not and so on and so forth. 6:35 And then Pi doesn't have real concurrency, 6:41 meaning you always have a single agent loop. 6:46 And if the process dies, we do not recover within that agent loop. 6:51 So an agent loop is basically get the user input, 6:54 send it to the LLM, get the response back, check if there's tool calls, 6:59 execute the tools, and then do that in a loop until the LLM says I'm done. 7:02 In Pi coding agent, that just goes away. 7:06 We do not keep track of where we are in that loop basically, right? 7:11 So with Pi durable, we introduced a bunch of things that make this automatic basically. 7:18 So it kind of forces you into a durable way of thinking. 7:21 So first of all, that loop I just described, that is actually decomposed into what we call tasks. 7:27 And the task is basically just saying I intend to do a thing. 7:31 I store this in a storage so I can remember it later. 7:35 I do the thing and then I store the result of the thing I did, right? 7:41 So this gives me kind of like a sandwich and I can resume at any point here. 7:46 I can resume in between intent and execution of what we call the effect. 7:50 And in between the effect and the final result and after the result, right? 7:57 And this is very nice because now you just write the thing that says I'm about to do the thing. 8:02 I am doing the thing. 8:03 I did the thing. 8:05 And if you resume, we know exactly where in this sandwich you were and can resume from there. 8:10 And this extends to everything. 8:12 This extends to the request. 8:13 It's being made to the LLM provider. 8:15 This extends to tools which can now mark themselves as replay safe. 8:21 For example, a read tool is replay safe. 8:24 You can just call it again and you get whatever. 8:28 A sub-agent tool is also replay safe because that same agent runs the same 8:34 tasks as the main agent. 8:36 So all of this durability of the execution graph applies to it as well. 8:40 So that agent can just be spun up again and be told, 8:44 not even be told, it automatically does continuous from where it left off. 8:49 And so this is probably the biggest selling point of Pi Durable. 8:53 We took some inspiration from structured concurrency. 8:59 It doesn't cleanly apply to everything in an agent, which to me was a very interesting observation. 9:05 So structured concurrency is basically something owns another thing that does something. 9:11 And this other thing is not allowed to survive past the lifetime of this. 9:16 This works for like 90% of tasks inside Pi Durable. 9:19 But there's a handful of things that need this child execution to survive 9:25 the lifetime of the parent. 9:27 So it's not 100% structured concurrency, but it's very close to that. 9:32 And it gives you an escape hatch if you don't need it. 9:36 Yeah, and that's durability. 9:37 I hope any of that makes sense. 9:38 And I think there's another thing we're mentioning. 9:40 So if you go to the earendil works GitHub repository, you will find that there's a project called Absurd, 9:46 which is a durable workflow engine sitting on top of Postgres. 9:49 And we built that originally to support a Pi-driven agent, 9:55 which is called Levos, 9:57 to basically continue and suspend across basically a cloud deployment. 10:05 So when we built this originally, the assumption was, well, sandbox are quite expensive. 10:09 So we want to build the agent in a way where you just distribute it to any worker that wants to work on it 10:14 and continue work doing there. 10:17 And unsurprisingly, the problem that we ran into quite early on, 10:22 I think a lot of people ran into it, is that there's a lot of states that's not just the agent state. 10:28 So initially just transmitting the session transcript around was obviously not enough. 10:33 Agents love to write to the file system and so forth. 10:35 And so you very quickly discover that durable workflow engines are great, 10:43 but unless the agent harness actually really supports them well, that's not enough. 10:50 So like the agent harness in itself needs to be durable. 10:53 Now, that doesn't mean that everything becomes durable, right? 10:56 So it's like if the file system now holds on to really important data, 11:00 you also need to find a way to either snapshot that, 11:02 or you build the agent around in a way that it can go away. 11:06 And so I think for someone who is looking at this and saying like, 11:10 hey, we have a Dora workflow engine, all of our problems are gone, 11:12 it's like you might still want to use it because it might still be useful. 11:16 But just having that did not actually make your agent durable. 11:22 And in fact, there's still a chance that the agent will not come out particularly durable out of the box 11:29 if you let the agent just write files to file systems that you never bring back or stuff like this. 11:35 So there's things inside the harness that can all be durable, 11:40 but the harness also operates on an external environment. 11:43 And in the effect sandwich, which I described earlier, 11:46 the intent is the durable part inside the harness, 11:48 the effect, which is write to a file, execute bash, call some service, deploy something, blah, blah. 11:54 That is the external system and you do not have control over that. 11:56 Well, usually. 11:58 So the only way to make this part durable as well is to have some kind of, 12:02 and I can't pronounce the fucking word, 12:04 basically, if I say deploy with the ID, blah, blah, blah, 12:11 and I call this again and the deploy for that ID is already running, 12:16 it shouldn't trigger again. 12:17 That's what I'm saying. 12:18 So if your external system works like this, great. 12:20 File systems don't work like this unless you have like a snapshotting file system. 12:25 But yeah, the thing with durability is you can have a workflow engine like temporal or absurd 12:31 or I don't know what other stuff exists out there. 12:35 But then you run into issues of atomicity. 12:38 Like for durability to work, everything that needs to be durable in one step 12:42 needs to be atomically persisted. 12:45 And if you have this outside of the agent, 12:50 then anything inside of the agent that needs atomic persistence for durability 12:54 needs to be shoehorned into that somehow. 12:57 And that really doesn't quite work. 12:59 So that's why we decided for Pi Durable to actually make it a durable workflow engine itself, 13:04 which solves a lot of problems because it also allows you to build your own tasks 13:09 that run inside this scheduler and durable scheduler for your own application. 13:14 And it also allows you, and that's the second part of durability, 13:18 persistence of state, not only of the harness, but also of your application. 13:23 Now, obviously, you do not want to stuff every application state you have into the harness storage. 13:30 But what you can do is you persist large blobs, for example, into your own storage. 13:37 And then inside the harness storage, you persist a pointer to that persisted blob. 13:40 And with that, it's very easy to have durable execution and durable storage of state. 13:47 I think another thing worth remembering here is 13:53 so absurd, like temporal, 13:57 and I think that gazillions of other durable workflow execution engines out there, 14:02 they were actually largely built on one premise that is now quickly being invalidated. 14:09 And that premise was that you can actually cheat on durability quite a bit 14:14 by assuming that the code that is durable is sort of maintained by an expert programmer. 14:22 And I'm saying this because the way almost all durable workflow engines work today 14:28 is they either say like your code has to be fully deterministic on the workflow orchestration part, 14:35 so that if I suspend for like a week and I come back in a week later, 14:39 the code is the same, so I just run through all the checkpoints, 14:43 I'm pulling the data as it was written into durable storage, 14:46 and I'm continuing from there. 14:49 Or it has to be written in a way that is maybe non-deterministic, 14:53 but all the branches that you were taking on the durable information is still compatible 14:58 or has like explicit code in it for upgrades. 15:00 And the problem with this with LLMs is that we are 15:06 now living in a world 15:08 where the agents are writing their own code. 15:10 And it's not that the agents cannot be expert programmers, 15:12 but the agents, it is very hard for them to basically maintain 15:18 like a perfect understanding of all the possible tasks 15:22 that will ever come by again in this durable state. 15:25 And so this, I would say like this is an unsolvable problem, 15:27 but the trade-off now is less to make it friendly for a programmer 15:31 by abstracting it away, but to make it a lot more explicit. 15:35 Because if it's explicit, then the agent can see it. 15:38 It's like, oh, I wrote this durable information away 15:40 and we need to load it again. 15:41 So the trade-off actually now is sort of almost entirely inverse 15:45 to what we did with Absurd, 15:46 where we tried to make it as friendly for a human programmer to do. 15:49 And now it's like S in your face. 15:53 So did the LLM. 15:55 There's going to be a little bit more code, 15:56 but the LLM is going to be explicitly going through checkpoints. 16:00 So like as an example, state suspension in Absurd was like, 16:06 yeah, like every time you went to a step, 16:08 we would increment the counter. 16:09 And so like if you, if a loop for this four times, 16:12 then there's a hidden like first time, second time, 16:15 third time and so forth. 16:17 In Pi Durable, what is just a memo function, 16:20 where you give it a name and that's the name. 16:24 And if you want to version it, you version it. 16:27 So there's a lot less hidden magic in durable 16:30 than there is in temporal or Absurd. 16:33 Yeah, definitely. 16:33 Like the workflow scheduler is like 1,200 lines of code 16:39 that you can actually read and understand. 16:41 And it's very easy to reason about the execution graph. 16:44 And that is actually the difference to temporal or Absurd. 16:47 In temporal or Absurd, 16:49 you kind of pretend you're writing a box standard JavaScript program. 16:53 And it just will magically pick up from where you left off, right? 16:56 That's more true for temporal than Absurd, I think. 17:00 And that is, as you said, for humans, a great model 17:03 because you kind of sort of understand. 17:04 But then if you actually use it in anger, 17:07 you figure out, 17:08 they actually have to do a lot of shit to make this work. 17:12 Like something as simple as set timeout, 17:14 they have to kind of hook into 17:18 and ensure that it works with this. 17:20 And the biggest problem, actually, 17:22 with these kinds of workflow engines 17:23 is that you have an execution graph. 17:26 Like function A calls function B, 17:27 calls function C. 17:29 And then somewhere in there, 17:31 we stopped the process 17:34 and now we restart it. 17:35 And now we have to reestablish the call stack, right? 17:38 Function A, function B, function C. 17:40 So how do you do this? 17:42 You have to replay the fucking shit. 17:44 And the replay works like this. 17:45 You go into function A again 17:47 and function A calls something that says, 17:49 do the thing. 17:50 And instead of actually doing the thing, 17:52 it gives you the last result. 17:53 The thing did. 17:54 The last time you ran it. 17:56 Then you take that 17:57 and you call function B with that. 17:58 And that does the same thing. 18:00 And then you take that 18:00 and call function C. 18:01 And it's a terrible model, actually, 18:03 if you think about it. 18:04 Because it requires a journal of everything 18:06 and it reruns the entire code. 18:09 And if anything, as Armin said, 18:10 changes in any of these functions, 18:13 you are fucked. 18:14 One of the biggest criticisms of Temporal 18:16 is that versioning is just terrible. 18:18 So what does Pi Durable actually do? 18:21 Pi Durable doesn't give you 18:23 a classic call stack of JavaScript functions. 18:25 Pi Durable gives you a tree of tasks. 18:30 And each task is self-contained. 18:32 It can call other tasks and wait for them. 18:35 And that's trivial to reestablish 18:38 compared to a normal JavaScript function call graph. 18:43 And that's the big difference here. 18:44 The units of work are very small, 18:46 are very easy. 18:47 The intent, effect, storage kind of model. 18:52 And if you want to do more things in parallel, 18:55 you can. 18:56 It's very easy. 18:56 You just create a new task 18:57 and other tasks can wait for that. 18:59 And that's it. 19:00 And it's very super easy to kind of rewind, 19:03 to kind of reestablish that virtual call graph, 19:06 if you want to call that. 19:08 So since we already talked about the effects. 19:09 Oh, yeah. 19:10 We should maybe talk about the other elephant in the room, 19:14 which is effect.js. 19:16 And before we talk about effect.js, 19:18 I also made a little side note here 19:22 on the sort of constantly implied elephant in the room, 19:26 which is JavaScript in the first place. 19:27 And I think this, again, is TypeScript. 19:32 And it's TypeScript because we really like 19:34 the idea of agents extending themselves. 19:36 So it's like right now, 19:37 JavaScript seems like a good environment. 19:40 But I also want to say, 19:41 in many ways, 19:43 what we are trying to do here 19:44 is we're trying to build 19:45 almost like 19:47 a way of building an agent 19:50 and that stores itself durably. 19:53 Durable storage means 19:54 commitment to some stable system 19:57 that you should be able to describe 20:00 in a way where you could rebuild 20:01 in a different language, right? 20:02 So it's like, 20:03 keep in mind, 20:04 it's like this is for sure JavaScript right now 20:06 and might even be JavaScript 20:07 for a very, very long time. 20:09 But the goal is that the system in itself is so, 20:13 like, 20:13 you should be able to write an agent in this 20:15 that maintains a two-year-old Slack channel 20:18 and then you eventually want to rewrite it in TypeScript, 20:20 but you can pick up the data as it was. 20:22 Right? 20:22 So the idea here is like 20:23 that the underlying system 20:24 is a trustworthy foundation to build on. 20:30 So why did we not use Effect.js? 20:34 Because it didn't solve any of our issues. 20:36 So out-of-the-box, 20:37 Effect.js is great for in-memory, 20:40 in-process, 20:41 structured concurrency, 20:42 for example, 20:42 along with a metric ton of other stuff 20:44 like structured errors, 20:46 dependency injection, 20:48 cancellation and logging 20:49 or telemetry traces 20:50 and all of that. 20:51 It's fantastic. 20:52 Truly fantastic. 20:53 Although I have to say 20:54 that one of the maintainers recently said 20:56 he found out 20:58 he can no longer write Effect.js himself. 21:03 But it's not a problem 21:04 because agents understand it. 21:08 So again, 21:09 we love Effect.js 21:10 and I think it's the right solution 21:11 for many, many problems. 21:12 But for us, 21:13 it didn't solve a problem 21:14 because Effect.js are out-of-the-box. 21:16 That's exactly zero 21:18 for durable execution. 21:20 So the function A, 21:22 function B, 21:23 function C thing, 21:24 that works great in memory, 21:28 but Effect.js doesn't give you any primitives 21:30 to reestablish that call graph 21:32 if you have to resume the process. 21:35 So then the question is, 21:38 okay, 21:38 if it doesn't help us with durability, 21:39 can we get anything else out of it 21:42 and how hard would it be 21:43 to add durability to this? 21:45 And it turns out 21:45 adding durability to this 21:47 basically ends up 21:49 looking like what we have. 21:51 A task-based workflow engine. 21:54 There's Effect.js workflow, 21:55 the package, 21:56 or effect 21:57 slash workflow, 21:59 and it's basically that. 22:00 They call tasks activities 22:01 and it's basically 22:03 the same thing, 22:05 except you also inherit all of Effect.js 22:07 and we didn't want to make that. 22:09 It's not a bet 22:10 because a lot of people build on Effect.js, 22:12 but we want it to be self-contained 22:14 and in terms of portability, 22:16 it's actually very easy to port 22:18 Pi Durable to something else. 22:20 Ask me how I know. 22:24 It's less trivial 22:25 to port Effect.js 22:27 and its generator-based semantics to something else. 22:31 So ultimately, 22:32 that's why we decided against Effect.js. 22:35 And I also think 22:35 a useful, 22:37 so I actually think 22:38 Effect is quite useful. 22:39 I don't actually think it's particularly useful for Durable 22:41 for a bunch of reasons, 22:42 but I did 22:43 in an attempt 22:44 to use what we built and also to, 22:49 like a lot of the side projects that I'm doing 22:50 just to figure out how to build a software factory 22:53 and things like that 22:54 are in fact based on Effect. 22:57 And the reason 22:58 they're based on Effect 22:58 is because I think 22:59 Effect does actually 23:00 solve some of the guardrail problems today because like, 23:04 for instance, 23:05 it strongly encourages 23:06 you not to throw exceptions. 23:08 It strongly encourages 23:09 you to compose services. 23:11 So it is a pretty useful crutch almost 23:14 to get a little bit 23:16 of structure 23:18 into a TypeScript project 23:19 where it would otherwise 23:20 not have a lot of it 23:21 when you leave an agent to it. 23:23 But simultaneously, 23:26 if I then start the same project 23:29 without Effect, 23:30 like I did this 23:31 a bunch of times now, 23:32 like build this with Effect, 23:33 build it without Effect, 23:34 it's actually not entirely clear 23:35 that the Effect solution 23:36 is also definitely 23:38 better for the agent 23:39 because for the very least, 23:41 you actually do end up 23:42 ironically with more code usually. 23:44 And more code 23:45 is still a little bit of a challenge 23:47 for this. 23:49 But I think the Effect people 23:50 are on the right track 23:51 in general about expressing software like this. 23:55 And I already 23:55 proposed to 23:57 at least two of them, 23:59 why not try to build 24:01 it into TypeScript, 24:04 have a version of TypeScript 24:06 that expresses effects. 24:08 Because then 24:09 a lot of these hacks 24:11 that are currently there 24:12 with Effect Gen 24:14 and the service declarations might actually become 24:18 core permittives of a language. 24:20 I definitely want someone 24:22 to do some experiments 24:23 there and see where it goes. 24:25 But I think 24:26 the portability part about it is a pretty 24:29 big part. 24:31 The Effect 24:32 adds a lot of effect to the 24:35 effect ecosystem. 24:38 It's viral, 24:38 basically. 24:39 It is very viral. 24:40 Even using a database usually goes through other effect libraries. 24:43 And so 24:43 you inherit all of that, 24:47 which you might 24:49 individually want to do 24:50 if you build an application. 24:51 I think it's a much tougher sell 24:53 if you want to build 24:54 a foundation framework 24:55 that other people then use. 24:58 Because you would basically 24:59 force that choice onto the user. 25:02 Which is pretty tough. 25:04 Again, 25:05 we are absolutely not dunking on Effect. 25:08 We actually really, 25:09 really like it. 25:10 And if you're building applications, 25:11 you should use it. 25:13 If you build applications with TypeScript. 25:15 Yeah. 25:17 Yeah. 25:18 So that's what we have today. 25:20 I think one thing that 25:21 we want people to do 25:23 is to play with it and particularly 25:25 see if the data model that we have makes sense. 25:29 And I think the vacation 25:31 example that we have 25:32 in the repository 25:33 is a good starting point. 25:35 Because we would like to know 25:37 that this solves your problems 25:41 before we fully commit to it and then rewire the world 25:44 on top of that. 25:46 But that you mean rewiring Pi Coding Agent? 25:50 For instance, 25:51 the Pi Coding Agent on it 25:52 or just to build more 25:54 functionality that people 25:58 might want to have because the core 26:00 doesn't support it yet. 26:01 I know that 26:02 some of the people that were playing 26:03 with it already said like, hey, 26:04 maybe there needs to be more abstraction over the state system. 26:09 Whatever we would want to build 26:11 on top of that 26:12 even just to assist people to building with it. 26:16 Ideally, 26:16 we'd know that the core 26:18 data storage abstraction system works. 26:22 So, 26:22 I have high confidence that 26:24 this kind of low-level API 26:26 that we now have is a good primitive pending some plus and minuses. 26:31 But I do agree that, for example, 26:33 some sugar on top of it, 26:35 specifically with regards to state management, 26:36 is probably a good idea. 26:38 Not just for humans, but for agents as well. 26:40 It's just unclear 26:41 to me what a nice abstraction there would look like. 26:45 And also, 26:46 we are not just relying on external builders using 26:48 Pi Durable to figure out what works and what doesn't. 26:51 We are actually betting on this 26:52 for all of our Earendil products going forward. 26:56 And in the future, 26:57 that means in the 26:57 next couple of weeks, 26:58 we have a bunch of team members who 27:00 will use it in 27:01 anger, build some tools that we actually use in 27:04 production that are 27:05 very focused on a specific task, and we will document our journey and what we learned and obviously also provide the code 27:12 that we created 27:13 in a part of that process so other people can get inspired by that. 27:18 Also, don't be surprised if there are some new faces coming up on these meditations in the future. 27:23 That's all. 27:23 Bye! 27:24 Bye-bye.
Download captions .vtt ↓