WEBVTT 00:00:00.000 --> 00:00:06.000 Hello, today it's just me. I want to talk with you a little bit about what's hard about AI software engineering. 00:00:08.000 --> 00:00:13.680 And this is in part because I got some really lovely emails recently where people shared their experiences, 00:00:13.680 --> 00:00:18.000 but also because I asked on Twitter and I actually got a ton of responses. 00:00:19.000 --> 00:00:22.580 And really the question that I have sometimes is, 00:00:22.580 --> 00:00:28.000 how is it that after 18 months of using agentic software engineering tools like Claude Code, 00:00:28.000 --> 00:00:31.000 nothing really improved. 00:00:32.000 --> 00:00:38.160 And I know this is a little bit not true because clearly the models are way more capable, 00:00:38.160 --> 00:00:45.000 but it didn't really feel like the hard problems are solved. 00:00:46.000 --> 00:00:48.000 And in some ways things got worse. 00:00:48.000 --> 00:00:54.500 And I think most consistently the problem that I 00:00:54.500 --> 00:01:00.220 see that we have working ourselves with agentic tools, 00:01:00.220 --> 00:01:06.000 as well as what I see other people report is really that because we haven't solved some of those fundamental problems. 00:01:07.080 --> 00:01:08.080 Speed is now becoming 00:01:08.080 --> 00:01:09.360 a new problem. 00:01:10.000 --> 00:01:17.000 It generates so many more pieces of code that our capacity to keeping up is going down and down. 00:01:18.000 --> 00:01:24.340 And it wouldn't be so 00:01:24.340 --> 00:01:30.000 bad if we just were willing to slow down, but we're not. 00:01:30.000 --> 00:01:34.000 We're committing ourselves to being faster and faster. 00:01:34.000 --> 00:01:36.000 And that's because we're no longer doing this alone. 00:01:36.000 --> 00:01:37.000 We're doing this in teams. 00:01:37.000 --> 00:01:43.240 And I can't tell you how many of you have expressed 00:01:43.240 --> 00:01:49.400 frustration about the fact that other people are sending you slop and other people are sending you 00:01:49.400 --> 00:01:51.000 pull requests at their volume they just can't keep up with. 00:01:51.000 --> 00:01:56.840 And so we have this unfortunate situation where the models are both 00:01:56.840 --> 00:02:03.000 getting really, really good, but not good enough that we still can't trust them. 00:02:04.000 --> 00:02:09.000 And this gap doesn't seem to be improving meaningfully. 00:02:10.000 --> 00:02:17.000 And it's actually for me a huge problem right now on a very personal level, because I mean, I wrote about this before. 00:02:17.000 --> 00:02:23.340 I don't trust Astra and increasingly I don't trust Fable to be 00:02:23.340 --> 00:02:29.500 good software engineers, because what's happening in their training processes is that these models are 00:02:29.500 --> 00:02:33.000 rewarded for succeeding, but not for writing code that I like. 00:02:34.000 --> 00:02:39.000 And in many ways, some of the behaviors you get out of these models now are counterproductive. 00:02:39.000 --> 00:02:42.540 They are, 00:02:42.540 --> 00:02:49.000 for instance, no longer willing to use edit tools as much as they used to, which means that as I'm watching the edits fly by in 00:02:49.000 --> 00:02:52.920 Pi, very often I can't even understand what the model is doing, 00:02:52.920 --> 00:02:54.000 let alone which tools it's invoking. 00:02:54.000 --> 00:02:58.000 And so as all of this is going on and on, we're shipping faster. 00:02:58.000 --> 00:03:00.000 The quality of the code is deteriorating very quickly. 00:03:01.000 --> 00:03:06.840 And our 00:03:06.840 --> 00:03:12.980 tenacity and energy is just not there anymore to 00:03:12.980 --> 00:03:17.000 apply the same kind of rigor and quality standards to it as we used to have. 00:03:17.000 --> 00:03:21.000 And then we're increasingly losing understanding. 00:03:21.000 --> 00:03:27.240 And I think one of the more devious parts about this is 00:03:27.240 --> 00:03:30.300 that in situations where my understanding is really, 00:03:30.300 --> 00:03:35.000 really necessary, there are too many cases where seemingly nobody has it anymore. 00:03:35.000 --> 00:03:40.000 And I heard this story from friends I used to work with. 00:03:40.000 --> 00:03:45.980 I had heard stories like this from people talking about it on Twitter that 00:03:45.980 --> 00:03:52.380 you can now end up in an incident where a bunch of people 00:03:52.380 --> 00:03:54.000 are huddling in a Slack room trying to figure out what's going on. 00:03:54.000 --> 00:04:01.000 And it's just all talking with the machines because nobody actually really wants to understand anymore, let alone understand the code that was committed. 00:04:01.000 --> 00:04:07.000 And that's not a situation that I think we should be encouraging. 00:04:07.000 --> 00:04:13.280 But I do want to let you know, 00:04:13.280 --> 00:04:15.000 a lot of us are feeling like this right now. 00:04:15.000 --> 00:04:20.900 And I think we owe it to us and our industry 00:04:20.900 --> 00:04:24.920 to find solutions to this problem and not just 00:04:24.920 --> 00:04:31.000 let it happen and not push back against this. 00:04:31.000 --> 00:04:34.000 This is a very shared experience. 00:04:34.000 --> 00:04:38.000 And don't get me wrong. 00:04:38.000 --> 00:04:39.000 I love those tools. 00:04:39.000 --> 00:04:42.000 I think they're absolutely fascinating. 00:04:42.000 --> 00:04:49.000 And it is so many times just such a magical experience to work with them. 00:04:49.000 --> 00:04:53.000 But they are also deeply frustrating. 00:04:53.000 --> 00:05:00.000 They're little machines where sometimes they produce an amazing result. 00:05:00.000 --> 00:05:05.520 And then you're spending two weeks trying to massage one pull request into a shape that you like 00:05:05.520 --> 00:05:09.000 with no feeling of forward progress. 00:05:09.000 --> 00:05:14.000 And it seems to be random. 00:05:14.000 --> 00:05:20.260 I don't know if it's just if me and my little community around us are just not the best 00:05:20.260 --> 00:05:25.000 prompters because I've heard that some people express that we're all just prompting wrong. 00:05:25.000 --> 00:05:27.000 But I think it's just hard. 00:05:27.000 --> 00:05:33.460 And I just really felt like because 00:05:33.460 --> 00:05:38.500 we're all, maybe not all of us, but because I think a lot of us struggling with this, 00:05:38.500 --> 00:05:42.000 I think it's just good to let you know that you're not alone. 00:05:42.000 --> 00:05:48.220 And I think we probably need to do some changes here in how we approach certain 00:05:48.220 --> 00:05:52.000 problems and how we engineer to actually really get to the bottom of it. 00:05:52.000 --> 00:05:56.600 Because better models and more capable models, 00:05:56.600 --> 00:06:03.060 if anything, right now are making it a little bit worse by writing code that 00:06:03.060 --> 00:06:08.000 is a lot less pleasant to look at and a lot harder to understand. 00:06:08.000 --> 00:06:13.000 So that's it for today. Have a nice weekend.