WEBVTT 00:00:00.000 --> 00:00:01.700 But you can't be loud. 00:00:01.820 --> 00:00:03.160 Be quiet with the sword. 00:00:03.860 --> 00:00:04.920 Be quiet with the sword. 00:00:07.840 --> 00:00:10.520 Code is less important than it used to be, obviously. 00:00:11.480 --> 00:00:17.680 But the question is, how good can your agent deal with a shit ton of code, 00:00:17.680 --> 00:00:20.540 given its limitations with respect to context window? 00:00:21.300 --> 00:00:25.920 So the agent is ultimately hurting itself by generating too much code. 00:00:26.400 --> 00:00:28.100 Is the code as important as it used to be? 00:00:28.100 --> 00:00:29.100 No, definitely not. 00:00:30.400 --> 00:00:35.860 But I would say that it is still important in Astra is cool and all, 00:00:35.860 --> 00:00:38.040 just like Fable, but there's still slot machines. 00:00:38.360 --> 00:00:43.000 And I would say there is a lot of code where you don't need to know what it does and how it works. 00:00:43.340 --> 00:00:48.700 But there's also a lot of very important infrastructure code where you still want to know what it does. 00:00:50.340 --> 00:00:52.980 I think ultimately it comes down to control. 00:00:53.580 --> 00:00:56.700 I personally like controlling interfaces and types. 00:00:56.700 --> 00:01:02.300 And as long as I have control of that, the LLM is kind of limited in how much damage it can do. 00:01:02.560 --> 00:01:09.140 Yes, the models can shit out immense amounts of code now, but by doing so, they're hurting themselves because ultimately they still have a limited context window. 00:01:09.140 --> 00:01:13.020 And if it doesn't fit, you get more slop. 00:01:13.020 --> 00:01:18.900 What I found interesting with the Astra release is that I, 00:01:18.900 --> 00:01:25.060 for the first time, felt like less and less desire to actually look at the code that it produces because it's so hard to see. 00:01:25.060 --> 00:01:27.580 And not, I mean, it's hard to look at. 00:01:27.720 --> 00:01:32.020 It is incredibly compressed in situations in where it shouldn't be, 00:01:32.020 --> 00:01:36.480 even when it actually uses code generation for writing code. 00:01:36.680 --> 00:01:39.860 But it's particularly unreadable when it uses code to call tools. 00:01:40.220 --> 00:01:47.040 So like actually following along what it's doing is also getting harder and harder just by how it writes code. 00:01:47.040 --> 00:01:52.780 And I think like to Mario's point, like it actually does get, like during the training process, 00:01:52.780 --> 00:01:54.640 it probably gets rewarded for token efficiency. 00:01:55.360 --> 00:01:59.940 But as a result of this token efficiency, it is hard to read. 00:02:00.420 --> 00:02:06.880 And there's a tension, I think, sort of forming where I think even if you get the 00:02:06.880 --> 00:02:11.000 interfaces right and everything else, like if I look at an Astra produced code, 00:02:11.000 --> 00:02:13.000 I feel an urge to do code formatting. 00:02:13.000 --> 00:02:16.360 Which I didn't feel for a very, very long time. 00:02:16.500 --> 00:02:22.160 I basically haven't used code formats at all because I felt like the LLM is going to follow along the style of the code that's already there. 00:02:22.280 --> 00:02:22.900 And now it doesn't. 00:02:23.420 --> 00:02:24.640 Yeah, and it is kind of odd. 00:02:24.860 --> 00:02:25.780 It is really odd. 00:02:25.860 --> 00:02:26.840 And I don't know where it's going. 00:02:27.280 --> 00:02:32.120 But I'm a little bit feeling like this might become more of a thing now. 00:02:32.260 --> 00:02:33.520 And I don't know how I feel about this. 00:02:35.360 --> 00:02:38.020 You said something earlier, which I thought was interesting. 00:02:38.020 --> 00:02:41.800 As the models are trained to not only be coding agents, 00:02:41.800 --> 00:02:48.140 but also like personal open claw-like assistants, it's that's a reason that the training might 00:02:48.140 --> 00:02:54.280 contain traces or things like just create whatever the 00:02:54.280 --> 00:02:58.380 fucking code you want to create, as long as it does the thing the user asked for. 00:02:58.380 --> 00:03:01.700 And the user is a non-technical user and the user doesn't write production systems, 00:03:01.700 --> 00:03:08.080 but just tiny little things like build me a daily news brief website or something. 00:03:08.300 --> 00:03:09.340 Or Minecraft clone or anything. 00:03:09.700 --> 00:03:15.640 So if the models are now being trained to produce slop code just to make their users happy, 00:03:15.640 --> 00:03:20.760 which is totally fine, then we'll have a detrimental effect on using these models for production software. 00:03:20.760 --> 00:03:25.320 So what I found quite interesting was, 00:03:25.320 --> 00:03:31.120 I noticed as you sort of look into the code that it produces, 00:03:31.120 --> 00:03:34.780 like in tests, in shaders, in embedded JavaScript, 00:03:34.780 --> 00:03:37.940 you can clearly notice that it's way worse than in the regular code that it writes. 00:03:38.180 --> 00:03:41.460 So I wrote in a blog post, I think it's like one step removed from regular code. 00:03:41.560 --> 00:03:45.600 It sort of starts falling into this pattern of really highly compressed things. 00:03:45.600 --> 00:03:47.860 But then I was like, okay, maybe this just happens to me. 00:03:47.880 --> 00:03:53.520 So I asked a bunch of people on Twitter, like, hey, if you build like an Astro game over the weekend and it works for you, 00:03:53.520 --> 00:03:54.720 just show me the code. 00:03:54.880 --> 00:03:58.100 And there were two people who replied like, oh yeah, it wasn't so bad. 00:03:58.180 --> 00:04:00.540 And I just opened the code and it was really, really bad. 00:04:01.560 --> 00:04:05.440 And then it was, I think both of them replied, yeah, it's probably not nice. 00:04:06.000 --> 00:04:11.120 The thing is, like, we don't really expect it to actually write code this bad, I think. 00:04:11.120 --> 00:04:16.940 Like, I got fully trusted that the models are at least certain level of, 00:04:16.940 --> 00:04:18.660 like, passing bar. 00:04:18.900 --> 00:04:21.300 And it feels a little bit like it violated this now. 00:04:21.420 --> 00:04:21.820 I don't know. 00:04:22.020 --> 00:04:23.700 It definitely feels like a shift. 00:04:25.020 --> 00:04:30.940 Yeah, just yesterday I did some work on an old C, C++ code base in Spine. 00:04:30.940 --> 00:04:35.420 I switched back to 5.6 Sol 00:04:35.420 --> 00:04:40.220 because Astro just couldn't write any sensible kind of code. 00:04:40.420 --> 00:04:41.680 It's very weird. 00:04:41.900 --> 00:04:42.860 It's very, very weird. 00:04:43.000 --> 00:04:44.960 I blame personality fans. 00:04:46.400 --> 00:04:53.040 One thought that I had was that if you have the idea, 00:04:53.040 --> 00:04:57.620 at least in theory, that you don't have to at all consider the code because, 00:04:57.620 --> 00:05:00.500 like, the test coverage is going to guarantee you that the code works perfectly. 00:05:00.940 --> 00:05:05.440 Then, even if you have, like, 100% branch coverage, 00:05:05.440 --> 00:05:10.300 which I don't think is likely, do your reader cover all the possible inputs to that function? 00:05:10.300 --> 00:05:16.700 And one of the things that I noticed, like, with this is that because it 00:05:16.700 --> 00:05:20.240 compresses the code so much, I, 00:05:20.240 --> 00:05:25.820 when I just look at the diff that it does, I don't have any confidence that, 00:05:25.820 --> 00:05:30.800 like, I could even without asking the LLM tell you if the tests are sufficient. 00:05:30.800 --> 00:05:34.320 I'm really struggling with this whole thing because, 00:05:34.320 --> 00:05:38.020 like, if the code is too unreadable, then, and I'm not saying, 00:05:38.020 --> 00:05:43.860 like, it's completely unreadable, but, like, it becomes so that you're one step further away from it. 00:05:43.860 --> 00:05:49.320 And then you rely more on, like, okay, but there are tests and the tests pass and I can still navigate in my slop game. 00:05:49.460 --> 00:05:51.760 And so, surely the whole thing must be working. 00:05:52.560 --> 00:05:59.320 But it's, like, it becomes so much harder to understand if you feel repulsed looking at the code. 00:06:00.100 --> 00:06:04.700 So, at that point, the tests also just become performative, right? 00:06:04.700 --> 00:06:07.840 It just gives you a good feeling of there are tests. 00:06:08.280 --> 00:06:13.100 I don't fucking know what they do and if they actually cover what we need to cover in the tests, but they exist. 00:06:13.600 --> 00:06:14.660 And that's why I feel good. 00:06:16.020 --> 00:06:18.560 It's very emotion-based, all of it. 00:06:19.620 --> 00:06:26.720 But if they produce code that is unreadable, then you have even less incentive of checking whether the code they produced makes any kind of fucking sense. 00:06:26.960 --> 00:06:31.980 So, it feels like we are getting pushed to being removed from the code. 00:06:31.980 --> 00:06:32.120 Okay. 00:06:34.700 --> 00:06:37.940 What if you use something like Astra, which is clearly, 00:06:37.940 --> 00:06:41.320 like, trained to succeed, 00:06:41.320 --> 00:06:45.780 but you use it only for the tool calls. 00:06:45.900 --> 00:06:47.160 Like, you don't use it for the writing. 00:06:47.260 --> 00:06:50.440 So, like, whenever it tries to write some code, you're like, oh, you stop. 00:06:50.840 --> 00:06:55.600 Some agent, I don't know, something else you implement now. 00:06:57.080 --> 00:07:00.700 Because I really feel like there's a tension in the training process. 00:07:00.700 --> 00:07:05.620 They're like, either this is really good and we should like this because, 00:07:05.620 --> 00:07:10.120 I don't know, software engineering is solved and it's just a bug fix that we have to deal with now. 00:07:10.520 --> 00:07:11.780 And so, we don't look at the code anymore. 00:07:11.900 --> 00:07:12.920 So, it doesn't matter how it looks. 00:07:13.060 --> 00:07:15.040 And it's, like, really purely optimized for the LLM. 00:07:15.040 --> 00:07:20.820 Or, these models are actually only behaving this way because they're being trained to do other things, 00:07:20.820 --> 00:07:22.700 particularly really good tool calling using code. 00:07:23.460 --> 00:07:29.840 And so, there's a tension in the training process and you should actually use a different model that's being trained to be an 00:07:29.840 --> 00:07:31.120 expert software engineer. 00:07:31.120 --> 00:07:34.680 And I really don't know what the answer is here because, 00:07:34.680 --> 00:07:39.000 like, it's not like you go to OpenAI page and it's like, yeah, do that. 00:07:40.340 --> 00:07:44.440 No, but maybe that's like a model capacity issue now. 00:07:44.440 --> 00:07:50.680 Like, as they try to accommodate more and more use cases or vertical slices of 00:07:50.680 --> 00:07:57.340 specific industries, obviously, the model needs to know about how to operate within these environments. 00:07:57.340 --> 00:08:03.760 But that will ultimately mean that there are less parameters available to the things they used to be good 00:08:03.760 --> 00:08:06.000 for, like software engineering. 00:08:06.300 --> 00:08:07.780 So, maybe we are hitting that now. 00:08:08.160 --> 00:08:12.880 Maybe it's the, you're trying to accommodate as many industries as possible. 00:08:13.160 --> 00:08:19.300 So, your industry, specifically the software industry, is now suffering because there's not enough parameters left for your specific industry. 00:08:20.060 --> 00:08:20.340 I don't know. 00:08:20.340 --> 00:08:26.160 The token spent on engineering went 00:08:26.160 --> 00:08:27.700 up in a certain way. 00:08:27.740 --> 00:08:30.680 So, you could see, like, month by month what that was. 00:08:30.980 --> 00:08:33.760 But none of the top line numbers moved up at all. 00:08:35.220 --> 00:08:37.700 In a way, that was correlatable to this. 00:08:38.060 --> 00:08:39.680 But obviously, the number of commits went up. 00:08:39.820 --> 00:08:41.760 The number of, like, other things that you could measure went up. 00:08:42.460 --> 00:08:49.020 And so, it went from, like, an insignificant part of, like, the expenditure of the company to a really meaningful part of the expenditure of the company. 00:08:49.020 --> 00:08:54.340 And it might just be that we haven't figured out yet how to make that work. 00:08:55.140 --> 00:09:01.480 Or, like, what is being produced actually isn't that much 00:09:01.480 --> 00:09:03.440 more meaningful overall. 00:09:04.260 --> 00:09:10.540 And so, because now everybody, like, as an example, like, the baseline expectation that we 00:09:10.540 --> 00:09:13.080 have now for a piece of software is that it does more. 00:09:14.060 --> 00:09:17.620 So, you need now to compete with everybody else who's in your space as well. 00:09:17.620 --> 00:09:20.320 So, like, of course, your thing now should have a chatbot in it. 00:09:20.700 --> 00:09:26.240 Of course, there should be, I don't know, like, the UI should behave in a certain way that was previously not necessary. 00:09:26.920 --> 00:09:31.480 Like, more people now build mobile apps, I think, than they did before just because it became easier. 00:09:31.980 --> 00:09:37.400 So, maybe this is sort of, like, a really weird form of, like, we are required to build more. 00:09:37.540 --> 00:09:40.260 But, like, the amount of consumers is still the same. 00:09:40.560 --> 00:09:44.040 So, it just has become more expensive to the software engineering. 00:09:44.180 --> 00:09:45.120 I don't know if that's the answer. 00:09:45.120 --> 00:09:51.620 I mean, I guess it's hard to measure at the moment just because 00:09:51.620 --> 00:09:57.420 of the economic turmoil overall with oil being up and so on and so forth. 00:09:57.560 --> 00:10:01.020 So, just taking GDP is probably a bad, bad idea. 00:10:01.280 --> 00:10:07.960 But if you look at individual companies, I don't see them producing more meaningful features for the users if you're looking at software companies. 00:10:08.500 --> 00:10:09.580 I just don't see it. 00:10:09.580 --> 00:10:17.140 Like, can you name a single software or SaaS product that has meaningfully changed in the past 12 months? 00:10:17.620 --> 00:10:19.700 I mean, I think in the AI space, you see that a little bit. 00:10:20.400 --> 00:10:23.820 Yeah, but then the question is, who is using all of that stuff from the AI space? 00:10:24.380 --> 00:10:24.600 Yeah. 00:10:24.600 --> 00:10:29.240 I found, like, iOS 27 to be an interesting proof point here because, 00:10:29.240 --> 00:10:35.120 like, clearly there should be more people in Apple using AI tools. 00:10:35.360 --> 00:10:41.120 Like, I heard from people at Apple that there is, like, plenty amount of, like, AI supports of engineering going on. 00:10:41.120 --> 00:10:46.540 But, like, from iOS 26 to 27, I felt like it was the most 00:10:46.540 --> 00:10:49.420 boring iOS release. 00:10:50.760 --> 00:10:54.180 Like, if anything, it's sort of the push against this in a way. 00:10:55.080 --> 00:10:57.220 I mean, I don't think that's bad, actually. 00:10:57.440 --> 00:11:03.620 I don't want a Cambrian explosion of new features and the home of Simpsons car, right? 00:11:03.620 --> 00:11:10.040 But then I must wonder, does the additional resources that you have through agentic coding just ensure that 00:11:10.040 --> 00:11:11.700 whatever you push out has higher quality? 00:11:12.740 --> 00:11:14.160 I would like to think so. 00:11:14.380 --> 00:11:17.660 I personally would like to use agents in a way where I can say, 00:11:17.660 --> 00:11:22.620 okay, my velocity is kind of the same, but everything I release is actually of higher quality, 00:11:22.620 --> 00:11:25.340 even if the quantity stays kind of the same. 00:11:25.720 --> 00:11:29.660 That would be the goal for me when using agentic engineering tools. 00:11:29.660 --> 00:11:35.840 But I'm not sure if you're seeing that either, if you look at companies with their 999s going into the 00:11:35.840 --> 00:11:38.700 9100 territory. 00:11:40.120 --> 00:11:43.380 I think where you see it a little bit in terms of, 00:11:43.380 --> 00:11:49.840 like, quality, presumably, like, depending on how you look at this, is that software should become more secure because we are having 00:11:49.840 --> 00:11:51.540 to work on all of the security reports. 00:11:52.380 --> 00:11:52.460 Right? 00:11:52.560 --> 00:11:54.000 And I think, like... 00:11:54.000 --> 00:11:57.780 Well, that's not happening, I can tell you from first-hand experience. 00:11:57.780 --> 00:12:00.220 It's definitely not more secure. 00:12:01.660 --> 00:12:06.260 But it's also riskier to be on the internet now with software. 00:12:07.460 --> 00:12:11.720 One of our colleagues, like, he's a little bit more now worried about, 00:12:11.720 --> 00:12:17.680 like, smart contracts and crypto wallets just because they have become such attractive targets for 00:12:17.680 --> 00:12:21.060 automated security research. 00:12:21.060 --> 00:12:25.100 But, 00:12:25.100 --> 00:12:30.400 I mean, there are definitely certain areas where, like, on average, 00:12:30.400 --> 00:12:34.300 like, there should be more fixes going into, like, curl is a good example, 00:12:34.300 --> 00:12:35.240 or, like, other software. 00:12:35.320 --> 00:12:36.240 This is really, really old. 00:12:36.240 --> 00:12:40.780 So, they got a lot more valid security reports, so they fixed them. 00:12:40.980 --> 00:12:46.860 So, like, in general, curl is probably safer now than it ever was. 00:12:47.260 --> 00:12:49.840 And it doesn't slop a ton of new features into it. 00:12:50.480 --> 00:12:55.560 But also, the risk profile just went up simultaneously because of the same thing that... 00:12:55.560 --> 00:13:00.640 And that is the problem, because the security patches still take humans to verify and apply. 00:13:01.620 --> 00:13:06.020 But finding the security vulnerabilities is now, like, 100x easier. 00:13:06.320 --> 00:13:12.740 I literally just had my clanker kind of try to break the DRM of one of my old 00:13:12.740 --> 00:13:14.380 products, and it just did it. 00:13:14.840 --> 00:13:17.780 It just, like, without intervention from my end. 00:13:17.780 --> 00:13:20.860 So, yeah, interesting times.