1 00:00:00,280 --> 00:00:02,730 So here's a big research bet that all the labs are making. 2 00:00:02,960 --> 00:00:06,470 They think that if we train AIs to accomplish millions of verifiable 3 00:00:06,470 --> 00:00:11,060 tasks across thousands of diverse RL environments, then we will have 4 00:00:11,070 --> 00:00:16,079 basically built AGI, because this kind of training will have created a kind of 5 00:00:16,079 --> 00:00:20,289 problem-solving agent: the kind of thing that can make progress on open-ended 6 00:00:20,289 --> 00:00:24,509 tasks for weeks on end in the face of errors and mistakes and ambiguity. 7 00:00:25,109 --> 00:00:27,790 And the people who are optimistic about this vision will say that all 8 00:00:27,790 --> 00:00:31,660 these things we talk about as the fundamental deficits in the current 9 00:00:31,660 --> 00:00:35,279 training paradigm — for example, the data inefficiency of these models, 10 00:00:35,309 --> 00:00:39,540 or the fact that they lack continual learning — can just be steamrolled if we 11 00:00:39,550 --> 00:00:43,670 scale training more, in the same way that all the fundamental research problems 12 00:00:43,670 --> 00:00:48,120 in natural language processing collapsed when we threw enough compute into LLMs. 13 00:00:48,709 --> 00:00:52,259 So in the previous essay, I talked about how these models are one 14 00:00:52,259 --> 00:00:56,509 one-millionth as sample-efficient as humans, and the people who are in favor 15 00:00:56,509 --> 00:00:59,579 of the current training paradigm will say, "Look, that might be true, but 16 00:00:59,579 --> 00:01:01,290 this is only true during training." 17 00:01:01,869 --> 00:01:05,890 Training is this one-time cost that is amortized across billions of 18 00:01:05,890 --> 00:01:07,609 sessions that a model will experience. 19 00:01:08,169 --> 00:01:11,410 What really matters is how smart and general and sample-efficient 20 00:01:11,419 --> 00:01:15,460 the model is during a session, and this has clearly been improving as 21 00:01:15,460 --> 00:01:17,259 we've been doing more RL training. 22 00:01:17,610 --> 00:01:20,459 AI agents are able to solve more and more ambitious problems over 23 00:01:20,459 --> 00:01:21,740 longer and longer time spans. 24 00:01:21,830 --> 00:01:24,029 Anybody who has used these models for coding knows that. 25 00:01:24,279 --> 00:01:27,819 Similarly, people would say, look, continual learning — this capability 26 00:01:27,820 --> 00:01:31,439 I keep harping about, where the model's weights get updated based on 27 00:01:31,439 --> 00:01:34,969 what it's learning from deployment — may simply not be necessary. 28 00:01:35,309 --> 00:01:40,079 Because if in-context learning gets so good across longer and longer 29 00:01:40,089 --> 00:01:44,739 time horizons, then you don't need to distill everything the model is learning 30 00:01:44,750 --> 00:01:46,219 on the job back into the weights. 31 00:01:46,509 --> 00:01:49,079 People often say that their employees are not net productive 32 00:01:49,099 --> 00:01:51,970 until six months or more on the job. 33 00:01:52,229 --> 00:01:55,719 So clearly, online learning is necessary for competence. 34 00:01:56,009 --> 00:01:58,440 But what if you could just fit those six months into the context window? 35 00:01:58,779 --> 00:02:01,750 There have been tons of architectural innovations that dramatically increase 36 00:02:01,779 --> 00:02:06,779 the amount of information, or the amount of context, that a transformer can store. 37 00:02:06,979 --> 00:02:08,600 And why not think that, with a couple more years of progress, 38 00:02:08,600 --> 00:02:11,949 we might have what feels like infinitely large context windows? 39 00:02:12,429 --> 00:02:15,190 Okay, so before we discuss this research bet a bit further, I want to step 40 00:02:15,199 --> 00:02:19,679 back and ask a completely tangential question, which I find actually 41 00:02:19,940 --> 00:02:24,290 very interesting and confusing about the nature of current AI progress. 42 00:02:24,699 --> 00:02:30,089 Why has progress on computer use been so much slower than other domains? 43 00:02:30,479 --> 00:02:33,949 Computer use is so clearly verifiable. 44 00:02:34,089 --> 00:02:38,299 You could ask a question like: did the desired Etsy item I ordered get delivered? 45 00:02:38,509 --> 00:02:41,809 Is the venue for an event I'm trying to organize booked? 46 00:02:41,989 --> 00:02:43,219 Have my taxes been submitted? 47 00:02:43,589 --> 00:02:46,120 So isn't it weird that computer use has been making so much slower 48 00:02:46,120 --> 00:02:49,440 progress than coding and math and these other verifiable domains? 49 00:02:49,899 --> 00:02:52,740 I'm sure there are many reasons for this, and one of them, of course, 50 00:02:52,759 --> 00:02:56,790 is the fact that the models are exposed to far less high-quality 51 00:02:56,790 --> 00:02:58,429 multimodal data during pretraining. 52 00:02:59,170 --> 00:03:03,230 But one reason that I think is actually quite underrated by people, and which I 53 00:03:03,240 --> 00:03:07,039 think reveals the canyon walls against which this river of AI progress will 54 00:03:07,059 --> 00:03:12,340 only slowly chip away, is that it is not enough for a domain to be verifiable. 55 00:03:12,670 --> 00:03:16,159 It also has to be very grindable, in the sense that you have to be able 56 00:03:16,160 --> 00:03:20,879 to run lots of parallel rollouts against a deterministic and replayable 57 00:03:20,900 --> 00:03:24,189 simulator, and you have to run those rollouts from the same starting point. 58 00:03:24,490 --> 00:03:27,640 If you're trying to make a model better at coding, you can define some 59 00:03:27,640 --> 00:03:30,874 container that has a software repo with some missing feature that you 60 00:03:30,874 --> 00:03:32,494 have tasked the AIs with creating. 61 00:03:32,494 --> 00:03:36,359 And then you have a thousand parallel agents go at the problem, each of which 62 00:03:36,360 --> 00:03:38,190 has an identical copy of the container. 63 00:03:38,430 --> 00:03:41,070 But this doesn't work with computer use, at least not trivially. 64 00:03:41,329 --> 00:03:43,909 You can't just have a thousand agents go try the same checkout flow 65 00:03:43,910 --> 00:03:48,529 on Amazon to get better at using websites, because Andy Jassy will 66 00:03:48,529 --> 00:03:50,280 find your bots and shut your ass down. 67 00:03:51,049 --> 00:03:54,570 You can solve this by making clones of Slack and Gmail and all the other 68 00:03:54,570 --> 00:03:55,979 common applications and websites. 69 00:03:56,669 --> 00:03:59,909 But at least currently, this is a very labor-intensive and unscalable 70 00:03:59,910 --> 00:04:01,140 way to build environments. 71 00:04:01,809 --> 00:04:04,689 Of course, once AIs get good enough at coding themselves to build these 72 00:04:04,689 --> 00:04:08,060 clones with extremely high fidelity, then I'm sure computer use will make 73 00:04:08,369 --> 00:04:09,789 quicker progress than it is right now. 74 00:04:10,059 --> 00:04:13,489 And you're also killing two birds with one stone with this kind of procedure, 75 00:04:13,489 --> 00:04:18,150 because getting AIs to rebuild whole applications from scratch is also 76 00:04:18,150 --> 00:04:19,709 a great RL objective for coding. 77 00:04:20,159 --> 00:04:23,679 So while computer use itself may soon be solved, its current lethargy 78 00:04:23,699 --> 00:04:27,809 is telling us the following: that unless you can build a very replayable 79 00:04:27,820 --> 00:04:31,209 training target for a domain, the models will struggle to make much progress. 80 00:04:32,009 --> 00:04:34,999 And the reason this is true, of course, is that the models are incredibly 81 00:04:35,009 --> 00:04:37,270 sample-inefficient during training. 82 00:04:37,270 --> 00:04:39,359 This is the point I was making in my last video essay. 83 00:04:39,809 --> 00:04:42,809 So for computer use, we might be able to make up for the sample-efficiency 84 00:04:42,809 --> 00:04:46,199 deficit by building these farmable deterministic simulators. 85 00:04:46,680 --> 00:04:49,510 But for so many other different kinds of skills that we need AIs 86 00:04:49,649 --> 00:04:51,539 to have, we simply can't do this. 87 00:04:52,169 --> 00:04:55,590 How do we train an AI to get really good at building a business from scratch? 88 00:04:55,820 --> 00:04:59,559 How about winning court cases, or having a profitable day of trading in the markets, 89 00:04:59,829 --> 00:05:01,500 or helping a candidate win an election? 90 00:05:02,019 --> 00:05:05,639 The rollout here requires interacting with the real world, and you can't 91 00:05:05,649 --> 00:05:07,969 recreate it from just within a datacenter. 92 00:05:08,339 --> 00:05:11,969 The outer-loop verification here may take months or even years of 93 00:05:12,179 --> 00:05:16,540 real-world actions to elicit, and you can't re-observe it by perturbing the 94 00:05:16,869 --> 00:05:22,030 model's actions slightly in thousands of parallel rollouts to isolate exactly 95 00:05:22,030 --> 00:05:24,120 what the model did that actually worked. 96 00:05:24,709 --> 00:05:27,599 Now, dealing with such reset-free, non-stationary environments 97 00:05:27,610 --> 00:05:29,379 is a known open problem in RL. 98 00:05:29,449 --> 00:05:30,780 I'm not pointing out anything new. 99 00:05:31,089 --> 00:05:33,969 But I really do want to emphasize that because of the idiosyncratic 100 00:05:33,970 --> 00:05:38,769 and sparse nature of data in most domains in the world, you need sample 101 00:05:38,769 --> 00:05:40,639 efficiency in order to get proficient. 102 00:05:41,000 --> 00:05:45,599 If AIs are to develop all the skills that humans have, and even skills 103 00:05:45,599 --> 00:05:49,309 that humans don't have, then they need to be able to learn from information 104 00:05:49,319 --> 00:05:54,730 revealed in unstructured, unverifiable, and ambiguous ways from scarce 105 00:05:54,730 --> 00:05:56,350 amounts of real-world interaction. 106 00:05:56,730 --> 00:06:00,169 Because in many domains, the relevant training information simply 107 00:06:00,180 --> 00:06:01,980 doesn't exist in any other way. 108 00:06:02,510 --> 00:06:05,910 What is the RL environment to make an AI that is as good at politics as 109 00:06:05,910 --> 00:06:09,579 Lyndon Johnson, or as good at building a space-launch business as Elon Musk? 110 00:06:10,139 --> 00:06:12,460 The labs are betting that RLVR will generalize. 111 00:06:12,580 --> 00:06:15,029 That is, that if you train on enough containerized, reproducible 112 00:06:15,029 --> 00:06:19,699 environments, you will develop a very general agent that can make and 113 00:06:19,699 --> 00:06:24,260 execute plans and learn rapidly from new information, and even pick up new 114 00:06:24,260 --> 00:06:26,779 skills, all within a single session. 115 00:06:27,230 --> 00:06:32,580 If you drop this endlessly RLVR'd AI into Texas politics in 1948, it 116 00:06:32,580 --> 00:06:35,599 could give you better advice than LBJ about winning the Senate seat. 117 00:06:35,829 --> 00:06:38,199 And if you gave it a hundred million dollars in 2002 and let it cook, 118 00:06:38,209 --> 00:06:39,400 it would build SpaceX for you. 119 00:06:39,719 --> 00:06:43,369 Now, whether RLVR can generalize this well is an empirical question. 120 00:06:43,709 --> 00:06:48,020 If the labs went from spending billions of dollars on RL environments to a trillion 121 00:06:48,020 --> 00:06:52,480 dollars, would you get the kind of thing that is a fully human-like general 122 00:06:52,480 --> 00:06:54,059 intelligence within the context window? 123 00:06:54,649 --> 00:06:57,549 Dario gave a telling quote during our podcast together, which I think 124 00:06:57,549 --> 00:07:02,280 hints that RLVR generalization is not infinitely strong. 125 00:07:02,570 --> 00:07:05,019 When he was explaining why model performance tends to degrade at long 126 00:07:05,019 --> 00:07:07,050 context, he said: "There's two things. 127 00:07:07,050 --> 00:07:09,969 There's the context length you train at, and there's a context 128 00:07:09,970 --> 00:07:11,130 length that you serve at. 129 00:07:11,509 --> 00:07:15,149 If you train at a small context length and then try to serve at a long context 130 00:07:15,150 --> 00:07:17,069 length, maybe you get these degradations." 131 00:07:17,470 --> 00:07:19,730 Now, maybe I'm reading too much into this, but it seems like he's 132 00:07:19,730 --> 00:07:23,500 saying that short-horizon RL training doesn't necessarily generalize 133 00:07:23,519 --> 00:07:25,329 to long-horizon RL performance. 134 00:07:25,669 --> 00:07:29,599 And if you can't generalize from short horizon to long horizon, then how are 135 00:07:29,599 --> 00:07:33,369 agents supposed to generalize from getting trained at a bunch of white-collar tasks 136 00:07:33,369 --> 00:07:38,309 to, say, having the ability to be dropped in the real world and build a business 137 00:07:38,309 --> 00:07:40,059 from scratch as well as Sam Walton? 138 00:07:40,279 --> 00:07:42,989 And even if, after enough in-context experience, the AIs could become 139 00:07:42,989 --> 00:07:46,519 like Henry Ford or Albert Einstein, all that would be ephemeral and 140 00:07:46,519 --> 00:07:49,979 wasted if you couldn't get those learnings back into the weights. 141 00:07:50,339 --> 00:07:54,449 Around 30 to 50 percent of a lab's compute goes to inference, and that compute is 142 00:07:54,449 --> 00:07:58,559 currently not playing any productive role in helping improve the model. 143 00:07:58,559 --> 00:07:59,929 This seems like a huge waste. 144 00:08:00,129 --> 00:08:03,499 And it's even worse than it sounds, because it is only in deployment 145 00:08:03,639 --> 00:08:06,219 that the most valuable bits of information which your model could 146 00:08:06,220 --> 00:08:07,879 learn from are actually revealed. 147 00:08:08,689 --> 00:08:11,410 What's actually happening in the organizations where I'm being used? 148 00:08:11,410 --> 00:08:12,939 What are they using me for? 149 00:08:13,129 --> 00:08:15,689 And what kinds of mistakes do I tend to make in the real world? 150 00:08:16,089 --> 00:08:19,279 We've got some genius grad student who's never been allowed to take a real 151 00:08:19,279 --> 00:08:22,989 internship, and we keep giving it more and more classroom case studies in the 152 00:08:22,989 --> 00:08:24,849 form of RL training on environments. 153 00:08:25,029 --> 00:08:29,169 It's so bizarre that we have AIs that are broadly deployed through the 154 00:08:29,169 --> 00:08:33,669 economy already, and are participating in so many different kinds of tasks, 155 00:08:33,819 --> 00:08:39,169 and are privy to so much domain- and organization-specific tacit knowledge, 156 00:08:39,399 --> 00:08:40,940 and they're not able to make use of it. 157 00:08:41,449 --> 00:08:45,079 But this kind of continual learning requires going back to the weights. 158 00:08:45,500 --> 00:08:48,840 AIs can't just keep building up a bigger and bigger KV cache as they 159 00:08:48,840 --> 00:08:50,140 learn from more and more users. 160 00:08:50,459 --> 00:08:53,399 That's just not scalable, and that's also not how humans do it. 161 00:08:53,589 --> 00:08:56,650 There's no clean separation in our brain between parameters and activations, 162 00:08:56,910 --> 00:09:01,089 and it's not like some part of your skull keeps expanding as you learn 163 00:09:01,090 --> 00:09:02,750 more things throughout your lifetime. 164 00:09:03,179 --> 00:09:07,180 When we learn stuff, there's clearly some kind of compression, and this 165 00:09:07,219 --> 00:09:09,589 aids our generalization and grokking. 166 00:09:09,990 --> 00:09:13,950 There are, in fact, some humans who have this autistic-savant-type ability 167 00:09:13,950 --> 00:09:19,099 to recall random tables of numbers or nonsense syllables years later 168 00:09:19,309 --> 00:09:23,189 — basically the kind of fidelity of information that models have in context. 169 00:09:23,570 --> 00:09:27,680 And such sheer volume cripples these humans' ability to understand 170 00:09:27,680 --> 00:09:29,000 abstractions and metaphors. 171 00:09:29,439 --> 00:09:32,889 Human continual learning is less about having all your observations at the tip 172 00:09:32,889 --> 00:09:36,699 of your tongue and more about chiseling the right intuitions and big-picture 173 00:09:36,700 --> 00:09:38,450 knowledge back into the weights. 174 00:09:38,910 --> 00:09:41,549 But the moment you move into the weights, you have to give up on 175 00:09:41,559 --> 00:09:43,359 in-context learning's sample efficiency. 176 00:09:43,629 --> 00:09:46,909 Because gradient updates are super sample-inefficient, all of the 177 00:09:46,910 --> 00:09:50,669 successfully shipped online-learning models have had to learn the exact 178 00:09:50,669 --> 00:09:52,499 same thing across millions of users. 179 00:09:52,969 --> 00:09:57,210 For example, the Cursor Tab model online-learns by predicting the 180 00:09:57,210 --> 00:10:00,589 same exact objective for over 400 million requests a day. 181 00:10:00,889 --> 00:10:04,089 The objective here is which edits actually got accepted by the user. 182 00:10:04,639 --> 00:10:09,429 At least so far, we haven't seen models online-learn different kinds of things for 183 00:10:09,430 --> 00:10:13,370 different users, because while a single session may generate more than enough 184 00:10:13,370 --> 00:10:18,249 data for a human to learn from, it's not enough to train a more capable AI. 185 00:10:18,619 --> 00:10:22,370 Current online learning can work for a very limited number of use cases. 186 00:10:22,729 --> 00:10:26,139 But the whole point of continual learning is that the world is very 187 00:10:26,139 --> 00:10:31,849 complicated, and each job and company and problem is different, and you need 188 00:10:31,849 --> 00:10:36,189 your intelligence to be able to learn the specific information related to a 189 00:10:36,189 --> 00:10:40,809 particular deployment, which simply can't be stuffed into some shared training run. 190 00:10:41,419 --> 00:10:43,839 These are all the things we're talking about when we talk about on-the-job 191 00:10:43,839 --> 00:10:47,350 learning: things like how everything in your organization works and fits 192 00:10:47,350 --> 00:10:50,849 together, how to cooperate with all the infrastructure and the other people 193 00:10:50,849 --> 00:10:55,489 around you to make progress on some larger project, what the common failure modes 194 00:10:55,489 --> 00:10:57,529 are, and many other things like this. 195 00:10:58,319 --> 00:11:01,989 As the podcast has grown, I've had to deal with more and more operational overhead. 196 00:11:02,619 --> 00:11:03,429 Take paying bills. 197 00:11:03,679 --> 00:11:06,470 In the past, contractors would just email me their invoices. 198 00:11:06,529 --> 00:11:09,629 Every few weeks, I'd dig through my inbox, create a folder with all the 199 00:11:09,629 --> 00:11:11,560 bills, and manually pay each one. 200 00:11:11,849 --> 00:11:14,159 At this point, though, I just give everybody an email address 201 00:11:14,179 --> 00:11:16,979 that goes straight to Mercury, which is my banking platform. 202 00:11:17,110 --> 00:11:20,470 Whenever anybody sends an invoice to that address, Mercury automatically 203 00:11:20,470 --> 00:11:24,479 downloads it, scans it, and extracts all the relevant information — things 204 00:11:24,479 --> 00:11:27,710 like the contractor name, address, payment amount, invoice number, 205 00:11:27,739 --> 00:11:30,619 and due date — and then uses all of this to create a draft payment. 206 00:11:30,970 --> 00:11:33,739 Mercury then stores a list of these drafts for me to review. 207 00:11:33,979 --> 00:11:35,849 I just go through the list and double-check that 208 00:11:35,860 --> 00:11:36,750 they've been built correctly. 209 00:11:36,929 --> 00:11:39,570 I don't have to track anything or enter any information myself. 210 00:11:39,789 --> 00:11:42,980 Mercury does all the fundamental things for your business extremely well, 211 00:11:43,190 --> 00:11:44,729 and it puts them all in one place. 212 00:11:44,929 --> 00:11:47,760 If you want to learn more, go to mercury.com. 213 00:11:48,289 --> 00:11:51,919 Mercury is a fintech company, not an FDIC-insured bank. 214 00:11:52,300 --> 00:11:54,850 Banking services provided through Choice Financial Group 215 00:11:54,870 --> 00:11:56,790 and Column N.A., Members FDIC. 216 00:11:57,860 --> 00:12:01,149 In this way, sample efficiency and continual learning are actually 217 00:12:01,190 --> 00:12:02,430 deeply connected problems. 218 00:12:02,929 --> 00:12:06,780 Relatively little data is available to the model on the job. 219 00:12:07,160 --> 00:12:11,030 Now, to learn from this data requires sample efficiency, and models can do that 220 00:12:11,090 --> 00:12:16,099 in context, but using the fast weights that are built on the fly by attention, 221 00:12:16,359 --> 00:12:20,429 which allow for this sample efficiency, scales very poorly in terms of memory. 222 00:12:20,779 --> 00:12:23,809 So we need architectural innovations that allow for some kind of 223 00:12:23,830 --> 00:12:25,130 intermediate representation. 224 00:12:25,509 --> 00:12:28,389 I talked before about how we already have many different working ideas 225 00:12:28,399 --> 00:12:31,780 for this kind of thing, from sparse attention to KV cache compaction. 226 00:12:32,290 --> 00:12:35,219 And every week, somebody releases a new paper suggesting some kind of 227 00:12:35,730 --> 00:12:37,469 other architectural optimization. 228 00:12:37,959 --> 00:12:40,280 It doesn't seem to me that architecture is fundamentally what 229 00:12:40,280 --> 00:12:41,949 is bottlenecking continual learning. 230 00:12:42,500 --> 00:12:44,359 So perhaps the bottleneck is the loss function. 231 00:12:44,770 --> 00:12:48,759 How do we update the weights, AKA how do we improve the model itself, 232 00:12:49,039 --> 00:12:52,349 based on information that was learned from one particular session? 233 00:12:53,020 --> 00:12:56,049 Even here, naively, it seems like there are many ideas that ought to work. 234 00:12:56,250 --> 00:12:59,559 A lot of people are talking about this technique called on-policy 235 00:12:59,569 --> 00:13:00,839 self-distillation recently. 236 00:13:01,169 --> 00:13:03,970 If you want to learn more about it, I recorded a little impromptu 237 00:13:03,970 --> 00:13:07,909 blackboard lecture on my iPhone with Sasha Rush a couple weeks ago, and 238 00:13:08,009 --> 00:13:09,520 it's in the link in the description. 239 00:13:09,930 --> 00:13:14,169 But to summarize the explanation, the idea is that we encourage the base model 240 00:13:14,169 --> 00:13:18,560 to make the same predictions when trying to solve some real-world problem as the 241 00:13:18,560 --> 00:13:22,310 model with all the context accumulated after a long session would have made. 242 00:13:22,939 --> 00:13:26,369 The whole point of this procedure is to distill what the model learned in a 243 00:13:26,369 --> 00:13:28,679 session back into the weights themselves. 244 00:13:29,110 --> 00:13:31,479 This is better than RLVR for two reasons. 245 00:13:31,779 --> 00:13:35,859 One, OPSD doesn't require us to have some outer-loop verifiable reward. 246 00:13:36,389 --> 00:13:39,969 We just need a model that can learn the right things within the context window. 247 00:13:40,369 --> 00:13:44,199 And as long as we have that, we can train the base model to match our veteran 248 00:13:44,199 --> 00:13:48,139 teacher model, which has built up all this experience during the session. 249 00:13:48,729 --> 00:13:53,139 And two, OPSD provides a much denser supervision signal than naive RL. 250 00:13:54,049 --> 00:13:56,249 Instead of projecting a single reward through the whole 251 00:13:56,250 --> 00:14:00,089 trajectory, you can train on the per-token probability discrepancy 252 00:14:00,309 --> 00:14:01,899 between the teacher and student. 253 00:14:02,270 --> 00:14:06,069 For continual learning, OPSD is also superior to supervised fine-tuning. 254 00:14:06,639 --> 00:14:09,610 The most naive version of SFT for this application that you can 255 00:14:09,610 --> 00:14:12,519 imagine is just to train the base model to predict all the tokens that 256 00:14:12,520 --> 00:14:13,719 are observed during the session. 257 00:14:14,129 --> 00:14:16,489 But this makes no sense if you think about it as a learning target. 258 00:14:16,830 --> 00:14:19,509 The way you get better at your job is not by recalling the transcript 259 00:14:19,509 --> 00:14:22,869 of every single thing that happened every day with perfect fidelity. 260 00:14:23,089 --> 00:14:26,539 Rather, it's by consolidating the handful of insights and pieces of 261 00:14:26,619 --> 00:14:30,199 knowledge that are actually relevant to you getting better at your job. 262 00:14:30,209 --> 00:14:32,739 RL training doesn't suffer from this failure mode. 263 00:14:32,750 --> 00:14:36,889 RL is great at concentrating the update to only what is relevant 264 00:14:36,899 --> 00:14:38,239 to getting the outcome right. 265 00:14:38,550 --> 00:14:38,860 That's why 266 00:14:41,460 --> 00:14:43,220 the updates from RL are incredibly sparse. 267 00:14:43,220 --> 00:14:46,139 And this is a very important property for continual learning, because as 268 00:14:46,139 --> 00:14:49,409 you're learning on the job, you don't want to overwrite and forget all the 269 00:14:49,410 --> 00:14:51,490 other things that the base model knows. 270 00:14:51,880 --> 00:14:55,160 I wrote a post a few months earlier arguing that RL learns 271 00:14:55,180 --> 00:14:58,060 much less information per sample than supervised learning. 272 00:14:58,559 --> 00:15:00,640 But this may be a good thing rather than a bad thing. 273 00:15:00,910 --> 00:15:04,769 You only change the model as much as is absolutely necessary to 274 00:15:04,770 --> 00:15:06,719 achieve the outcome, and no more. 275 00:15:07,320 --> 00:15:11,339 OPSD preserves this property of RL, where instead of slingshotting towards 276 00:15:11,340 --> 00:15:15,059 the teacher distribution as supervised learning would have you do, you only 277 00:15:15,059 --> 00:15:19,719 extract the knowledge that is necessary to achieve the same results as the 278 00:15:19,719 --> 00:15:22,569 teacher on actual real-world tasks. 279 00:15:22,919 --> 00:15:25,880 OPSD is one way to attack the sample-efficiency problem. 280 00:15:26,250 --> 00:15:29,389 You take this scarce real-world experience, and you squeeze all the 281 00:15:29,389 --> 00:15:31,319 signal into a tiny, well-targeted update. 282 00:15:31,860 --> 00:15:34,540 But there's also another much more speculative idea. 283 00:15:34,789 --> 00:15:35,629 Let's call it dreaming. 284 00:15:36,239 --> 00:15:40,389 If the AI can build a good simulation of reality against which to rehearse 285 00:15:40,389 --> 00:15:44,530 new skills, or try alternative strategies and reinforce what actually 286 00:15:44,530 --> 00:15:48,959 works, then AIs could experience orders of magnitude more simulated 287 00:15:48,959 --> 00:15:51,259 samples in the same wall-clock time. 288 00:15:51,680 --> 00:15:52,809 Let's go back into history a bit. 289 00:15:53,149 --> 00:15:56,689 A couple years after DeepMind released AlphaZero, a group of researchers 290 00:15:56,689 --> 00:15:59,570 trained a model called EfficientZero, and the whole point of this model 291 00:15:59,599 --> 00:16:01,480 is to be very efficient with data. 292 00:16:01,930 --> 00:16:06,610 So if this model and a human both got two hours to play against a simulator 293 00:16:06,610 --> 00:16:10,030 of an Atari game that they hadn't seen before, this model would actually 294 00:16:10,030 --> 00:16:11,190 probably beat the novice human. 295 00:16:11,860 --> 00:16:14,449 Does this mean that the model was more sample-efficient than the humans? 296 00:16:14,629 --> 00:16:17,820 Well, that was the goal of the training, but it depends on how 297 00:16:17,820 --> 00:16:19,169 you measure sample efficiency. 298 00:16:19,459 --> 00:16:23,669 Because for each step in the real game, EfficientZero is playing dozens 299 00:16:23,669 --> 00:16:25,529 of simulated games in its head. 300 00:16:25,979 --> 00:16:29,079 In a similar way, future LLMs might be able to consume far less 301 00:16:29,089 --> 00:16:32,610 real-world data while practicing endlessly against environments 302 00:16:32,610 --> 00:16:33,789 that they build for themselves. 303 00:16:34,529 --> 00:16:37,209 The big difference, of course, is that it will be much harder to build 304 00:16:37,209 --> 00:16:41,279 a simulation of the whole world than it is to emulate the game of Go. 305 00:16:41,599 --> 00:16:43,609 That's why I said this is a much more speculative idea. 306 00:16:44,149 --> 00:16:48,079 If it works, it would become a fourth axis of scaling alongside pretraining, 307 00:16:48,589 --> 00:16:50,209 RL, and inference-time compute. 308 00:16:50,559 --> 00:16:52,939 You could call it test-time training or dreaming. 309 00:16:53,319 --> 00:16:56,569 The model spends compute writing up RL environments and then training 310 00:16:56,569 --> 00:17:00,799 against them, and it's rehearsing all the skills that will actually be used 311 00:17:00,809 --> 00:17:03,179 in production for a specific user. 312 00:17:03,569 --> 00:17:09,150 So instead of hitting /compact in Codex or Cursor or Claude, which kindles a small 313 00:17:09,150 --> 00:17:13,090 amount of compute to write up a summary, and which gives you the simulacrum of 314 00:17:13,139 --> 00:17:14,719 continual learning, you hit /dream. 315 00:17:14,720 --> 00:17:20,179 And this incinerates huge amounts of compute to build and train against a 316 00:17:20,179 --> 00:17:23,579 video-game version of what the model is witnessing in the real world. 317 00:17:24,000 --> 00:17:26,609 So what might continual learning look like by 2027 or 2028? 318 00:17:26,719 --> 00:17:28,799 And how do we get there? 319 00:17:29,369 --> 00:17:30,269 Here's one scenario. 320 00:17:30,690 --> 00:17:34,929 All of this RLVR training is producing an agent that can get its bearings when 321 00:17:34,929 --> 00:17:38,200 it's thrown at an unfamiliar problem, and it can try different strategies, and 322 00:17:38,200 --> 00:17:39,719 it can iterate when it hits a roadblock. 323 00:17:40,110 --> 00:17:44,210 This is the crucial thing that RLVR has given you: an AI that 324 00:17:44,210 --> 00:17:47,960 is at least competent enough to start getting some real-world 325 00:17:47,960 --> 00:17:49,820 experience, if it could learn from it. 326 00:17:50,170 --> 00:17:53,139 And once you have that, you send it out into the world to do real 327 00:17:53,140 --> 00:17:56,210 work, even on projects that are off the training distribution. 328 00:17:56,950 --> 00:17:59,950 Now let's say at this point, the effective context lengths have expanded 329 00:17:59,960 --> 00:18:04,960 such that AIs can jam and co-work with you for a full week of wall-clock time. 330 00:18:06,289 --> 00:18:08,650 At the end of a week, you give it a thumbs up or a thumbs 331 00:18:08,650 --> 00:18:09,969 down, you give it a work review. 332 00:18:10,459 --> 00:18:13,429 And if you give it a thumbs up, the base model distills everything that 333 00:18:13,439 --> 00:18:17,879 the AI learned during the session, and it may use OPSD, it may use dreaming, 334 00:18:17,969 --> 00:18:21,049 it may use some other technique that we aren't even aware of, or it'll use 335 00:18:21,049 --> 00:18:22,570 a combination of all of the above. 336 00:18:23,259 --> 00:18:28,960 And AI can get better at domains that are adjacent to what it was explicitly 337 00:18:28,960 --> 00:18:30,629 trained for beforehand with RLVR. 338 00:18:32,169 --> 00:18:35,750 And in the next round it gets better at the thing adjacent to what it 339 00:18:35,750 --> 00:18:37,080 was previously online learned. 340 00:18:37,520 --> 00:18:41,119 In this way, the gamut of AI skills and knowledge and capabilities can 341 00:18:41,119 --> 00:18:46,210 expand far beyond the verifiable domains that the model was originally 342 00:18:46,210 --> 00:18:48,099 trained against before it was deployed. 343 00:18:48,379 --> 00:18:52,259 Just as pretraining created a base intelligence that was smart enough to 344 00:18:52,309 --> 00:18:58,280 become a competent agent with enough RLVR on top, so RLVR has created an agent 345 00:18:58,549 --> 00:19:02,460 that is competent enough to actually be broadly deployed in the world, and 346 00:19:02,460 --> 00:19:06,579 from this broad deployment to learn on the job once the training recipe for 347 00:19:06,580 --> 00:19:08,309 continual learning actually arrives. 348 00:19:08,830 --> 00:19:12,120 By this point, the main way that AIs get better is not from the 349 00:19:12,120 --> 00:19:14,399 training they have received before they are released to the public. 350 00:19:14,930 --> 00:19:18,850 Rather, it's from all this experience that they'll be accumulating from being broadly 351 00:19:18,850 --> 00:19:22,409 deployed in the economy and engaging in so many different kinds of tasks. 352 00:19:22,669 --> 00:19:26,149 Every time that you interact with an AI, it'll be smarter, not only because 353 00:19:26,149 --> 00:19:28,829 it's been learning from your previous sessions, but also because it's been 354 00:19:28,829 --> 00:19:32,699 learning from all its interactions with all the other users in the world. 355 00:19:33,139 --> 00:19:36,619 And that's very scary and exciting and different from the 356 00:19:36,619 --> 00:19:38,359 way that AI improves right now. 357 00:19:38,869 --> 00:19:42,109 This was a narration of a blog post that I also released on 358 00:19:42,110 --> 00:19:44,579 my website at dwarkesh.com. 359 00:19:44,969 --> 00:19:47,319 Go there if you want to read all the footnotes, or if you want to 360 00:19:47,320 --> 00:19:50,049 sign up so you can find out when I release the next blog post. 361 00:19:50,449 --> 00:19:52,470 Otherwise, I'll see you on the next episode.