Updated July 31, 2026
0:00 Welcome to Colaberry AI podcast brought to you by Colaberry AI Research Labs and Carl Foundation. So glad to be here for this one. Yeah. It's, it's gonna be a really fascinating deep dive today. Yeah. 0:11 I mean, normally, when you look at physical infrastructure, whether that's building a skyscraper or laying down fiber optic cables, the rules of the game are just incredibly visible. Right. You can physically see the steel. You can measure the concrete. Exactly. 0:24 But if you step into the world of frontier AI development right now, those physical rules are just being completely subverted. We're looking at a landscape where massive government roadblocks are being bypassed. And not just by legal loopholes either. No. Not at all. 0:39 They're doing it through some of the most extreme, almost underground software and network engineering that's ever been attempted. So today, we are exploring those extreme technical lengths. We're getting highly technical diving into specific methods, parameter counts, benchmark results. We've got a lot of ground to cover. We really do. 0:57 We're looking at Moonshot AI's massive hardware workarounds, OpenAI's breakthrough in benchmark debugging, and the whole industry's push for new safety mechanisms. I think the overarching theme in the source material today really comes down to friction. Yeah. Friction is a great word for it. Because you have this intense geopolitical friction crashing right into the physical limits of current computing architecture, and the byproduct of that collision is just this totally new breed of engineering. 1:26 It really is. Let's start with the physical constraints of training because the technical lengths these developers are going to are just staggering. Oh, they're wild. The sources focus heavily on Moonshot AI, which is already prepping its next model, Kiwi k four. Which is crazy timing when you think about it. 1:42 Right. Because k three, which is currently the largest open source model out there at 2,800,000,000,000 parameters, k three is barely two weeks old. But the immediate wall they hit for k four is silicon. Specifically, NVIDIA's top end Blackwell chips. Exactly. 1:57 Those chips are heavily restricted from Chinese firms under US export rules. Which turns the very existence of a 2,800,000,000,000 parameter model like k three into a well into a highly contested technical puzzle. It really does. And the origin story of k three depends entirely on who you ask. Yeah. 2:14 Right? Like, it paints two very different pictures of the global supply chain. Oh, completely. So The US perspective, and this was pushed publicly on x by officials like Michael Katios, is that Key three was trained at a data center in Thailand. And just a quick note for you listening, we are simply reporting Katios' claims, and the sources counterclaims here. 2:33 We are not taking any political sides at all. Right. We're just looking at the logistics. And from a regulatory standpoint, that Thailand framing offers a really clean explanation. How so? 2:43 Well, if they trained in Thailand, utilizing cloud infrastructure that sits outside of China's borders, Chinese developers accessing those NVIDIA chips might actually be permissible under current US export parameters. Okay. That makes sense. But the technical sources on the ground paint a very different picture. They point to domestic training. 3:01 Yeah. And there's a huge technical reason for that. Right. It's China's own strict cross border data transfer regulations. Mhmm. 3:08 If you are training a 2,800,000,000,000 parameter model, you aren't just sending a zip file of text over an email. No. You're moving petabytes of highly sensitive pre training data. Exactly. Moving all that to an offshore facility in Thailand is a logistical and legal nightmare under Beijing's laws. 3:27 So you effectively have The US saying, you know, you can't have the compute hardware here, and Beijing saying, well, you can't take the training data there. So the model basically gets trained in this intense regulatory gray zone, but the actual mechanics of how they achieve this domestically is the real marvel here. It's unbelievable. Because traditionally, to train a model of this magnitude, you require a massive unified computing cluster. We are talking tens of thousands of Blackwell chips all sitting in the exact same facility. 3:55 Connected by something like NVIDIA's NVLink. Exactly. NVLink provides incredibly high bandwidth, low latency communication between the GPUs. But nobody in China could just walk up to a single supplier and rent a giant coherent cluster of restricted black wells. So Moonshot had to improvise across boundaries. 4:14 They really did. The sources say they tapped at least two different Chinese cloud firms. They essentially stitched together these tiny eight chip Blackwell servers that were scattered across entirely different data centers. And to understand how difficult that is, you have to look at how distributed training actually works. Walk us through that. 4:34 So during a training run, chips process massive batches of data, they calculate gradients, and then they all have to synchronize and update the model's weights together. Right. In a standard setup, if one chip has to wait for data from another chip, the entire system stalls. It's known as the communication bottleneck. I mean, it sounds like trying to build a world class supercomputer by hot wiring together two different data centers using standard eight cylinder car engines instead of just dropping in one massive v 80 engine. 5:02 That's a perfect analogy. The latency overhead must just be a total nightmare. I mean, how do you even manage latency and synchronization in a multi provider distributed setup like that? Like, how do you mask a fifty millisecond latency spike between two different cloud providers during a gradient update? Honestly, you have to fundamentally tear down and rebuild the network topology. 5:24 Moonshot's engineers couldn't rely on standard interconnects. So what did they do? They had to design entirely custom routing protocols to manage severe packet loss and fluctuating bandwidth limits across this disjointed structure. They're likely using extreme pipeline parallelism. Okay. 5:40 And what does that mean exactly for the model? It means different layers of the model are split across these different data centers in a highly specific way that attempts to hide that network latency. Wow. Yeah. The geopolitical irony here is just thick. 5:54 Both US export rules and Beijing's data laws are inadvertently forcing these Chinese engineers to become absolute masters of decentralized resilient network design. And we're seeing it across the board too. Alibaba just dropped Quinn 3.8 max, which is a 2,400,000,000,000 parameter model utilizing similar disjointed NVIDIA hardware setups. Right. And there are even reports from December of DeepSeek training its v four model on Blackwell chips that were reportedly smuggled into the country. 6:25 Because doing this on domestic ships wouldn't just be harder at this massive scale, The failure rates and the lack of mature networking stacks on alternative silicon make a 2,800,000,000,000 parameter run border on mathematically impossible right now. Right. And that just covers pre training, which is really only the first hurdle. Once you lock in those wakes and the model is built, you have to actually serve it to millions of users. And inference hardware brings an entirely different set of technical and supply chain nightmares? 6:53 It really does. So for inference, the sources note Moonshot uses NVIDIA's h 20 chips. Now these are explicitly permitted by The US. Yeah. The h 20 is a heavily cut down piece of silicon designed specifically to comply with those export controls. 7:06 The crucial metric here is compute density and memory bandwidth. Because the h 20 is throttled, you need vastly more of them to execute the same inference tasks compared to an unrestricted chip. The sources show that to host k three, a provider needs a minimum configuration of 64 server chips just to run the model. That is just the absolute floor. To to get it to output a single word. 7:31 And when Moonshot released k three to the public, the demand was so intense, they had to completely halt new subscriptions within forty eight hours. I remember that statement. Our GPUs are feeling it. Yes. Because they physically could not rack enough compute servers to handle the traffic. 7:46 And this perfectly highlights the technical divergence between training and inference. How so? Well, training is all about massive parallel compute calculating those gradients we talked about, but inference is deeply bound by memory capacity. Okay. A 2,800,000,000,000 parameter model is so monstrously large that its parameter weights simply cannot fit into the VRAM of a single chip or even a small cluster. 8:08 So it has to be sliced up. Right. It has to be sharded across those 64 chips. So every time a user asks a question, those 64 chips have to fire perfectly in sequence, passing the data back and forth just to generate one single token. Okay. 8:22 Wait. Let me push back on this a bit. If the h 20 chick is explicitly allowed by The US, why wouldn't these companies just buy thousands of them to meet the inference demand? That's a fair question. Like, we're seeing reports of companies like Tencent actually renting compute capacity from a data center near Osaka, Japan packed with restricted Blackwell chips, which is currently under formal investigation by the US Bureau of Industry and Security, by the way. 8:47 Right. So why take that massive geopolitical risk if a legal h 20 alternative is just sitting right in front of them? Because of the physical reality of data center physics and local supply chains Okay. You can legally buy the h twenties. But as we discussed, the lower compute density means you need an enormous physical footprint. 9:06 Yeah. Racking, powering, and cooling that many additional servers takes time they just don't have. I see. And local alternatives. Well, Beijing is actively discouraging the purchase of h twenties, and they're aggressively pushing domestic alternatives like Huawei accelerators. 9:21 Right. But the delivery times on those Huawei chips are stretching up to six months. Wow. Six months in AI is an eternity. Exactly. 9:29 If your user base is exploding today, a six month supply chain delay is a literal death sentence for your product. So you rent the dense compute in Japan because you literally have no other physical way to serve the model to your users right now. That makes total sense. Yeah. And to really understand why this inference compute is so heavily bottlenecked, we have to look at what these massive models are practically being applied to on the front end. 9:53 Right. The actual use cases. The sources detail a really fascinating technical application, something you listening can actually try, called Director. Oh, yeah. Director is amazing. 10:02 It's a chat based LLM experience running on Claude Opus 4.8. And Director is a brilliant case study in how developers are trying to abstract extreme computational complexity entirely away from the end user. It's designed to generate commercial ready video. Right? Like UGC ads, explainers, complex camera angles. 10:23 Yes. And good video generation is an incredibly highly parameterized task. But with director, the user relies on a method called vibe directing. I love that term. It's a great term. 10:34 Think of vibe directing like the difference between driving a manual transmission car and a fully autonomous vehicle. Okay. I like this analogy. In a manual car, you are physically manipulating the clutch, the gears, the RPMs. That's traditional prompt engineering for video. 10:49 Where you have to define specific parameter weights for lighting, motion blur, panning. Exactly. But the LLM in the director tool acts as the autonomous driver. You provide your high level creative intent of vibe and natural language, and the LLM translates that into the highly technical multimodal rendering pipeline required to actually generate the video. It keeps the product and the style mathematically consistent across up to a five minute video without you needing to understand any of the underlying architecture. 11:17 Right. It's doing all that heavy lifting in the background. And for those listening who wanna test how these massive parameter models handle complex inference tasks, the sources note there are discounted models available until July 31. Oh, that's good to know. Yeah. 11:31 You can try Seedance, 2.5 Nano, Gemini Omni Flash, and Cling v three. Testing these tools gives you a real world feel for why companies are fighting so hard for every ounce of inference compute. It really puts the hardware struggle into perspective. It absolutely does. Yeah. 11:46 But while Moonshot is battling these physical hardware limits, there's another layer to this. A recent breakthrough from OpenAI demonstrates that sometimes the limit on a model's intelligence isn't the silicon at all. No. Sometimes it's the software environment, the harness that we use to test it. Right. 12:03 Let's talk about the OpenAI ARC AGI three debugging logs. This is arguably the most fascinating technical revelation in the stack today. It's a massive shift in how we evaluate AI. The paradox is just stark here. GPT 5.6 internally called Sol is undeniably brilliant. 12:19 It solved long standing open math problems like the cycle double cover conjecture. It beat Pokemon FireRed autonomously too. Right. But when they ran it on ARC AGI three, which is a benchmark specifically designed to test two d puzzle games and spatial reasoning, its initial score was a dismal 7.8%. Now to contextualize that, 7.8 is actually still vastly superior to the previous frontier model, GPT 5.5, which scored point 4%. 12:47 Wow. That's a low bar. Yeah. Historically, no frontier model had ever even passed level one on this leaderboard. But given what Sol was achieving in complex mathematics, a 7.8% failure rate on two d puzzles indicated that something was artificially suppressing the model's measured intelligence. 13:03 So what did they do? OpenAI's engineers dug into the system logs, and they found two massive critical flaws in the generic testing harness. Okay. Let's break these down. The first flaw was discarded reasoning. 13:15 Right. After every single game action the model took, the testing harness wiped its private reasoning tokens. Which is devastating for a model. So every single turn, the model was forced to rebuild its understanding of this unfamiliar two d environment from absolute zero. The mechanics of reasoning tokens are so vital here. 13:32 Modern frontier models are trained to output a private chain of thought before they commit to an action. And this isn't just generating text, is it? No. Not at all. This is how the model mathematically maps out the latent space of the problem. 13:46 It is actively calculating probabilities and spatial relationships. So when the harness deletes that? When evaluation harness arbitrarily deletes those reasoning tokens after a turn, it is actively deleting the model's spatial and strategic awareness. It sounds like giving someone a highly complex multistage puzzle to solve, but every five minutes you inject them with amnesia and throw away their scratch pad. That's exactly what was happening. 14:10 It's no wonder the model looked dim, and that ties directly into the second massive flaw they found, which was rolling truncation. Yes. Rolling truncation. Truncation. The harness had a hard context window limit of a 175,000 characters or tokens. 14:23 Yeah. And once the conversation hit that limit, the harness simply chopped off the oldest messages. And rolling truncation is mathematically mathematically disastrous for multi step logic. Why is that? Because if you are solving a sequential puzzle, the spatial insights the model generated in level one are prerequisites for solving level five. 14:40 Oh, I see. If the harness blindly drops the first 20,000 tokens just to stay under the memory limit, the model loses the foundational rules it deduced. It is effectively flying blind in the later stages of the test. Okay. So how do they fix it? 14:55 The sources say the engineering fix was twofold. First, they swapped to the responses API. Right. Which is a huge structural change. And that allows the developer to pass the previous response ID. 15:06 So this automatically retains the model's private reasoning tokens across different turns and tool calls. Exactly. It fixes the amnesia problem. And then second, they ripped out rolling truncation entirely and replaced it with compaction. Compaction is a significantly more sophisticated approach to memory management. 15:22 Walk us through how compaction works technically. Instead of dropping tokens chronologically, compaction utilizes a lighter weight background process to semantically summarize the older context. It's compressing it. Exactly. It compresses the vital strategic information into fewer highly dense vectors. 15:40 This keeps the total token count under that a 175,000 limit, but ensures the core logic and rule discoveries remain totally accessible to the model in its active memory. That is brilliant. And the impact on the benchmark was just staggering. By simply fixing the software harness, the score skyrocketed from 13.3% That was with the official harness on the public set all the way up to 38.3%. Which is roughly triple the original score. 16:07 Triple the score. And astoundingly, it achieved this using six times fewer output tokens. It's incredible efficiency. Sol went on to solve all six levels of a game where no model had ever passed level one. Now the human baseline for this benchmark is around 48%, so that gap is closing rapidly. 16:22 It proves that the latent intelligence was already embedded in the model's weights. The hardware had done its job. The intelligence was simply being suffocated by an inefficient testing environment. Which yields a really highly specific technical recommendation for the developers listening to this deep dive today. If you are building applications on top of these frontier models, follow OpenAI's technical advice. 16:44 Move away from legacy chat completions and adopt the responses API. It's crucial. Make sure your architecture retains the model's private reasoning and implement semantic compaction rather than rolling truncation. Don't artificially kneecap the intelligence of the model in your own application. And if they want to visually grasp the complexity of the spatial reasoning we're talking about, I'd highly encourage listeners to try the puzzles themselves at arcprize.org tasks. 17:10 Yes. Definitely check that out. Because when you combine these two phenomenons, engineers hotwiring disjointed data centers to overcome massive hardware bans and developers tweaking an API call to triple a model's IQ overnight, you really begin to understand the sheer velocity of this industry. And moving incredibly fast. And that velocity is starting to trigger severe alarm bells among the people actually building these systems. 17:33 Right. Which brings us to the final piece of our source stack today, the industry backlash and the push for pacing the frontier. Yeah. The political and technical spheres are clashing hard right now. Earlier, we mentioned Michael Kathios. 17:47 Beyond the location of the k three training run, he also accused Chinese labs of distillation. Yes. Distillation. He claimed they're using US model outputs to train models like k three. And, again, just impartial reporting here. 18:00 No sides taken. But what is distillation technically? Distillation is a highly efficient training technique. Essentially, you take a massive, highly capable teacher model, and you use it to generate mountains of high quality synthetic Okay. You then feed that perfectly structured data into a smaller student model during its training phase. 18:21 Oh, wow. Yeah. The student model learns to mimic the complex reasoning of the teacher model using a fraction of the original compute. It is incredibly clever engineering, but it heavily blurs the lines of intellectual property. And The US tech industry is completely fractured on how to respond to this. 18:36 NVIDIA CEO Jensen Huang is aggressively defending open source ecosystems. Right. He stated that US companies should absolutely be integrating these highly efficient open weight Chinese models. And most legacy tech incumbents actually support that view. But you have fierce descent from Frontier Labs like OpenAI and Anthropic. 18:56 They're citing severe cybersecurity and national security concerns. Exactly. And this fracture has escalated from philosophical debates into formal demands. Over 1,100 lab employees, and we're talking senior figures from Meta, Anthropic, OpenAI, and Google, they've signed an initiative called pacing the frontier. It's a huge movement. 19:17 They are formally petitioning the US government to establish international governance tools designed to deliberately throttle and pace the development of AI. And the signatories on this are heavy hitters. Dario Emadet, the CEO of Anthropic, Don Song from Meta, Jacque Pichocchi, the chief scientist at OpenAI. The sources note that Anthropic specifically reference their own internal research on recursive self improvement as the primary evidence here. Right. 19:42 They believe these models could soon improve their own architecture faster than external safety risks could be managed. Recursive self improvement is really the critical technical threshold here. Can you contextualize what that actually means? Sure. Currently, a model like k three or GPT 5.6 is static. 19:58 Yeah. Once the training run ends, the parameter weights are frozen. It does not get intrinsically smarter sitting on a server. Right. However, recursive self improvement envisions deploying an autonomous agent that possesses the capability to write more efficient training code, optimize its own neural architecture, and eponymously initiate its own fine tuning runs. 20:20 So it's basically a feedback loop. Exactly. It creates a compounding loop of intelligence. An autonomous agent with read and write access to its own source code running loose on a server cluster represents a fundamentally different dynamic threat model compared to just a static contained LLM waiting for a user prompt. And the broader industry is already bracing for the security implications of this. 20:43 Following a recent security incident at Huggy Face, NVIDIA just announced a massive new AI safety and cybersecurity coalition with Adobe and CrowdStrike. Which is a step toward governance. It is. But looking at the technical reality we've unpacked today, I have to ask a difficult question. Go for it. 20:58 It. We just detailed how engineers are successfully routing around international hardware embargoes by hotwiring eight chip servers across entirely disparate cloud providers. Right. We just saw software engineers fundamentally rewrite evaluation harnesses overnight to triple a model's IQ without even touching the hardware. Yeah. 21:18 Given that level of decentralized ingenuity, does a coordinated pause or a centralized governance tool even work technically? Or is recursive self improvement and decentralized software development just going to inevitably outpace any choke point regulators try to build? That is really the defining question of this era of computing. Historically, software routes around restriction. Always. 21:41 If you lock down the hardware, like restricting Blackwell chip's engineers, optimize the network topology just as Moonshot did. Right. If you rely on a flawed generic harness to measure safety, the models will optimize the software API as OpenAI did with compaction. Regulation heavily relies on the existence of a centralized physical choke point. But the technical reality of Frontier AI is rapidly becoming decentralized. 22:03 It's decentralized, it's compressed, and it's highly distributed. So as you listening digest all of this, what is the through line? We've explored hot wired multi provider Blackwell clusters, the massive inference compute required for Vibe directing video tools, tripling benchmark scores through semantic compaction, and the very real industry panic over autonomous self improvement. The thread connecting every single one of these developments is the relentless drive for efficiency. Right. 22:33 The entire industry is obsessively focused on how to extract more intelligence out of less compute. And that drive for efficiency leads to a final provocative thought I wanna leave you with to mull over on your own today. It's a big one. Right now, the global focus is totally fixated on the physical hardware. Governments are fighting intense geopolitical wars over massive NVIDIA clusters, h 20 shipments, and export bans. 22:55 All physical silicone. Exactly. But think about the trajectory of the software we just discussed. If AI models become exceptionally proficient at compacting their own reasoning, if they achieve recursive self improvement and autonomously write highly efficient code that requires only a fraction of the compute to achieve the exact same result Then the hardware paradigm shifts completely. Yes. 23:19 Those massive power hungry hardware clusters we are fighting over today might become entirely obsolete. It's a profound paradigm shift. The bottleneck of intelligence may not be tied to physical silicon for much longer. We might be pouring all of our global regulatory energy into controlling the equivalent of giant room sized mainframe computers right on the eve of the smartphone revolution. That is a wild thought to end on. 23:43 It really is. Thank you for listening in. Subscribe and follow Colaberry on social media links in the description, and check out our website, www.colaberry.a I backslash podcast for more insights like this.
