The 20-watt reference design

Your brain runs on about 20 watts. That's a dim light bulb. For that budget you get real-time vision, audio, language, motor control, a continuously updated physics model of your surroundings, and enough prediction ability to catch a set of keys someone tosses at you without thinking.

Now price the same capability in silicon. A single H100 draws 700W and handles a narrow slice of that skill set. A rack handles more, but a rack doesn't fit in your skull and doesn't run on lunch. We're building gigawatt campuses and negotiating with power utilities like nation states. The reference implementation has been running on leftover sandwiches for 200,000 years.

OpenAI CFO Sarah Friar's recent post framed the industry's actual strategy: advances across chips, compute, models, and products compound into more useful intelligence at lower cost. That's the polite version. The uncomfortable version is that the hardware bill is so absurd that a movie interpretation of it doubles as a good infrastructure metaphor.

What the Matrix is really about

The Matrix gets a bad rap for the battery scene. Thermodynamically it's nonsense. If the machines wanted energy from liquefied protein slurry, they'd burn the slurry directly. The human is a lossy middleman with opinions. But the machines didn't build a planet-sized data center to run a money-losing power plant. They built a GPU cluster, and the battery speech was the cover story.

The simulation is the workload. A sedated human makes a terrible battery, but a human running a rich, coherent world model with politics and taxes and dial-up internet is a perfectly utilized compute node. That reframes the plot's failure modes too. The first Matrix, the "perfect world" Agent Smith describes, failed because a frictionless paradise is a trivial workload. Nobody's brain does interesting work in heaven. The machines didn't crank up suffering because humans need it to feel real. They cranked up difficulty to keep utilization high.

The metaphor maps cleanly onto infrastructure: you don't want idle capacity, you want a workload that keeps every node busy. The Matrix is a planet-scale job scheduler with an excellent retention story.

The biological detour and what it teaches us

The science fiction isn't entirely fiction. FinalSpark runs lab-grown human brain organoids as a cloud platform. Cortical Labs taught a dish of neurons to play Pong and productized the unit. Both pitch the same appeal: orders of magnitude less power than silicon for certain kinds of learning.

The catches are real. Organoids live weeks, maybe months. Nobody can program them in any conventional sense. And neurons are slow, firing in milliseconds against transistor nanoseconds. This was never a drop-in H100 replacement. Different machine, different job.

The more likely path is neuromorphic silicon. Loihi, NorthPole, SpiNNaker: steal the architecture from biology, spiking neurons, memory sitting next to compute instead of across a bus, massive parallelism, then implement it in silicon so you get nanoseconds instead of milliseconds. Copy the design, drop the latency, skip the ethics review.

Substrate	Power	Signal speed	Maturity	Where it fits
Human brain	~20 W	Milliseconds per spike	200k years of field testing	Low-power real-world cognition
H100-class GPU	700W+ per chip	Nanoseconds	Shipping in volume	Dense matrix math at scale
Neuromorphic silicon (Loihi, NorthPole, SpiNNaker)	Watts to low tens of watts	Nanoseconds (emulated spikes)	Research and early products	Sparse, event-driven workloads
Organoid platforms (FinalSpark, Cortical Labs)	Watts	Milliseconds (real spikes)	Lab experiments	Learning research, not production

What the community is saying: when I worked through the 20-watt comparison, the part that stuck with me is a comment from the dev.to thread about where the real bottleneck sits. We keep improving silicon and scaling data centers, but the brain still does an enormous amount of parallel, adaptive processing at almost no power. The harder question is whether the trick is in the architecture or in the wet chemistry. The neuromodulators, the dendritic computation, all the analog mess that doesn't draw cleanly on a whiteboard. If it's biochemical, then neuromorphic chips copy the blueprint and miss the point. The bottleneck may not be hardware speed at all. It may be that we don't yet understand how the brain actually computes.

Chip land-grabs: Anthropic's $7B MatX conversation

The infrastructure fight is moving up the stack into silicon itself. Reuters reported that Anthropic discussed acquiring MatX for around $7 billion before the talks shifted from acquisition to potential partnership. The failed deal is the tell: model companies are now willing to spend billions to own chips.

MatX is a serious target. Founders Reiner Pope and Mike Gunter both came from Google. Pope worked on TPU software and large-model infrastructure. Gunter did TPU hardware design. MatX closed a $500M Series B in February with Jane Street and Situational Awareness participating, and it designs chips specifically for large language model training. That training focus is the most interesting detail, because it sits in contrast to OpenAI's first self-designed chip, Jalapeño, which targets inference after training.

Anthropic's hiring tells the same story. The company brought in Amir Salek, who spent eight years at Nvidia running its SoC design group, then led Google's TPU program through seven generations. It also hired Clive Chan, a former OpenAI chip engineer. Anthropic reportedly continues exploring multiple AI chip startups, planning to keep a multi-vendor approach with Nvidia and Google. The strategy reads clearly: build internal expertise, survey the landscape, acquire if the right asset appears.

Key numbers

• 20W: the human brain's entire power budget
• $7B: Anthropic's reported acquisition price discussion for MatX
• $500M: MatX's Series B, led by Jane Street and Situational Awareness
• 7: generations of TPU that new Anthropic hire Amir Salek helped deliver

Quick Take: every major lab is vertically integrating down to silicon because a few percentage points of training efficiency swing hundreds of millions of dollars in compute spend. Chip design is becoming part of model architecture.

This isn't just Anthropic. Google has TPU, Amazon has Trainium, OpenAI has Jalapeño. The pattern is structural. When models get big enough, the hardware becomes a co-design problem, not a procurement problem.

Memory is the other front

The chip race gets the headlines, but memory is the quieter constraint on deployment. A Reddit user spotted a 192GB Framework motherboard on the manufacturer's site. At existing memory SKU price points, that board likely lands around $4,500. Reports also suggested the PCIe slot opens at the back of the chassis, possibly delivering up to 75W.

Why does a laptop motherboard matter in an infrastructure story? Because 192GB of unified memory means you can run a 70B-class model locally without quantization gymnastics. That changes the inference cost equation for an entire category of workloads: private data never leaves the machine, no per-token API pricing, no cold starts.

Apple's unified memory architecture already pushed Macs to 128GB and beyond. Framework is the first mainstream x86 vendor to chase that same target. For a developer toolkit, that's a meaningful shift. The most expensive part of local AI isn't the GPU anymore. It's the memory bandwidth and capacity to hold the weights.

The practical implication: model deployment is splitting into two tiers. Cloud for frontier-scale training and high-concurrency serving. Local unified memory for interactive work, privacy-bound data, and latency-critical single-user inference. The 192GB board makes that second tier far more credible.

The abstraction layer over uncertain silicon

Most teams can't design chips or buy $4,500 motherboards. They need the hardware story abstracted away. The latest version of that abstraction is Gradio's gr.Workflow.

The pitch is direct: make the pipeline the interface. You describe steps as a graph of typed nodes, and Gradio serves a drag-and-drop canvas where every node is runnable and every intermediate result is visible. The same graph is a REST API, and each output automatically gets its own endpoint (/sticker, /voiceover, /episode_title). Callable from Python, reachable over curl, deployable to Spaces with one command.

The fan-out pattern is where it gets useful. One idea feeds multiple operators simultaneously: a base image from FLUX, two re-imaginings, a gallery title from an LLM, all generated in parallel. You can also run a model inside the Space itself by decorating a bound function with @spaces.GPU, letting ZeroGPU grab a GPU for the call, run the model, and release it.

The deeper point: workflow tools are becoming the deployment plane. They abstract away whether your model runs on your hardware, on Hugging Face Inference Providers, or in a GPU-equipped Space. The graph is the architecture. The infrastructure underneath is replaceable.

A production system that actually works

For the clearest view of how this all fits together, the Papers with Code search system is a small masterclass. It runs hybrid search over 110,000+ papers using three Hugging Face services, each doing the job it's best at.

The design principle is strict separation of throughput work from latency-sensitive work. Full-corpus embedding runs as a batch Job on an L4 GPU, writing to a Storage Bucket that acts as the durable handoff between systems. The live query path only does one small thing: embed the user's query with a pinned model revision, then run vector search. Jobs handle rebuilds and backfills. Inference Endpoints handle interactive queries and small incremental updates. Buckets preserve every build's artifacts.

Two technical calls stand out. First, the embedding contract is treated as a versioned API. They pin the model revision, output dimension, input format, normalization method, and a content hash. Vectors can't silently drift between corpus builds. Second, they use Qwen3's Matryoshka embeddings truncated to 256 dimensions. On a 5,000-paper pilot, the 256-dim index hit 0.9955 Recall@20 against exact search with 1.31ms p50 and 2.21ms p95 HNSW lookup latency. For context, those latencies mean semantic search adds imperceptible overhead to a query; a user can't tell the difference from plain keyword search.

The cold-start handling is the part most production systems get wrong. The endpoint can scale to zero, which saves money, but a cold endpoint can't serve a query in time. Their client has a one-second timeout, a circuit breaker, and a lexical fallback. If semantic search is slow or broken, users still get full-text results. The system degrades gracefully instead of failing.

One thing I appreciated in their writeup: restarted Jobs skip completed shards using marker files. Retrying a huge corpus build resumes work instead of redoing it. That's the kind of detail that separates an architecture that works from one that works on the demo day.

Common pitfalls

Pinning only the model name, not the contract. The revision, embedding dimension, prompt format, normalization, and input formatter all affect retrieval. Store them together and validate them everywhere. A silent model revision update can change your embeddings and degrade search quality without any error.

Using the same pipeline for batch and interactive workloads. Corpus embedding is throughput-bound. Query embedding is latency-bound. Run them separately. A batch job waiting on an interactive endpoint is a design failure, and an interactive endpoint buried in batch work will time out for real users.

Scaling to zero without a fallback path. Cold starts take time. If your endpoint can spin down, your application must handle a slow or unavailable endpoint. Design for cold starts as a normal state, not an exception. Hybrid retrieval is the natural fallback: lexical search always works.

Assuming a custom chip solves your bottleneck. Vertical integration into silicon only pays off at enormous scale. If you're not running tens of thousands of accelerators, the operational complexity of chip development will eat the savings. The labs buying chip startups are doing it because a few percent training efficiency saves them billions, not because GPUs are bad.

Forgetting that workflows are graphs, not scripts. With gr.Workflow, the fan-out pattern is the main performance lever. If you chain nodes linearly when they could run in parallel, you're paying serial latency for no reason. And if you're calling a remote model when a local GPU node would work, read up on @spaces.GPU.

The bottom line

If you're building a retrieval or search product, adopt the throughput/latency split with an explicit storage handoff. Batch embed into immutable, versioned artifacts, serve queries through a fallback-protected endpoint, and pin your model contract to an exact revision. That architecture carried 110,000 papers and it'll carry your workload.

If you're deploying models locally, the 192GB-class unified memory boards are the sweet spot for 70B-class models. Don't wait for the ecosystem to standardize. The price lands around $4,500, which beats per-token cloud costs within months for sustained interactive use.

One thing to watch: the vertical integration wave is accelerating. Anthropic's MatX talks, OpenAI's Jalapeño, Google's TPU, Amazon's Trainium. Expect at least one of the major labs to tape out a training chip in the next 24 months. When that happens, training economics change, and every infrastructure decision you made on GPU pricing gets revisited.