I’m also releasing v0.1 of the ”Silicon Vanguard” dataset, which powers the bulk of this analysis. Chinese chip companies aren’t as forthcoming about individual unit specs, so it’s difficult to find performance data from reputable sources. I decided to compile my own from securities filings, archived company pages, and other firsthand sources, and I will be adding more to it over time.

This is my longest post yet - 13,000 words - so fair warning, instead of just diving in, I’d recommend the following order:

POLICYMAKERS / BIG PICTURE FOLKS:

• Executive Summary

• Skim Profiles

• Bookmark the Silicon Vanguard dataset

• Industry Certifications

• Strategic Analysis → Entity List Impact

• Strategic Analysis → Strategic Backing

TECH-FORWARD ANALYSTS:

• Executive Summary

• Skim Profiles

• Bookmark the Silicon Vanguard dataset

• Data & Methodology (all)

• Technical Analysis (all)

• Strategic Analysis → Entity List Impact

• Strategic Analysis → Commercial Adoption

While the content itself is written by me, this was a valuable stress-test on modern LLM limitations for deep-tech, bilingual industry analyses, revealing fault lines on citations and hallucinations. Western model providers almost never use firsthand Chinese-language sources, but handle mixed-language PDFs okay. DeepSeek, Kimi K2, and Chinese models are way better at sourcing, but suffer from more serious hallucinations in long context windows, requiring a lot of triple-checking and filter-maxxing. To this end, if anything is inaccurate, please comment with other firsthand sources! Will edit and gladly attribute the correction.

Lastly, my friend Lesley Gao (author of Shear Force) was incredibly helpful in sourcing additional materials to pad out financials and leadership information, as well as contributing to general takeaways. Her recent post, How a $1 Lighter Defied Inflation for 20 Years, is an excellent primer to Chinese industrial clusters, and as a framework carries serious relevance for all kinds of industrial policymaking.

Introduction

If you ask the question “how far behind is China on leading edge semiconductors,” you’ll get as many different answers as the number of people you ask. 20 years1, 10 years2, 5 years3, 2 years.4 If you ask “who will become China’s NVIDIA,” you may most commonly hear “Huawei,” or more recently “Cambricon.”5

To me, all variants of these questions are unsatisfactory. Focusing solely on the immediate, visible events - the “leading edge” - overlooks underlying systems that generate and sustain technoeconomic momentum. After all, the Titanic didn’t strike the ice above the surface… it struck the entire iceberg underneath. This is why technological change often feels gradual, then all of a sudden.

Additionally, while most domestic chips lag foreign competitors in raw performance, obsessing over isolated stats ignores an objectively rapid pace of development. Their performance on energy-compute metrics, I believe, are more germane to questions on sovereign AI.

A more nuanced understanding of China’s chip ecosystem is therefore warranted.

In this post, I’ll pull back the veil on 16 companies we’re tracking in this space, and attempt to make sense of their relative positioning in the market. These breakouts are finding new ways to bypass tech bottlenecks with heterogeneous approaches, deep cross-sector collaborations, and novel design paradigms. There are also clear losers which highlight limits to a state-led innovation ideology. Truly there are dozens of companies worth mentioning, but only a handful have made meaningful contributions to the country’s sovereign chip ecosystem thus far.

To be clear, we’re focusing solely on the fabless accelerator (GPU and ASIC) chip designers of today. We’ll save CPUs, FPGAs, and memory chip IDMs like SMIC, CXMT, and YMTC - all critical parts of the value chain - for a future deep-dive.

It’s fair to argue this analysis is incomplete without fully assessing domestic ability to reliably fabricate these chips (e.g. SMIC, YMTC, CXMT), but as we’ll see, even if you hold domestic yields frozen at current levels, you have to admit designers have made considerable strides along the tech tree.

Executive Summary

I’ll summarize the findings of this analysis up front, and lay out its implications for American technologists and policymakers. Read on for fuller details on individual companies, performance specs, and qualitative measurements.

Takeaways

First, while U.S. entity listings may have had a deleterious effect on some individual first movers (e.g. Biren, Moore Threads), this strategy undoubtedly catalyzed domestic chip and software innovations bypassing one-off tech bans. As an example, this month Huawei HiSilicon plans to open-source its Unified Cache Manager (UCM), an AI inference acceleration toolkit which shards model memory workloads across different memory types (HBM, DRAM, even SSDs), reducing latency by up to 90% and improving system throughput by up to 22x.6 This would significantly blunt the impact of HBM restrictions on sanctioned entities, which experts agree is a critical bottleneck for domestic accelerators.7

Second, the domestic ecosystem is converging on model, hardware, and precision standards meant to establish “good enough” performance for large-scale deployments. DeepSeek has also announced its support for UE8M0 FP8 precision as a performance standard for domestic chips, and chipmakers are proclaiming FP8 support for their newest SKUs (Cambricon 思元690, Enflame 邃思L6008, SOPHGO SC11-FP3009). CAICT recently began issuing “AI Chip and Large Model Adaptation Test Certificates” to chipmakers (Moore Threads, SOPHGO) that have demonstrated “passable” inference performance for full-blooded DeepSeek R1 671B. This third-party baseline establishes acceptability thresholds for performance and energy efficiency, and provides clarity to buyers and sellers in the procurement process for multi-million dollar chip contracts.

Third, hardware-specific investments are bearing fruit in sparse computing and in-memory / near-memory (PIM / NDP) designs. Moffett AI’s first-gen chips have already out-performed the NVIDIA H100 in MLPerf Inference benchmarks, achieving >1.6x performance throughput at a ~3x smaller energy footprint, yielding ~5x greater tokens / joule in single-card and supernode environments.10 Continued advancements in sparsity, analog computing, and heterogeneous computing techniques are improving energy-compute yields and reducing total cost of ownership.

Finally, the last pillar may be NVIDIA’s software moat (CUDA), though this too may be about to crack. While CUDA (and even AMD’s ROCm) are still preferred by the majority of Chinese AI engineers, leading chipmakers (Baidu Kunlunxin, Moore Threads, MetaX, Enflame) have announced top-level priorities to ensure full CUDA compatibility for their newest chips, and vendor-specific transcompilers are seeing remarkable improvements in CUDA translation performance (Cambricon’s Qimeng-XPiler yielding 95%+ accuracy and <5 hours debugging time)11. This is likely a greater threat to CUDA dominance than a head-to-head CUDA vs. CANN open-source developer battle.

In sum, we’re witnessing an inflection point in domestic Chinese chip capabilities. Despite lagging behind trailing-edge NVIDIA chips, domestic hardware is evolving beyond mere sufficiency for powering a sovereign AI ecosystem - it may soon be sufficient to begin exporting the Chinese AI stack.

Implications

In case I haven’t made this abundantly clear… the horse has left the barn. Continued attempts to control, deny, or retard domestic Chinese progress will only accelerate this now inevitable transition. Cold War-era technology restrictions are completely at odds with how international developer ecosystems function, and in the modern era create antithetical outcomes to their stated objective.

Senior AI Policy Advisor Sriram Krishnan articulates this reality well, stating the U.S. needs to maximize developer market share of the “American AI Stack” - the models, chips, and software being used to train and run them. It’s a classic developer flywheel.12

We are already in the first bouts of this “AI Stack” competition. Here in the Valley, most developers and startups I know are using Chinese models in some capacity (Martin Casado of a16z estimates 80%)13. And who can blame them - these models are open-source, meet or exceed American model quality, and are 10x-30x cheaper when hosting for inference.

Meanwhile, domestic first-movers OpenAI14 and Anthropic15 are implementing restrictions to manage inference costs, nerfing performance, throughput, and output quality in the process. This is a fast track to AI platform decay - or “enshittification” (平台屎化, píngtái shǐ huà) - and discourages developers from building on volatile infrastructure.

This reality demands a strategic recalibration. While a holistic approach to building the American AI Stack is a starting point, evidence from China’s Silicon Vanguard demonstrates that energy-compute optimization is becoming the decisive factor in at-scale deployment of AI systems… and will come to define sovereignty in the new world order.

Containment is no longer an option. The only option is to compete.

The Players

This won’t be an all-inclusive list. There’s a lot of investment dollars going into this sector as China rapidly builds up its domestic semiconductor manufacturing and design muscle.

Here are the names of the 16 entities we’ll be covering.

Below are other logic chip designers primarily building CPUs and FPGAs. They aren’t the subject of this post, but we’re tracking them nonetheless.

Groupings

We categorize these 16 players out into 6 groups based on their stages of development, adoption, and inherent competitive advantages.

Tier List

Since most people are going to skip to this section anyway, I’ll tee up my domestic tier list for the players we’ve evaluated. This was inspired by Nathan Lambert in his China Research Lab breakdown, but is by no means an authoritative declaration - it’s subjective and shouldn’t be taken too seriously.

To be clear: this is strictly a domestic tier list. We’ll include comparisons to China-available NVIDIA and AMD chips later on, but including them here would pollute the purpose of the exercise.

S through D tiers (along with the 狗屎 bottom tier) are ranked based on a rubric assessing company performance across 5 dimensions: product, leadership, developer adoption, commercial adoption, and strategic backing.

A separate tier, “Edgerunner,” is reserved for players pursuing novel hardware designs that effectively bypass systemic energy-compute bottlenecks. They may have either limited or unavailable information on commercial deployments. This is not an ordinal ranking per se, but a callout that we should pay close attention to progress on these design paradigms.

Ranking Rubric

The analyses sections go into plenty of detail on evaluations - the rubric is here for reference.

Company Profiles

Taxonomy Primer

Skip if you’re already familiar.

In semiconductor coverage, a bunch of acronyms can get confusing. It also doesn’t help when technical and marketing terms are often conflated (e.g. Google “TPU” brand, SOPHGO’s TPUs, and Zhonghao Xinying’s “GPTPUs”). This confuses retail investors, policymakers, and seasoned analysts alike.

To keep it simple, here’s a primer for logic chip taxonomy, as well as where each of our companies fit in that schema. If the entity designs chips in multiple categories, it’s mentioned in each.

Italicized names are not covered in this post.

As a general rule of thumb, this list is in order of generalizability and energy efficiency for AI workloads. GPUs are general-purpose and strike a good balance between performance and efficiency, but are outclassed by ASICs on energy-compute terms. ASICs have slower iteration times due to hardware life cycles and a more limited customer base, but are often the logic chip of choice for large hyperscalers who are their own customers. Lastly, PIM / NDP chips are the most energy-efficient logic chips by orders of magnitude, but are still in an experimental stage of development. Data on their performance in production environments is somewhat limited.

🦾 Heavyweights

Well-funded silicon teams wholly-owned by a major Chinese technology incumbent, with access to top talent, recurring revenue, and AI cloud real estate for deployments and co-designs. Strategy is generally to win at rack-scale, not at card-scale - custom interconnects a major boon.

Huawei HiSilicon (华为海思)

1991 | hisilicon.com | Private | Entity Listed 🚩

China’s most vertically integrated NVIDIA alternative: Da Vinci core-based 910A/B/C parts bound into Atlas boards/servers and rack-scale CloudMatrix systems, with CANN software and optical backplanes. The 910C centers on dual-chiplet packaging (~53 B transistors, ~64 AI cores) with 3D die-to-die fabric and HBM3, serving as the building block for CloudMatrix 384 across 16 racks.

Strategy is “win at rack scale,” trading device-level peak for throughput-per-rack and supply security; pricing (≈¥110k for 910B; ≈¥180–200k for 910C) underscores the TCO pitch versus scarce H100/H200. Adoption is broad across state-linked telcos, finance, and internet platforms; some 910C deliveries push into late-2025 as capacity ramps.

T-HEAD / Alibaba (平头哥)

2018 | t-head.cn | Private

Alibaba’s chip arm merging C-SKY + DAMO chip team: open-source Xuantie RISC-V cores on one flank, Hanguang NPUs and Yitian CPUs on the other. Hanguang 800 (12 nm; large on-chip SRAM, flexible precision) was architected as a high-efficiency inference engine for Alibaba workloads (images/video, recommendation, “City Brain”). Patent surveys point to a new phase of chiplets and dataflow-centric fabrics with adaptive tiling and aggressive mixed precision. chiplet-based, dataflow-centric fabrics with adaptive tiling and aggressive mixed precision.

Adoption is primarily internal/partner-cloud: Panjiu/HPN7.0 fabric interconnects thousands → hundreds-of-thousands of accelerators; migrations from DCN+ to HPN show material end-to-end training gains. ALink aims to NVLink-like scale for both international and domestic chips - a vertically integrated path to cost/latency control rather than chasing peak single-card specs.

Kunlunxin / Baidu (昆仑芯)

2011; spun-out 2019 | kunlunxin.com | Private

Baidu-born XPU line integrated with Baidu Cloud + PaddlePaddle: 1st-gen Samsung 14 nm shipped widely; Kunlun II (7 nm, XPU-R) expanded the base; P800 (XPU-P) targets LLM training and 8-bit inference with rack-scale 超节点 cabinets. Advantage is tight alignment with Baidu’s software and services.

Adoption proof points include China Merchants Bank’s AI chip project (P800 backing full-fat Qwen/DeepSeek variants on 8-card nodes and clusters) and Baidu’s own 万卡→三万卡 P800 fleet growth in-cloud.

🐉 Four Little Dragons (四小龙)

An informal industry term granted to four well-funded, novel fabless semiconductor companies. Prospects are generally positive. All have announced their intent to IPO or have already filed prospectuses.

Enflame (燧原科技)

2018 | enflame-tech.com | Private

Shanghai ASIC house led by AMD alumni (Zhao Lidong, Zhang Yalin), optimized for scale-out systems over single-chip hero numbers. Tencent is both anchor investor and proving ground; Big Fund participation adds patient capital. 2024–25 pushed mass inference (S60) while refreshing training (L600 训推一体 with higher on-package memory/interconnect).

Traction is real: ~70k S60 units shipped; the Gansu Qingyang 10,016-card S60 cluster under 东数西算 is a marquee win; earlier provincial clusters (e.g., Yichang) and wide Tencent workload coverage bolster credibility. The software ecosystem (“驭算/鉴算”) still trails CUDA, but the company is investing to close op coverage and porting costs. WAIC 2025 showcased “DeepSeek-ready” all-in-ones for R1-671B, doubling down on visible, workload-level validation.

Moore Threads (摩尔线程)

2020 | mthreads.com | Private (IPO Prospectus) | Entity Listed 🚩

Beijing fabless GPU vendor founded by ex-NVIDIA China GM Zhang Jianzhong; heavily backed (ByteDance, Tencent, China Mobile equity arm, Lenovo Capital) and fast-iterating across 苏堤→春晓→曲院 with the MUSA stack; shifted from TSMC to SMIC post-Entity List.

Adoption signal is improving: CAICT validated the MTT S4000 for large-model inference; deployments span Kuaishou (1,000-GPU cluster), the big three telcos, and university/energy partners. Datacenter SKUs (S3000/S4000) anchor B2B, while the E300 edge module (AB100 AI SoC, ~50 TOPS INT8) shows consumer/edge ambition. Performance lags NVIDIA’s top end but distribution, capital, and ecosystem deals give it staying power domestically.

MetaX (沐曦集成电路)

2020 | metax-tech.com | Private (IPO Prospectus)

Shanghai GPGPU startup (AMD-veteran founding team) with three series—曦思 N (inference), 曦云 C (training/compute), 曦彩 G (graphics)—plus CUDA-like MXMACA and MetaXLink interconnect. STAR Market IPO prospectus accepted 2025-06-30; >25k units shipped; next-gen dual-chiplet C600 touts 144 GB HBM3e.

Commercial motion leans on operator/provincial compute platforms and national distributors: disclosed multi-billion-RMB orders via integrators, Lingang partnerships, and “千卡级” clusters. Earlier parts relied on overseas HBM/foundry; newer N300/C600 are billed as domestic supply chain supported (“基于国产供应链”) signaling a risk-mitigation pivot while keeping cost-right inference as the spearhead.

Biren (壁仞)

2019 | birentech.com | Private | Entity Listed 🚩

Early bright star in domestic semiconductor design, Shanghai-based player built around the BR100 7nm chip with a star lineup of ex-NVIDIA/AMD/Huawei/Alibaba talent; later leadership churn followed Entity List actions and a pivot from TSMC to SMIC (N+2) with simplified derivatives (BiLi 106/110/166). Software stack (BIRENSUPA) is AI-first, not consumer graphics, and does not seem CUDA-friendly.

Commercially, Biren is lagging newer dragons, but has nonetheless landed state-linked telcos/clouds (China Mobile/Telecom), SenseTime, State Grid, and Shanghai AI Lab (1,000-GPU cluster). Newer liquid-cooled OAM lines and the “LightSphere X” photonic SuperNode (with Xizhi & ZTE) target density/efficiency at 2,000-GPU scale; first stop is the Shanghai INESA Intelligent Computing Center.

🏅 Public Champions

Publicly traded firms with considerable success in commercial deployments.

Cambricon (寒武纪)

2016 | cambricon.com | 688256.SH | Entity Listed 🚩

CAS/USTC child prodigies spun early domestic accelerator research (DianNao) into a commercial ASIC line; after early highs (Kirin 970 era) and a slump (Entity List, lost smartphone socket), the Siyuan 590 (domestic 7 nm) catalyzed a resurgence with A100-class training in some metrics. H1’25 revenue spiked ($400M) but is highly concentrated (top-5 ≈95%, one ≈79%—rumored ByteDance).

Cambricon is again relevant in training-class deployments (Siyuan 590 now, 690 next), fueled by heavy R&D (≈¥1.07 B in 2024, ~91% of revenue). Concentration risk is real, but the technical IP, government alignment, and renewed customer pull have shifted the narrative from “fallen star” to “second act.”

Hygon (海光信息)

2014 | hygon.cn | 688041.SH | Entity Listed 🚩

Originally established as a joint-venture between AMD, Chengdu Haiguang Microelectronics, and Chengu Haiguang Integrated Circuit Design. Best known for CPUs, Hygon has been reorganizing into accelerators with DCU-branded GPGPU-like parts (CUDA-compatible environment). DCU8100 accelerator reports parity with A100/MI100 in some precisions. A major 2025 move was the announced merger with Sugon (曙光) to tighten the HPC stack.

Current accelerator push seems to be riding existing HPC/government channels while CPUs remain the revenue bedrock. Sugon tie-up could accelerate system-level offerings as domestic alternatives mature, especially in heterogeneous computing systems.

🧪 Specialists

A mixture of startups and established players pursuing ASICs and other specialized hardware paradigms. While the Ascend, Hanguang 800, and Cambricon MLU lines are technically ASICs, their incumbency separates them from this bucket.

SOPHGO / SOPHON (算能科技)

2016 | sophgo.com | Private | Entity Listed 🚩

Bitmain-lineage consolidation of SOPHON (cloud/edge TPUs) + CVITEK (edge vision SoCs) into an end-to-end AI stack spanning BM1684X/1688 to the new BM1690 datacenter TPU, plus SG-series RISC-V CPUs and MLOps/toolchains. Recent flagship SC11-FP300 (BM1690, 256 GB LPDDR5X, ~1.1 TB/s) and a 128-TPU liquid-cooled “SuperNode” target LLM inference at scale; the SophNet service fronts popular domestic models.

Historically strongest in video analytics and city/industrial AI; 2025 marks a pivot to mainstream LLMs. CAICT validated SC11-FP300 against DeepSeek-R1-671B inference; broader hyperscaler uptake remains to be proven, but product cadence and platformization are heading in the right direction.

Zhonghao Xinying (中昊芯英)

2018 | zhcltech.com | 605255.SH (Pending SSE inquiry)

Hangzhou TPU upstart founded by a Google TPU alum (worked on v2/v3/v4); markets “GPTPU” training chips (刹那/Chana) with 1,024-chip scaling and a first “Taize” cluster concept. Positioning is assertive on perf/$ versus “overseas GPUs.”

Pilots are tangible: Guangdong Unicom’s initial 32-node TPU center slated to grow, China Mobile Tianjin’s TPU intelligent computing center with Taiji servers, and a Zhejiang University research platform. But corporate maneuvering (reverse-merger path, unusual trading signals) and aggressive revenue claims raise a lot of red flags.

Moffett AI (墨芯)

2018 | moffettai.com | Private

Silicon Valley-founded, now Shenzhen-based. Highly promising sparse-first accelerator company (Antoum chip/S4/S10/S30) focused on algorithm-hardware co-design for high-efficiency inference; adjacent work touches PIM/NDP and compiler/runtime research. Public technical presence includes Hot Chips, MLPerf, and a long paper trail on sparsity/near-data computation.

Large-scale hyperscaler adoption not yet substantiated, but public inferencing benchmarks support SparseOne cards for large model deployments. The bet is that structurally sparse LLMs + memory-centric designs can deliver outsized perf/W once software maturity and workloads align.

InnoStar Semiconductor (昕原半导体)

2019 | innostar-semi.com | Private

Interesting ReRAM/PIM specialist tackling the memory wall by colocating compute with storage (“ATOM” memory-compute). Backed by a mix of global and domestic capital (e.g., KPCB, Lam Research, Ant Group, ByteDance), reflecting both technical ambition and strategic relevance.

This is a platform-stage play where success hinges on reliable ReRAM arrays, compiler/tiling toolchains, and marrying the parts to transformer-era dataflows. As performance data is not available and commercial adoption has yet to be publicized, the inclusion is thematic more than a endorsement of their specific offering.

🐢 Trailing Incumbents

Established or older GPU incumbents whose commercial adoption is trailing newer competitors.

Iluvatar CoreX (天数智芯)

2015 | iluvatar.com | Private

Early GPGPU entrant (founded 2015) with IxRT SDK and the Tiāngāi (training)/Zhìkǎi (inference) lines; leadership turbulence (chair/CEO shuffle, probe) but continued technical progress and financing. Tiāngāi 100 was among the first 7 nm domestic training GPUs; follow-ons target parity with A100-class baselines in select tasks. 天该150 barely outside the A100 performance and efficiency window in energy-compute analysis.

Adoption has skewed to consortiums and research: BAAI training >70B-param models and integration at the National Supercomputing Center in Wuxi show capability; within heterogeneous clouds, presence is modest. Strategy favors ecosystem building and pilots over mega single-tenant wins.

Denglin Technology (登临科技)

2017 | denglinai.com | Private

Niche GPU/accelerator vendor with the Goldwasser II family (from 15–150 W SKUs covering MXM to PCIe). Positioning emphasizes INT8/FP16 density at low power envelopes rather than head-to-head datacenter training.

Commercially, the most visible activity is an “AI+ PC” tie-up with Lenovo; beyond that, evidence of large-scale traction is limited. Reads as an edge/embedded play more than a hyperscale contender.

💩 狗屎

Companies that have thus far failed to demonstrate any commercial viability for AI accelerators whatsoever, but tenure in the industry forces a mention.

Jingjia Micro (景嘉微)

2016 | jingjiamicro.com | 300474.SZ | Entity Listed 🚩

Changsha-based, defense-rooted fabless GPU/SoC designer and one of China’s longest-standing GPU houses; a designated “National Specialist, Unique, and New Key Little Giant.” The Little Giant status delivers both hard benefits (non-dilutive grants, priority “Big Fund” access, policy-bank credit) and soft advantages (procurement preference, reputational signaling). Founders/executives came from CETC 38th Institute with NUDT pedigrees.

Adoption concentrates in avionics/radar and government PC/workstation localization—little evidence of modern AI workloads; H1’25 revenue fell ~45% YoY with GPU segment down ~63%, indicating continued reliance on PLA/SOE demand. Has been completely unable to materialize early leads or government advantages in product development and commercial deployments for AI workloads. JM11 chip line seems to be a last-ditch effort to remain in the race.

Data & Methodology

I’ll summarize the scope of this analysis in the below table, then expand on some areas I know will have the comments section fraught with whataboutisms.

The attributes I care about revolve around energy-compute fundamentals, commercial traction, and leadership acumen. In absence of demonstrable commercial progress, strategic backing or announced collaborations in high-profile “Eastern Data, Western Compute” (东数西算)16 and Intelligent Computing Center projects are helpful, but not a substitute.

Introducing the “Silicon Vanguard” Dataset

I’ve compiled first-party and trusted third-party resources on foreign and domestic chip performance specs into a single artifact I’m calling the Silicon Vanguard dataset (for lack of a better name for now).

This initial version, which I’m sharing publicly, combines performance data from company resources, securities filings, semiconductor conference presentations, reputable equity researchers and industry analysts, and more. Third-hand data reported by tech bloggers or general tech media is more speculative than helpful, so I’ve ignored it at the expense of relative completeness of the dataset.

Source: Silicon Vanguard dataset (Machine Yearning).

In addition to basic specs, there are energy-compute metrics and hand-calc’d theoretical inference throughput for various cases. This will be expanded in future versions.

For now, let’s take a quick peek at some of the raw performance figures.

Raw Performance

On a gross basis, the gap in performance between American and Chinese accelerators is pretty stark. At a median computational throughput of 96 TFLOPS, Chinese accelerators trail American ones by a whopping 722 TFLOPS - about the equivalent of a Tesla V100, which precedes current cards by 4 generations. Chinese accelerators in the upper crust of performance beats out an A100, but not an H100 without using sparsification techniques.

Memory bandwidth speeds tell a similar story. On strictly a hardware basis, HBM restrictions hold back most Chinese chips on memory performance, though upper tier accelerators are landing somewhere around an A100 in performance. Inference acceleration toolkits are attempting to get around this problem while CXMT improves domestic HBM production quality.

Of the Chinese accelerator cards for which we have data, the best performing families seem to come from Huawei, Enflame, and MetaX. We haven’t yet found performance data on Cambricon’s latest accelerator, the 思元590, but industry adoption trends suggest this would be a top performer as well. Biren is excluded from this list as recorded performance data is from their TSMC-produced cards, which are no longer accessible since being added to the entity list.

Energy-Compute Performance

We start by looking at two primary energy-compute metrics - computational energy efficiency and memory energy efficiency. These are measured in FLOPS / Watt and bytes per joule, respectively. Both of these metrics give an indication for what kind of performance we can expect in a limited energy envelope.

The objective for energy-compute practitioners is to maximize tokens per joule (subject to quality and throughput thresholds). AI accelerator cards don’t need to have the absolute best stats - beyond a certain threshold of size, speed, and power, they just need to be good enough to handle industry standard models. Hence the Chinese ecosystem’s convergence on DeepSeek / UE8M0 FP8 precision as a newly established model performance standard.17

So, with these data, we can hand-calc tokens-per-joule by calculating the number of operations required to run a specific model (in this case, Llama 3.3 8B) at a specific precision (FP16, so 2 bytes per parameter). After determining whether the operation is compute-bound or memory-bound, we can solve for the time it takes to generate a single token during the decode sequence (seconds / token), and flip it to get the peak theoretical throughput (tokens / second). Based on the hardware’s energy consumption, we finally solve for tokens / joule.18

Alas, when translated into production workloads, both advantages compound for American chips. In a theoretical 8B parameter model inference run, American accelerators can hit around 200 tokens / second. Most Chinese accelerators can only hit about a quarter of that.

The gap in energy efficiency between American and Chinese chips is narrower than the raw performance gap, but still there. Across the board, American chips yield about 58% more tokens per joule than Chinese chips do. Peak throughput significantly affects this figure - time is energy, after all.

Interestingly enough, of the American chip line the NVIDIA H20 is the most energy-efficient at 1.1 tokens / joule, even beating out the Blackwell lineup at 0.95. The Ascend 910C is China’s most energy efficient chip at 0.86 tokens / joule, with the exception of Moffett AI’s SparseOne S30 (which natively supports up to 32x sparsity), and yields a whopping maximum of 2.72 tokens / joule. Aside from Moffett, nearly all GPUs and TPUs in this dataset are stuck underneath an efficiency frontier of ~1 token / joule.

If the Ascend 910D performs as rumored - up to 900 TFLOPS (assuming at FP16)19, 4,800 GBps of memory bandwidth in a 350W package - this would push its peak throughput to ~333 tokens/sec at an energy efficiency rating of 1.27 tokens / joule.20 That would place it in the 75th percentile of American chip inference performance while also making it the most energy efficient chip on the market, without sparsity.

FLOPS Are (Not) All You Need

Ending the analysis here would be pretty unsatisfactory. Rumors on the Ascend 910D are fine, but we’re concerned with discerning the ability for China to domestically serve its inference needs today. We can’t let perfect - or in this case, leading edge - be the enemy of “good enough.”

Why don’t I care as much about raw FLOPS? Well for one, it wouldn’t be as fun an analysis if I did, and two, that’s a misleading indicator for the overall utility of an accelerator card. Hardware is just one piece of the puzzle.

Past a certain point, we do not care if an accelerator card is best-in-class. We care only if it gets the job done.

I already know this statement is going to ruffle some feathers. It would sound like cope coming from a Chinese commentator. It’s well-understood that most developers would prefer to use the best chips and software available to them. But since the US seems keen on alienating NVIDIA from Chinese buyers as much as possible (Secretary Lutnick’s insistence on “addicting” Chinese engineers to America’s technology stack is just ridiculously Opium War-coded… it’s hard to believe that isn’t somewhat intentional21), we might as well try to figure out what constitutes a passable domestic replacement.

With this in mind, how do we determine what is considered “good enough” for the Chinese AI ecosystem?

Industry Certifications

Without applying some guardrails to it, that could be an entirely vibes-based debate. We could set an arbitrary threshold like NVIDIA A100-grade, but this still feels loose.

One of the takeaways Paul and I had from WAIC was the universal convergence on DeepSeek R1 671B as the effective “Wintel” standard for the AI era. This time around, instead of “Intel Inside” stickers, WAIC booths for vendors like SOPHGO, Enflame, and Moore Threads all featured all-in-one systems (supernodes) which they loudly proclaimed capable of running “full-blooded” DeepSeek R1, that is, the single-precision 671B parameter model. Seriously, the branding was everywhere.

When these chips costs tens of thousands of RMB per card, should unwitting procurement teams simply take these vendors at their word? This is a major question that even American neoclouds are grappling with - not just whether the chips work, but whether I can make a profit by hosting them.

Luckily, a third-party certification ecosystem is emerging to provide clarity. The China Academy of Information and Communications Technology (CAICT), a subordinate of the Ministry of Industry and Information Technology (MIIT), is in the business of conducting third-party benchmarking, assessments, and certifications for all kinds of hardware and software applications.22 Recently, it seems they’ve begun issuing certificates for “AI Chip and Large Model Adaptation Tests,” verifying that applicant vendors’ hardware can passably run inference for DeepSeek R1 671B.

This is useful for a few reasons:

1. It confirms beyond a reasonable doubt that DeepSeek R1 isn’t just a marketing push, it’s actually being adopted into industry baseline benchmarking

2. It contributes to our understanding of how the private and public sector in China can provide operational clarity to both vendors and purchasers

3. It gives us a concrete “good enough” threshold to conduct our own assessment

SOPHGO SC11-FP300 (Issued 2025-06-26)

The SC11 FP300 is SOPHGO’s latest TPU chip - the BM1690 - in a PCIe form factor. It uses LPDDR5x memory, and is much more optimized for energy efficiency than a GPGPU. This emphasis on energy efficiency is part of SOPHGO’s heritage as Bitmain’s original ASIC business for cryptocurrency mining.

Unlike the MTT S4000, the SC11 FP300 is confirmed to work with FP8 precision workloads - very important for future training and inference needs.

Advanced Computing Products

AI Chip and Large Model Adaptation

Test Certificate

Xiamen Suanneng Technology Co., Ltd.

Room 702-01, Xinghui Building, No. 9 Zengcuo'an North Road, Software Park, Xiamen Torch High-tech Zone

Product Name: Suanneng SC11 FP300 Computing Card

Product Model: SOPHON SC11 FP300

After testing by the China Academy of Information and Communications Technology, your SOPHON SC11 FP300 computing card has passed.

"AI Chip and Large Model Adaptability Passability Assessment Software and Hardware Environment and Test Details" (FT-L06-0196-01)

This certificate specifies the passability adaptation requirements for inference scenarios. The adapted large model is the DeepSeek-R1 671B large model developed by Hangzhou DeepSeek Artificial Intelligence Basic Technology Research Co., Ltd.

This certificate is hereby issued.

Moore Threads MTT S4000 (Issued 2025-04-30)

The MTT S4000 is the latest data center card in the Moore Threads lineup. It’s a general-purpose GPU (GPGPU) that uses the ChunXiao chip architecture, Moore Threads’ 2nd gen chip.

It comes with 48GB of GDDR6 memory, the same kind you’ll usually find in NVIDIA consumer GPUs. In addition to the KUAE, Moore Threads had an OAM module and edge device on display at their booth.

Source: Sina Weibo.

Advanced Computing Products

AI Chip and Large Model Adaptation

Test Certificate

MooreThread Intelligent Technology (Beijing) Co., Ltd., Building 3, Wangjing International R&D Park, Chaoyang District, Beijing. Product Name: MTT S4000 Training-Initialization Computing Card

Product Model: MooreThread MTT S4000

After testing by the China Academy of Information and Communications Technology, your company's training-initialization computing card, the MooreThread MTT S4000, has passed the inference scenario passability requirements of the "AI Chip and Large Model Adaptability Passability Assessment Software and Hardware Environment and Test Details" (FT-L06-0196-01). The adapted large model is the DeepSeek-R1 671B developed by Hangzhou DeepSeek Artificial Intelligence Basic Technology Research Co., Ltd.

This certificate is hereby issued.

As the rest of the Four Little Dragons seek a public offering, I expect we’ll see similar CAICT certifications as a positive signal for commercial adoption in the IPO process.

Calculating “Good Enough” Performance

The MTT S4000 and SC11-FP300 aren’t the best chips in our domestic chipmaker inventory, but they do provide useful examples to determine our “good enough” threshold. We can update this prior as new CAICT certifications become public.

Strictly on an energy-compute basis, both chips still fall short of NVIDIA’s A100 series from 5 years ago. But they offer decent enough performance to warrant a closer look.

First, we need to assess what takes longer: processing the entire prompt, or moving the weights from memory to the logic chip for processing. This determines whether the prefill stage is compute-bound or memory bound, respectively. We take the greater of those two values (in seconds), and add that to the total time it takes to decode our output sequence. The decode stage is almost always memory-bound.

From this limited dataset, it seems that CAICT deems peak throughput of ~47.6 tokens / sec passable for an 8B parameter model on a single card. Keep in mind that all tested hardware come in multi-card server node configurations - it’s likely the CAICT tests were conducted on those systems rather than single-card inference.

Nearly 13 seconds of inference time for an 8B model is a little long for my taste, and even a 3-generation old card like the A100 can do that in half the time at much better energy efficiency. Regardless, this is as good a starting point as any.

We’ll therefore set our passability threshold, as suggested by CAICT, for FP16 inference on an 8B model at:

With those requirements in place, we can now conduct a more informed analysis than just raw performance specs would have afforded.

Technical Analysis

As a refresher, the triple-product advantage is the compounding effect that moderate gains in individual vectors (energy efficiency, throughput speed, model quality) create when multiplied together.

This means that even if hardware is only “good enough” on efficiency and throughput speed, collaborative innovations in distributed inference, sparse memory operations, mixed precision, and so on can reduce computational complexity to fit within those limitations. Not to mention having plenty of surplus power to support initial energy footprints.

While NVIDIA and AMD are undoubtedly pushing out high-quality hardware, these systems have been stuck on the same iso-efficiency frontier: ~0.52 (± 0.21) tokens / joule since 2016. Newer systems boast incredibly powerful computational performance and boosted memory speeds, but without leveling off power draw for new chips, their energy efficiency hasn’t improved that much.

If an ecosystem is successfully co-designing models, hardware, and inference environments, then native participants of that ecosystem stand to benefit the most. Put another way, innovation is happening on all fronts.

1. Product Performance

Standouts: Huawei, Enflame, MetaX, Moffett AI.

We’ll start by plotting the SKUs of all vendors with available data on a throughput vs. power scatterplot. This highlights both accelerator performance and energy efficiency characteristics.

As a reminder, our “good enough” energy-compute thresholds for Llama 3.3 8B inference in FP16 precision are:

• Throughput: τ_sec >= 47.6 tokens / sec

• Energy Efficiency: η >= 0.14 tokens / joule

Readers may also remember the “Ubiquitous Edge Intelligence” milestone (the purple zone) discussed at WAIC 2025. This is the north star for domestic model-chip co-designs, defined as the following:

• Throughput: τ_sec >= 100 tokens / sec

• Energy Footprint: E < 20 watts

• Energy Efficiency: η >= 20 tokens / joule

Applying those restraints leaves 10 of 16 vendors having at least one SKU which makes the cut. Insufficient information on Denglin Technology, Zhonghao Xinying, Innostar Semiconductor, and T-HEAD prevents placing them on this graph, though Innostar and T-HEAD’s approaches to on-chip memory would likely place them in an energy-efficient tier.

However, Biren and Cambricon’s passable products are from an older generation of chips, back before they were added to the U.S. Entity List in 2022. Both have since shifted wafer production to SMIC, and performance data has been harder to come by for their newer lineups. So for fairness, we should remove these from the list. This doesn’t mean that newer chips powering Biren’s 壁砾166 lineup or Cambricon’s 思元590 series aren’t passable - we just don’t have the data for that yet.

That leaves 8 / 12 companies - Huawei, Enflame, Moore Threads, MetaX, Hygon, Iluvatar CoreX, SOPHGO, and Moffett AI - still in play.

Speaking to the considerable performance gaps, most vendors fall out of favor if we raise the passability threshold to within striking distance of an A100. Of the 39 original accelerator cards we have data for, only 4 remain: the Huawei Ascend 910C, Enflame 云燧T20, MetaX 曦云C500, and the Moffett AI SparseOne series.

I’m including Moffett AI in the mix as hardware-specific innovations have enabled high sparsification factors of up to 32x which, on <250W form factors, yield incredibly energy efficient computation. While the 32x case is visible, even 16x and 8x are sufficient to meet the CAICT bar.

Notably, MetaX is the only general-purpose GPU (GPGPU) in this list. The others are ASICs!

Memory Comparison

Note that with the exception of Moore Threads, SOPHGO, and Moffett AI, nearly all accelerators seem to require at least HBM2E-tier memory to make this list.

Most domestic chip designers use HBM2E or earlier, with some using GDDR6 or LPDDR5x memory. Cambricon, Enflame, Biren, MetaX, and Iluvatar CoreX all use HBM2E in their latest chips, with Enflame’s upcoming 邃思L600 rumored to be an early candidate for HBM3.23 After its initial entity listing and until its 思元590 chip (2022-2024), Cambricon used LPDDR5 memory, as does Moffett AI (LPDDR4x) for its low-powered SparseOne cards. Finally, Kunlunxin and Moore Threads all use GDDR6 memory.

The thermals, tradeoffs, and physical limitations of memory tech will be the subject of a future post. For now, it’s safe to assume that domestic chip designers will eventually have to align with a sovereign domestic supply chain for memory chips, which at the moment, remains a bottleneck to more advanced chip production.24

Sparse Computing

Critical to Moffett AI’s outperformance is its approach to sparsity. In neuroscience, it’s widely understood that human wetware leverages a high degree of sparsity in computations - in fact, less than 2% of neurons actually fire for any given input. This is a major contributor to the human brain’s considerable computational energy efficiency.

“Sparsity” in AI workloads refers to the proportion of zeroes (or near-zero) values in a neural network’s parameters or activations - the more zeroes, the fewer operations and data to have to work with. At most, typical GPUs like NVIDIA’s lineup can only natively support 50% (2x) sparsity.

Sparsity is represented in either percentages or multiples (e.g. 50% of weights set to zero = 2x sparsity, 87.5% = 8x, 96.9% = 32x). Some research has shown that LLMs can be sparsified to high degrees (often 50-90% sparsity) with minimal loss in accuracy when done properly.

A few years ago, Ian Chen and Zhibin Xiao, co-founders of Moffett, published a paper outlining their approach to this problem. The Antoum chip forms the core of their hardware lineup, the SparseOne cards, which are capable of up to 32x sparsification. While there is an inherent accuracy-speed tradeoff with sparsification (you are losing some information after all), the team found considerable speedups in throughput (and therefore, energy efficiency) with minimal accuracy loss.

The aforementioned S4 in their paper only draws 70W of power. The SparseOne S30, a 250W variant, previously demonstrated production-grade performance on large 100B+ parameter models like BLOOM-176B - 432 tokens / second in an 8-card deployment.25

Moffett AI has also repeatedly participated in MLCommons’ MLPerf Inference benchmarks, receiving consistently high marks for its SparseOne lineup. The S30 in particular nearly doubled H100 throughput on common LLM workloads on a 2.8x smaller energy footprint, yielding a ~5x boost in energy efficiency with virtually no loss in output quality.26

Finally, that Moffett’s Antoum cards were deployed on nodes from reputable OEMs (H3C and Inspur) demonstrated minimal or limited customizations required to run these hardware, hinting at commercial viability. Inspur is also an investor in the company.27

Future Moffett cards are confirmed to support FP8 precision workloads.28

2. Developer Adoption

Standouts: Cambricon, Moore Threads, Huawei, Enflame, MetaX.

Laggards: Iluvatar CoreX, Biren.

Before you ask, I’m not going to give a definitive answer on if and when NVIDIA’s CUDA moat gives way to domestic innovation. But I’ll highlight some contributing factors we can track.

Hardware isn’t useful if developers hate having to work with it. During our China trip, we spoke to a host of Chinese cloud providers asking about developer adoption of these various SDKs. Universally, preferences for CUDA (and even ROCm) remain over domestic alternatives thanks to their much larger extant developer and troubleshooting communities - a valuable moat to have.

Domestic players recognize this and attempting one of three paths:

1. Seek CUDA compatibility. Suiyuan plans to invest 20% of its IPO fundraising to develop a CUDA-compatible tool chain, with a goal of achieving 90% operator compatibility by 2025, and the migration cost will be reduced to 40 man hours per person. MetaX has also announced that its C500 series (and all future cards) would be CUDA compatible. Kunlunxin’s lineup is also reportedly CUDA compatible.

2. Press advantages into software. Huawei recently announced it would be open-sourcing CANN to help build up its developer ecosystem. Older players like Iluvatar CoreX and Biren are pushing their own custom programming languages, but may be facing adoption headwinds as a result.

3. Build out transcompilation libraries. Both Cambricon and Moore Threads have invested considerable resources into their in-house transcompilation libraries, QiMeng-Xpiler and MUSIFY, which convert CUDA into their native programming languages (BANG C and MUSA).

Programming Stacks & CUDA Compatibility Roadmaps

29

Of course, Heavyweights don’t have to worry about developer adoption, since they have the cloud real estate to justify significant hardware R&D investments. Third-party players, like the little dragons, have to partner with hyperscalers in some capacity if they want to see widespread adoption, and can grease the wheel with CUDA compatibility. But Huawei wants to establish itself as the de facto leader of the sovereign semiconductor ecosystem, and to do that, they need developers - lots of developers - to use CANN.

During this transitional stage, an “abstraction layer” market seems to exist for building transcompilation libraries, converting code written in CUDA into hardware-specific accelerator code. NVIDIA obviously wants to restrict this as much as possible, and has banned such translation layers in its licensing terms since 2021.30

Nonetheless, CUDA optimization and translation remains a big market nonetheless. Readers may remember Sakana AI (Japan’s sovereign research lab) announced, retracted, and re-launched its own CUDA engineer,31 and Y Combinator featured a similar request in its Spring 2025 “Request for Startups.”32 In China, Xcore Sigma (中科加禾) is one such example.33

The challenge with this approach is that third-party optimization libraries will always be a few steps behind first-party equivalents. Without a tight relationship between the software and hardware designers, it takes time for optimizations against those changes to diffuse.

Naturally, this routes highly competitive developers and downstream markets towards natively supported solutions that have the most up-to-date acceleration. This is why for domestic chip designers, reliable transcompilation between common libraries and custom hardware is a must-have. That’s basic network effects.

Moore Threads - MUSIFY

As an example, Moore Threads is promoting its MUSIFY toolkit to convert CUDA code into MUSA (Moore Threads Unified System Architecture), its native programming language. If it works, this would significantly reduce the technical barriers and time costs of switching platforms. Users speculate that it works similarly to ZLUDA, which translates PTX code at runtime.34

Cambricon - Transcompilation

A joint team between Cambricon (including the two co-founders) and ICT-CAS co-developed a transpiler called QiMeng-Xpiler, which converts between more commonly supported libraries (NVIDIA CUDA, AMD HIP, and Intel VNNI) and BANG C, Cambricon’s proprietary C-like language. It can handle these transcompilations with an average accuracy of 95% (admittedly weighed down by Intel), which outperforms both traditional rules-based methods and AI-native approaches.

Source: USENIX, Cambricon.

The last thing that developers want to do is debug, hence the preference for well-supported languages like CUDA. Plus, transcompilation can take hours - 3.7 on average - which is a long time to wait before realizing your code is riddled with errors. Even a few percentage points difference in final compilation accuracy can mean hours, if not days, of manual debugging.

Fortunately for Cambricon, while QiMeng failed to generate a functional-program the first time for its most challenging operation in this study (Deformable Attention, ~200 lines of code) around, it didn’t require long for programmers to debug it: half an hour for “senior coders” (software engineers), and 3 hours for “junior coders” (masters students). Plus the original transcompilation time of 4.5 hours, that’s about a day’s work compared to a week’s work of manual transcompilation - a ~20-30x productivity improvement when deployed alongside professional software engineers.

Strategic Analysis

Beyond individual product performance, there are qualitative factors to examine: strategic and financial backers, leadership teams, and commercial track records all contribute to relative standing. But the backdrop for much of this is the fallout and catalytic effects that American export controls have had on the ecosystem. To paint the picture, we’ll start with the entity list impact.

1. Entity List Impact

While being named to the U.S. Entity List has hit some first movers like Biren hard, for others they have been more speed bumps than Great Walls.

Second mover advantages for the other three dragons - Enflame, Moore Threads, and MetaX - seem to exist, as they’ve had an easier time finding commercial adoption from large hyperscalers.

Specifically, HBM is a critical bottleneck holding back domestic chip performance… for now. Time will tell how quickly CXMT / YMTC can ramp up HBM3+ production. Heterogeneous computing and software-first innovations are beginning to evade these bottlenecks somewhat successfully.

For now, poorer SMIC yields (~40%) are contributing to higher unit costs for domestic chips than NVIDIA-produced chips (TSMC at 80-90% yield depending on process node). However, as this improves, unit costs for domestic chips will systemically improve.

Leadership Turmoil: Biren

Founded in 2019, while Biren wasn’t the first domestic GPU company, it was widely considered to be an early darling.

The founder & CEO, Zhang Wen (张文), is the former president of SenseTime and holds a JD from Harvard University + an MBA from Columbia University.35 Notably, he does not come from a technical background, but he is an excellent dealmaker and headhunter: his extensive experience enabled him to assemble an Avengers-class team of founders and executives from Huawei, Alibaba, Qualcomm, and more.36 On the team alone, Biren was able to secure significant amounts of capital - nearly $700M USD and establish early partnerships with other hyperscalers.37

In 2021, Zhang added another heavyweight to the roster - Li Xinrong (李新荣), former AMD executive and its head of China R&D. Curiously, he was added as “Co-CEO,” likely to reinforce the technical leadership acumen Zhang Wen lacked in the executive role.38

Biren debuted its flagship chip, the BR100, in 2022 at Hot Chips 34 and positioned it as a domestic alternative to NVIDIA’s A100/H100 class, emphasizing strong BF16/INT8 throughput, flexible precision handling, and speedy interconnect bandwidth. This was objectively an impressive chip at the time, eclipsing the A100 (released just a year earlier) in several performance metrics.39

However, that conference may have attracted unwanted attention from Biden-era policymakers. Months later, Biren would be added to the U.S. Entity List, starving it from access to TSMC process nodes (and later, HBM from SK Hynix / Samsung) for fabrication.40 Zhang would also go on to lose both his co-founders - Xu Lingjie (徐凌杰) and Jiao Guofang (焦国方) - in the ensuing fallout. Biren has yet to recover its pole position in the domestic market.

There’s a Chinese idiom 树大招风 (shù dà zhāo fēng, “a tall tree attracts the wind”) which captures this problem. This is also probably why most Chinese chip designers are incredibly tight-lipped about performance specs for their product lineups.

Memory Bottlenecking: Moore Threads

Being named to the U.S. Entity List hasn’t helped Moore Threads’ HBM prospects, either - entity-listed or sanctioned companies are typically barred from using HBM products from SK Hynix, Samsung, and Micron, the international memory triumvirate.41 This means their primary options for domestic memory IDMs are Yangtze Memory Technologies Corp (YMTC) and ChangXin Memory Technologies (CXMT).

The majority of leading edge chips from NVIDIA and AMD use HBM3E memory, which is more advanced than the memory types that sanctioned domestic chipmakers have access to. More specifically, HBM3E achieves total package memory bandwidth speeds of over 1.2 terabytes per second (TBps) per memory stack, roughly 3x faster than HBM2E. Advanced cards use multiple stacks of HBM - 4, 6, or even up to 12.

Moore Threads uses GDDR6, rated at or around 512 GB/s. GDDR is the kind of memory you’d find in prosumer gaming GPUs - still good quality, but pales in comparison to HBM.

Source: Pacific Securities, CSDN.

Reportedly, CXMT has already been producing HBM2 since mid-2024, and has begun testing HBM3 with select industry partners.42 They expect to enter mass production for HBM3 and HBM3E grade memory chips in 2026-2027. Interestingly, YMTC - which has been producing stacked NAND chips for years - seems to be reaching across the aisle to assist CXMT with hybrid bonding techniques for more reliable stacking and thermal dissipation.43

It could also be the case that since Moore Threads manufactures both consumer gaming GPUs and datacenter cards alike, the decision to use GDDR6 rather than HBM2 may be to simplify its supply chain. Undoubtedly GDDR has pretty steep tradeoffs in comparison. Whatever the case may be, entity listing has definitely impacted the competitiveness of Moore Threads cards against non-sanctioned domestic and foreign competitors.

For more on HBM, I strongly recommend Ray Wang’s guest piece on Nomad Semi.

Nomad Semi

Deep Dive on HBM

This deep dive on HBM is written jointly with Ray Wang…

Read more

a year ago · 123 likes · 10 comments · Moore Morris and Ray Wang

Lower Yields, Higher Unit Costs

One area where the Entity List is having a measurable systemic impact is domestic chip unit costs. Sometimes, Chinese companies will list more granular cost of goods sold for specific line items in their reporting materials. For our analysis, we have a useful comparison: two Little Dragons, one entity-listed (Moore Threads) and one not (MetaX).

To proxy our deployment costs, $ / TFLOPS is a helpful metric for training, but less informative for inference. We’ll use a dollar cost ratio of memory bandwidth speeds.

The H100 edges out the C500 in memory bandwidth ROI by a small margin, owing to its more advanced HBM3 memory stack vs. the C500’s HBM2E, though the C500 costs ~28% less to produce than the H100.44

Despite lower performance stats, the MTT data center card lineup costs more than 2.5x as much as an H100, and 11x as much on a $ / GB/s basis.

Since Moore Threads is entity-listed (and MetaX is not), its chips - despite lower performance - cost considerably more than MetaX’s competing chip line (2.5x gross, 11x worse ROI). That puts Moore Threads at a severe disadvantage on hardware alone.

This is an illustrative impact of entity listing on available process nodes to a downstream customer. Yield rates - the number of successful dies cut from a wafer divided by total potential dies - amortize the cost of wafer production over all resulting chips sold. It’s generally understood that for more advanced process nodes, SMIC yields are currently between 30-40% compared to 80-90% at TSMC. All else being equal, TSMC is able to spread its wafer production costs over twice as many chips sold as SMIC can.45

Source: Bismarck Analysis, Granite Firm, SemiWiki

A deeper-dive on the domestic logic and memory chip fabrication supply chain will be the subject of a future post.

Unintended Consequences of Myopia (UCM)

Undeniably, the lack of leading-edge HBM from foreign suppliers is hindering domestic chip competitiveness. However, there are ways around this issue. In a dramatic reveal on August 12 2025, Huawei revealed a new software tool called the Unified Cache Manager (UCM) as a way to accelerate training and inference workloads without access to HBM.46

Specifically, UCM builds a three-tiered storage architecture to split up the KV cache of LLMs across different memory types:

• HBM for extremely common data, accessed in real-time and at high frequency

• DRAM for a balanced approach, storing data with moderate frequency

• SSDs for low-frequency data. This evades the bottleneck of VRAM capacity in a chip, and accesses much cheaper, more readily available storage for low-frequency values.

This would unlock astoundingly long context windows without the need for bigger HBM - critical for reasoning models with ballooning token volumes, and overcoming the HBM denial hurdle while CXMT ramps up production.47

The net effect is a triple-purpose optimization across performance, energy, and financial costs:

1. First token latency is reduced by up to 90%

2. Sharding KV cache across multiple memory types 10x’s the inference context window

3. Intelligent routing of data “heat” improves tokens-per-second by 2-22x

Huawei intends to open-source UCM this month, which would provide a systemic boost to all domestic chip companies currently gated by HBM restrictions. While benefits on single cards may be nominal, supernodes (multiple-card clusters) with multiple memory types (SSDs, DRAM, SRAM) would see substantial benefits. Not to mention cloud-scale deployments.

Heterogeneous Computing

In the same way that UCM leverages relative strengths of different memory types, heterogeneous computing optimizes and distributed workloads according to the relative strengths of different chips.

Paul Triolo and I have written about advancements in heterogeneous computing in the past. I’ll simply restate that there are large-scale projects sponsored by major hyperscalers and research labs achieving better training and inference performance through heterogeneous (multiple-vendor) computing clusters than homogenous ones.

AIStackDecrypted

China AI update: Innovation across a distributed and heterogeneous AI stack heats up

After a two-week swing through the Chinese AI sector in late July that included detailed discussions with Chinese AI players at the World AI Conference (WAIC) and visits to many individual companies, plus discussions with domestic and international investors, I was struck by the dynamism of the companies involved, and the out-of-the-box thinking on AI m…

Read more

8 months ago · 16 likes · 1 comment · Paul Triolo and Ryan Cunningham

This is a key part of the Shanghai municipal government’s “AI+Manufacturing” plan - building a low-latency, distributed industrial “smart compute cloud.” Sugon - the supercomputing company recently acquired by Hygon - also recently announced a high-profile partnership with 20 other hardware companies, OEMs, and research labs, with the intent to develop and promote large-scale heterogeneous computing systems.48 Sugon has already launched a new supercluster product to demonstrate these capabilities.49

Source: Sugon WeChat channel.

“Compared with closed systems, Sugon AI super cluster system not only works as efficiently as a single computer through its tightly coupled design, but also supports multi-brand AI accelerator cards and is compatible with mainstream software ecosystems such as CUDA, providing users with more open choices and significantly reducing hardware costs and software development and adaptation costs, thus protecting initial investments.”

That excerpt from the Sugon announcement is critical. Functional heterogeneity should not just be considered in the context of entity listing and trade wars. It is an inherently anti-fragile economic paradigm that will reduce unit costs for training and inference workloads overtime.

2. Strategic Backing

Standouts: Huawei, T-HEAD, Enflame, Moore Threads

Laggards: Iluvatar CoreX, Biren

Losers: Jingjia Micro, Denglin Technology

The “Made in China 2025” industrial policy is often cited as the blueprint for state-led “winner-picking” in high-tech sectors, semiconductors among them. In this limited analysis, that shorthand does not hold up.

First, many of these players share the same funding entities on the cap table (like the Big Fund), which eliminates any idiosyncratic advantage. Second, not all kinds of government capital in China are created equal. And third, an overly activist government presence on the executive team seems to slow things down faster than it can open doors for adoption.

Therefore it should not be taken on faith that Chinese government favorability or backing is a guarantor of success. Instead, we assign a higher weight to strategic or financial relationships with established technology incumbents. These players generally have the most sophisticated products, much faster iteration cycles, and strongest commercial adoption thus far.

Source: Crunchbase, Bloomberg, company announcements.

* Zhonghao Xinying is navigating a potentially botched attempt to go public via reverse merger with a publicly traded tire and rubber company. Source: Tempu Co., Ltd. has faced seven consecutive inquiries from the Shanghai Stock Exchange, leading to a halt in leveraged buyouts, with funds reportedly "in transit."

The Role of “Patient Capital”

“Patient capital” is a colloquial term used to describe investment funds (typically state-connected) with much longer horizons for expected returns, usually targeting sectors deemed critical for economic growth or industrial self-sufficiency. This is not to say that profitability and ROI don’t matter, just that that is second to longer-term goals.

National Team

The national team, led by the China Integrated Circuit Industry Investment Fund (“Big Fund”), focuses on strengthening national strategic industries, filling gaps in the supply chain, and anchoring private capital around key technologies.

Financial returns are not the priority. The goal is to build industrial security.

As an example, Cambricon received early funding from the Big Fund. The company originated from the Institute of Computing Technology at the Chinese Academy of Sciences, making it a textbook case of industry-academia-research integration.

The Big Fund’s investment was both a bet on a promising semiconductor startup and an endorsement of the pathway from state laboratories to commercial application, aimed at encouraging more frontier research to be translated into market-ready products that serve national strategy.

Local Funds

Local funds, by contrast, are closely tied to regional agendas. They nurture companies that place headquarters, research centers, or production facilities in their jurisdictions in order to strengthen the local industry ecosystem and reinforce the value chain.

For example, the Shanghai IC Fund’s mission is to consolidate and expand the city’s leadership in China’s semiconductor sector by building a complete industrial cluster that spans design, manufacturing, packaging and testing. They frequently act as bridge investors, supporting promising firms in early rounds before inviting the Big Fund to co-invest, creating “national plus local” synergies.

With the support of both the Shanghai and Beijing IC funds, these two cities are emerging as distinct national leaders: Shanghai as the hub of the full semiconductor value chain with champions such as SMIC (Semiconductor Manufacturing International Corporation), AMEC (Advanced Micro-Fabrication Equipment), MetaX, and Enflame.

Beijing is establishing itself as the capital of R&D and design, represented by companies like Cambricon, Moore Threads, and Hygon.

SOE Venture Arms

SOE venture arms sit somewhere in between. They pursue some of the same financial goals as commercial vc firms, but their deeper value lies in the strategic resources they provide.

Startups gain access to procurement pipelines, pilot projects, and anchor customers within the SOE system. They also benefit from credibility and political backing that smooth the path for future fundraising, bank creditline, and government project participation.

On top of that, SOE venture arms open doors to industrial expertise, supply chain partners, and even manufacturing capacity. Unlike financial investors, they are often more patient, willing to stay with companies through multiple growth cycles.

All four of the dragons have attracted both SOE venture arms and local funds, showing how deeply these funding vehicles are embedded in the country’s industrial rise. Together, national direction, local execution, SOE backing, and corporate upgrading create a layered investment ecosystem that de-risks financing while guiding capital recipients towards real use cases and sustained demand.

Drawbacks of State-Led Innovation

Patient capital is best when passive. Companies with the strongest connections to government are actually among the worst performers.

Despite a major head start, deep connections to China Electronics Technology Group Corporation (CETC), government, and PLA contract revenues, Jingjia Micro (景嘉微) has been wholly unable to transition their product line towards AI accelerators, seeing their GPU revenues completely collapse by 40-70% YoY from 2021 to present as new competitors have come online.

While Iluvatar CoreX (another early entrant) was founded by an Oracle veteran, in 2021 they appointed ​​Diao Shijing (刁石京), former Head of the Ministry of Industry and Information Technology, to the position. Though the company has had some success with their training card accelerator lineup, order volumes for shared heterogeneous computing projects are much smaller than competitors’ by an order of magnitude (100 inference cards vs 1000 Moore Threads GPUs, 2000 Huawei Ascend 910Bs, 3000 MetaX GPUs).50 It seems Iluvatar has had a slower time transitioning from R&D into widespread commercialization.

Hyperscaler Tagalongs

Based on the rate of commercial deployments, there seem to be two winning strategies for domestic chip designers: either you are the customer and you build it yourself (Huawei, T-HEAD, Kunlunxin), or you team up with an incumbent hyperscaler.

Even with advancements in heterogeneous computing technology, it seems far more difficult to go it alone without a close alliance to a major hyperscaler. Biren and Iluvatar CoreX, despite strong teams and early traction, have yet to see large-scale adoption in hyperscalar clouds. Jingjia Micro and Denglin Technology are completely absent from any major commercial rollouts.

It should come as no surprise that China’s hyperscalers - Huawei, Alibaba, Baidu, Tencent - have immense incentive to manage their cloud infrastructure costs, for the same reason that Western equivalents all have internal silicon teams (Meta: MTIA, Microsoft: Maia 100, Google: TPU, Amazon: Trainium/Inferentia). The first three have their own wholly-owned fabless design firms, while Tencent has facilitated close relationships with at least two little dragons, Enflame and Moore Threads, through several rounds of funding.51

This close relationship with Tencent is significantly paying off for Enflame, which is reporting the largest volume of commercial adoption for their S60 inference cards compared to other independent fabless firms. Even in the more hands-off approach that Tencent uses, it still leveraged Enflame’s expertise to co-design its custom Zixiao AI inference chip for Tencent Cloud.52

3. Leadership

Standouts: Huawei, Cambricon, Enflame, Moore Threads, MetaX.

Laggards: Biren (though initially promising, has suffered greatly from post-entity list turmoil).

The prevailing pattern for domestic chip leaders seems to be “硅谷基因+本土化创新”, or “Silicon Valley DNA + localized innovation.” Many of the top performers in the space have executives with deep experience in Silicon Valley companies, but have been implementing those solutions in the local ecosystem.

The phrase 技而优则管 (jì ér yōu zé guǎn, literally “those whose technical skill is excellent then manage”) accurately describes the best performing companies on this list. It more or less means “promote the best engineers to lead engineers,” and is a common saying in Chinese tech circles to insist that the person running an engineering team should be an engineer, not an MBA.53

The three best performing little dragons - Enflame, Moore Threads, and MetaX - all have 硅谷基因 in founding leadership. Enflame and MetaX have decade-plus AMD veterans with a long history of working together, and Moore Threads is effectively carved out from the NVIDIA China team.

Exceptions to this rule would be among the Chinese Heavyweights, where leadership is cultivated and promoted from within, and Cambricon. Cambricon is a unique story where the two founding brothers, both STEM prodigies, were fast-tracked into a Chinese Academy of Science-sponsored ecosystem through a program for gifted youths. Their prowess in mathematics and computer architecture, along with the CAS resources at their disposal, eventually led to the founding of Cambricon out of their research into custom accelerator hardware.

Aside from that, we note that an outsized government or ministry presence on the leadership team does not seem to be associated with better product performance (no surprise) or commercial adoption. Examples here would be Iluvatar CoreX and Jingjia Micro. While government connections may seem helpful in large-scale intelligent cluster contracts, the basis for procurement seems to be more meritocratic.

4. Commercial Adoption

Standouts: Huawei, Kunlunxin, T-HEAD, Enflame, Cambricon, Hygon

Honorable Mention: SOPHGO.

The most widely supported domestic chip designers to date include all three heavyweights (HiSilicon, Kunlunxin, T-HEAD), two of the four little dragons (Enflame, Biren), both public champions (Cambricon, Hygon) and finally Iluvatar CoreX.

The Chinese hyperscaler market is dominated by the Big Three: Huawei, Tencent, and Alibaba. Baidu AI Cloud, Kingsoft Cloud, and ByteDance (Volcengine) are also notable.54 Additionally, there are the state-owned Chinese telecommunications companies China Unicom (中国联通), China Mobile (​​中国移动​​), and China Telecom (中国电信​​) to consider.

Alibaba is the cloud computing leader in terms of market share, and provides the richest detail on supported accelerator programming languages and hardware platforms. NVIDIA and AMD-based computing instances are supported by nearly all Chinese hyperscalers - this is not the case for its domestic chip designers.

Source: Alibaba Cloud.

Until recently Chinese telco server procurement did include solutions from NVIDIA, AMD, and Intel, but the MIIT has specifically instructed the telcos to phase out foreign processor usage by 2027, accelerated by looming export controls and a push for domestic self-sufficiency. This instruction is limited to CPUs at the moment (hitting Intel and AMD), but a similar instruction for GPUs has not been ruled out.55

The below table highlights public confirmations of domestic chip adoption in high-workload commercial sectors. The heavyweights, little dragons, and public champions dominate the upper rankings. This is not all-inclusive since there is a long tail of neoclouds that are not the subject of this analysis.

In a major order from China Mobile, Huawei’s Ascend line won over 70% of the contract, and Baidu will be supplying its Kunlunxin processor line to many systems integrators in that contract.56

Enflame has reportedly shipped over 70,000 S60 inference cards to date, mostly to Tencent computing clusters. Outside of Huawei and Cambricon, this is actually the largest recorded figure of commercial shipments for the newer entrants.57

Finally, in other cloud contracts with multiple vendors, we’ve seen comparable deployment figures for MetaX, Moore Threads, and Ascend accelerator cards in the thousands, with additional (though meager) participation from Biren and Iluvatar.

Deployment breakdowns are not always available by vendor or SKU. We’ll attempt to track shipment figures in future versions of the Silicon Vanguard dataset. For now, what we generally observe is that the Heavyweights are well-positioned given their cloud computing footprint, Enflame seems to be leading the weyr of dragons, and Cambricon + Hygon are making strides.

16

Sinocities

How is China's "Eastern Data Western Compute"（东数西算) developing?

In 2018, Apple moved most of its data for China-based users to cloud computing servers in southwestern Guizhou’s Gui’an New Area. This decision reflected a reality since China’s 2016 Cybersecurity Law, that in order for foreign technology firms to operate in China they would have to store user data within the country. Apple no doubt saw this as a necess…

Read more

2 years ago · 27 likes · 5 comments · Andrew Stokols

17

AIStackDecrypted

Assessing China's support for open weight models, and the real meaning of DeepSeek's latest release

Much breathless prose has been written about China and open source or, more precisely, open weight AI models. Chinese companies now sport four of the top five or so open such models, with commentators claiming these models being free means that China will lead on AI and that the US government should support more open source/weight models. Now is the tim…

Read more

8 months ago · 10 likes · 3 comments · Paul Triolo

43

ChinaTalk

Mapping China's HBM Advances

Ray Wang is a Washington-based analyst formerly based in Taipei and Seoul. He focuses on U.S.-China economic and technological statecraft, Chinese foreign policy, and the semiconductor and AI industry in China, South Korea, and Taiwan. You can read more of his writing on his Substack…

Read more

a year ago · 35 likes · 3 comments · Ray Wang and Lily Ottinger

50

AIStackDecrypted

China AI update: Innovation across a distributed and heterogeneous AI stack heats up

After a two-week swing through the Chinese AI sector in late July that included detailed discussions with Chinese AI players at the World AI Conference (WAIC) and visits to many individual companies, plus discussions with domestic and international investors, I was struck by the dynamism of the companies involved, and the out-of-the-box thinking on AI m…

Read more

8 months ago · 16 likes · 1 comment · Paul Triolo and Ryan Cunningham

Anyway, back to Machine Yearning, where every week is infrastructure week.

Last week, Paul Triolo and I went on the Sinica Podcast with to tell Kaiser what we saw at Shanghai’s World AI Conference.

In it, I mentioned one of the keynote speeches at WAIC by Professor Wang Yu (汪玉), Chairman of the Department of Electrical Engineering at Tsinghua University, who laid out the blueprint for his philosophy towards AI infrastructure development.

I also talked about my investment philosophy - an emerging framework called “watt-to-bit” in some fringe Western investor circles.

Turns out, China’s consensus AI strategy is mapping precisely to this investment doctrine. So it’s probably time we pull back the curtain on it.

In this week’s post, I’ll cover:

1. The What: Introduce Energy-Compute Theory and its primary inputs

2. The How:

   1. Dissect Professor Wang Yu’s keynote as an application of energy-compute theory

   2. Identify implications for model and semiconductor development

3. The Why:

   1. Assess conditions for AI’s diffusion into the Chinese economy

   2. Lay out the implications for both China and US AI ecosystems

Key Takeaways to Expect

1. Energy is the constraint: Watts are the ultimate scarce resource; compute capacity and intelligence output scale with how efficiently energy is converted into tokens.

2. China’s objective function: China is explicitly reframing AI and industrial policy around energy-compute optimization, treating it as a physics-based equation rather than a vague race for “AI leadership.”

3. A first-principles playbook: Grounding strategy in watts, tokens, and physics creates a clearer, more durable framework for competing in the new economy.

4. The West’s blind spot: U.S. narratives fixate on talent, chips, or qualitative intelligence thresholds, ignoring the underlying energy-compute substrate.

5. Winning advantage: Nations and firms that master energy-compute efficiency will dominate AI capability, industrial output, and ultimately geopolitical power.

Watts and Bits

Watt-to-bit frameworks merge power generation, storage, compute, cooling, and other AI infrastructure into a single asset class: “energy-compute infrastructure.”

Maximizing ROI with energy-compute infrastructure requires optimizing models and hardware along a 2-dimensional plane of intelligence throughput (tokens-per-second) and energy inputs (power), rather than strictly deploying the best models or chips.

This is not yet a mainstream philosophy in Silicon Valley group chats. Today, most Westerners are “scale-pilled”, meaning our prevailing strategy is to throw as many resources at a compute problem as possible, assume that the model will disproportionately improve, and the earliest movers will capture most of the profits, justifying incredibly high sticker prices.1

“Resources” in this context could mean data, chips, capital, or even talent (see “The Sovereign AI Trap” in my previous post on DeepSeek).2

Common refrains of this strategy are the Blitzscaling playbooks of the preceding tech boom and Sam Altman’s famous call to action to “capture the light cone of all future value in the universe” via AGI.34 The latter is probably the most literal interpretation of the leading sector innovation strategy Jeffrey Ding argues against in his debut book, “Technology and the Rise of Great Powers.”5

I’m an accelerationist, but scale-pilled thinking is a memetic hazard. It dangerously restrains intellectual and industrial discourse to a Thucydides Trap6 where the “race to AGI” must be won by any means… without an empirical measure for what success means in the first place.

Also, it’s abundantly clear that the Chinese AI ecosystem doesn’t operate on that paradigm anyway.

Introducing Energy-Compute Theory

This isn’t a treatise or an econ lecture. It’s just a quick intro to energy-compute theory as I’ve been drafting it - an “efficiency-pilled” counter to the “scale-pilled” doctrine dominating accelerationist dialogue. It’s also a work in progress as my investment theses evolve overtime.

🫘 BeanChina vs. BeanUSA

Imagine two rival coffee shop chains, BeanChina and BeanUSA. Every day they try to deliver as many high-quality lattes as possible. Both stores have 3 levers they can pull, and are subject to a quality constraint.

1. Number of espresso machines on the counter. More machines = more cups you can produce simultaneously. It also means more energy.

2. Energy per machine. Machines with higher power ratings can generally brew faster, serving more customers.

3. Shots per bag of beans. How many shots you’re able to stretch out of a given bean quantity.

4. Taste score the barista must hit. This is your minimum flavor standard. Drinks below this quality won’t sell. Tastier drinks can be more expensive.

Throughput

The stores pursue different strategies:

• BeanUSA’s top-of-the-line H200 coffee-makers can process 4.8 units of coffee slurry per second, but the store uses a two-shot-per-lattee recipe. Its lattes are generally rated as tastier - a 9/10 on the scale. The H200 coffee-makers cost $35,000 each and have a 3 year useful life.7

• BeanChina’s H20 coffee-makers are a bit less powerful at 4.0 units per second, but they buy slightly better grinders and experiment with single-shot recipes that customers still rate at least 8/10. The H20s cost $14,000 each and have a 3 year useful life. Additionally, BeanChina has ~40% cheaper power on its side of the street. 8

Given the above, we estimate their respective throughputs at 60 cups (BeanUSA) and 100 cups (BeanChina) per-machine-hour. With cups per hour as brewing bandwidth ÷ (shots per latte):

Even with slower silicon in their coffee-makers, BeanChina can still serve ~67% more lattes per machine than BeanUSA.

Daily P&L

Now let’s see what that looks like on an income statement:

BeanChina sells ~70% more lattes, while its energy cost per cup is ~3.3× lower (0.28 cups per watt-hour vs. 0.09 cups per watt-hour), and its all-in cost per cup is roughly half of BeanUSA’s - even though the drink scores just one point lower on taste.

Corporate Innovation

BeanChina charges less per cup than BeanUSA, but makes virtually the same amount of profit.9 BeanChina invests the margin into one of three options:

• Efficiency gains. R&D for much denser bean variants. Further grinder technique refinements. At best, BeanChina now only needs 7 grams of beans per cup - and early taste tests indicate their lattes are even tastier than before.

• More machines. A linear increase to throughput.

• Hybrid strategy. A combination of greater per-cup efficiency and volume increase.

BeanUSA tries to make up for this by buying more of its H200 coffee-makers, but without changing the underlying recipe, their unit costs remain the same.

After the investments in bean R&D, BeanChina maintains its energy cost per cup advantage, while simultaneously reducing bean costs. Unit costs are now just 34% of BeanUSA’s.

In strats B and C, the two stores are still making roughly the same amount of profit, but BeanUSA had to shell out $350K in upfront capital to do it - $210K more than BeanChina’s incremental capex.

The DeepBean Price War

Finally, BeanChina’s scientists unveil a new recipe, the DeepBean-R1, which is rated as slightly tastier than BeanUSA’s most popular brew: a 9.2/10. And what’s more, DeepBean-R1 is available for just $2.25 / cup vs. BeanUSA’s $3.00.

Curious customers swamp BeanChina’s counters, and BeanUSA panics, slashing its own price to $2.25 while scrambling for a new recipe.10

BeanUSA’s daily profit drops by about $11K. BeanChina now clears $52K, a gap of $23K despite only spending 40% of BeanUSA’s capex.

🥤 Takeaways

BeanChina might have started at a disadvantage on machines and taste, but several key investments helped close the gap:

• Scarcity-driven innovation. Invested in key efficiency gains (the single-shot recipe, finer grinders) which materially lowered both their energy cost per bean, and the beans required per cup.

• Reinvestment in R&D. Overtime, investments in bean engineering yielded material improvements to taste that closed the quality gap with BeanPT-5 on a much lower cost basis.

• Energy abundance. Greater energy abundance on the China side of the street meant both lower energy costs and more power available for coffee-makers. BeanChina has plenty of runway to expand to larger store footprints.

In response, BeanUSA tried to maintain its lead by buying more coffee-makers. But starting from a higher cost basis put them at a long-term disadvantage, especially once BeanChina closed the quality gap and started a price war. BeanUSA is speeding up their recipe development cycle, but is now in a defensive position.

This is energy-compute theory in a nutshell.11

Doctrine

Energy-compute theory prioritizes the diffusion of valuable, affordable intelligence into sectors of the economy where those qualities are paramount for adoption.

• Value is determined by human-level performance (e.g. tasks completed) at or above a pre-determined quality constraint (yield, defect rate, MCAT score)

• Affordability is determined by energy efficiency (e.g. FLOPS / Watt, tokens / joule) as a proxy for cost. Importantly, we use physical energy units rather than fiat currency units to remove noise from market saturation, trade policies, or general animal spirits.

Empiricism

In plain English, your objective is to maximize valuable intelligence per energy unit, subject to a minimum IQ score and token throughput speed.

Here’s how we capture those values and constraints:

Where, for a given combination of model(s), chip(s), and cooling infrastructure:

• E = compute budget in joules (e.g. 1 megawatt-hour = 3.6 x 10^9 joules)

• eta = token efficiency (tokens per joule). This is a combination metric which is based on the specs of your model and the chip it’s deployed on.

• S = your time conversion factor, in this case seconds to days

• Q = model intelligence (e.g. MMLU, HLE, GPQA-diamond scores)

• Q_min = model intelligence threshold (defined as GPT-4o equivalent, DeepSeek-V3 equivalent, Llama 2 7B equivalent, etc. on some task-defined benchmark)

• tau_sec = real-time token throughput (tokens per second)

• tau_sec,min = Some throughput SLA (e.g. median 100 tokens / second in production)

Let’s break out each of those pieces.

Power (E)

This one is easy. What is your total budget of power over time (watt-hours, joules) for your compute load? This can be scaled up to a national level (a country’s total datacenter terawatt-hours per year)12, down to individual units (a Tesla Optimus 2.3kWh battery)13, or even at Kardashev scales (Type II civilizations = 4 x 10^26 watts, the total luminosity of our sun).14

Efficiency (eta)

Efficiency is derived from calculating token throughput of a given model-chip combination (tokens / second), subject to the total envelope of power for the hardware and cooling infrastructure. Using our coffee example, this is a combination of the coffee slurry your machine can process per second, and the beans required per cup.

Why does this matter? If you have a limited power budget - whether for edge robotics or a single data center - your output potential is governed by your token throughput on that budget. That could be revenue and breakeven prices if you’re a neocloud, or task completions if you’re a robot.

Fortunately, all model-chip deployment combinations can derive these values, provided you have the requisite specs. A recent survey from researchers at Shanghai Jiao Tong University found a range of energy efficiency values for different deployment combinations of 7B models, from 0.0167 tokens/J at the low end to as high as 46.66 tokens/J.15

Those highest values, deployed on PIM/NDP hardware, are no slouches on throughput speed either, achieving 481 to 1998 tokens/sec on a modest 12 to 43 watt power budget - see below.

You’ll notice that clusters of different hardware types - FPGAs, ASICs, PIM/NDP - are shifting upwards and to the left on this chart, compared to the left-skewed distribution of GPU values.

There are physical limits to how much power can be deployed into a single chip governed by its thermal resistance and offtake from cooling equipment. Since nearly all energy into a chip is converted into heat, past a certain point, the chip will literally melt.16

So yielding better performance-per-chip as we approach that thermal wall is of critical importance.

Investing in different combinations of model and chip designs can greatly influence the affordability of your energy-compute infrastructure.

Intelligence (Q, Qmin)

But suppose you didn’t want just a 7B parameter model. Suppose you wanted GPT-4o or DeepSeek-V3, with hundreds of billions of parameters?

The above chart keeps model intelligence (Q) constant but varies the hardware to create efficiency frontiers. All else equal, larger and more intelligent models require more storage and more power to run, so you can’t just compare efficiency values - they must be subject to IQ constraints.

A small model might be fine for some tasks, and insufficient for others. This depends entirely on the context in which the model is deployed. That context determines your Qmin, the minimum acceptable IQ threshold.

Generally speaking, Qmin negatively impacts energy efficiency. Memory footprints for larger models influence the size of the chips required to store them (more VRAM), and the computational load of inference operations (arithmetic intensity, or operations-per-byte).

Model architectural and inference advancements like sparsity, quantization, fast decoding, and more can significantly reduce your memory footprint and arithmetic intensity, letting you compress more capable models into smaller, more energy-efficient footprints.

Investments in all of these areas influence the value of your energy-compute infrastructure.

Constraints (tau_sec, tau_sec,min)

Human conversation and reading speeds typically fall between 4 and 8 tokens per second.17 That might be enough for basic chat, but it’s not a material improvement for productivity use cases. This is even more pronounced with reasoning models, which spend 10x more tokens “thinking” about their response before issuing a verdict.18

If an inference deployment wants the model to be considered useful, especially if the target for time between chat turns is between 240ms and 760ms, you need to significantly overshoot typical reading speeds.19 Artificial Analysis pegs some of the top performing API endpoints at a median of 100 to 500 tokens / second or greater.20

Furthermore, the longer your model takes to generate a response, the more energy (watt-seconds, joules) is being used in that response.

These constraints jointly govern perceived value and affordability of your energy-compute infrastructure.

An Efficiency-Pilled Ecosystem

With that summed up, let’s jump in to how the China AI ecosystem seems to be applying energy-compute theory at scale. The keynote speeches for the Shanghai State-Owned Capital Investment (SSCI) forum were live-streamed via the conference mobile app, so I was able to get the raw files, transcribe, and translate them while I was in-country.

What follows isn’t a one-for-one translation of the slide, nor is it all the slides. It’s just a narrative voiceover of what’s being discussed. Some of my translations might be off, so please correct me if I’m way off base on something.

China’s new objective function

In his keynote, and supported by several other speakers in the SSCI (Shanghai State-Owned Capital Investment) forum, Prof. Yu lays out his vision for breaking free of one-dimensional thinking - reframing the goal from “race to AGI” to “ubiquitous edge intelligence for applications” (泛端侧智能应用 - fàn duān cè zhìnéng yìngyòng).21

To achieve this, Prof. Yu calls for merging intelligence capability and accessibility into a single objective function: maximizing inference energy efficiency subject to a given IQ threshold. That IQ constraint guarantees we are focused on what Prof. Yu calls “high quality (高质量 - gāo zhìliàng) tokens.”

Prof. Yu explains that typical energy efficiency metrics for hardware - like FLOPS per watt, or OPS per watt - do not appropriately capture this tradeoff.

• In the AI 1.0 era, speech, images, and text were stored in various kinds of data formats, which made comparative performance for hardware and models difficult to measure

• In the AI 2.0 era however, newer multi-modal models are capable of converting all kinds of data inputs into tokens, then reformulating the problem into a next-token-prediction challenge

Prof. Yu calls the token the “core production factor” in the era of artificial intelligence. (核心的生产要素 - héxīn de shēngchǎn yàosù).

• Tokens are more generalizable than OPS for modern machine learning applications

• They can be used to represent all kinds of encoded information - pixels, code, language, conditional motion sequences (see Waymo)22, and so on

As the core production factor, developers care greatly about our systems’ ability to generate these quickly and efficiently.

Generating tokens requires energy. So let’s next assess our energy efficiency requirements (能效需求 - néngxiào xūqiú) across different intelligence levels.

According to Prof. Yu, he pre-defines target requirement zones (需求区域 - xūqiú qūyù) in green polygons for efficiency targets in each intelligence level.

1. Level 1 Intelligence: Intelligent chatbots (智能对话助手 - zhìnéng duìhuà zhùshǒu). Some systems have been able to break the target zone, hitting 10 tokens / joule. So we’re making progress there.

2. Level 2 Intelligence: Logical question-answering (逻辑问题解答 - luójí wèntí jiědá). No systems, which include various configurations of Qwen-3 and DeepSeek-R1 models (the text is hard to make out) at this point have been able to breach the target eta of >10 tokens / joule. In fact, we’re actually 1 to 2 orders of magnitude outside the target area.

3. Level 3 Intelligence: Embodied AI systems (具身智能控制 - jù shēn zhìnéng kòngzhì). This is the realm of robotics deployments and visual-language-action (VLA) models. Here, the target area is further restricted to 20 tokens / joule, due to higher throughput (吞吐 - tūntǔ) and energy efficiency requirements. Currently, we’re even further behind in this realm, by 2-3 orders of magnitude off in energy efficiency.

This brings up a key challenge in energy-compute theory: currently, as IQ requirements increase, your inference energy efficiency decreases. But our energy efficiency requirements actually increase as we ascend the intelligence curve.

So how can we solve these two conflicting equations?

Robotics as an energy-compute problem

Prof. Yu asks the audience to consider the challenge of mobile robotics. A mobile robot does not have access to wired power, and cannot offload more complex queries to a centralized, larger model. Like humans, it must be able to think on the fly within a limited energy envelope.

Background

Power limits chips, chips limit memory and computational throughput, which limits token throughput. If we want robots to perform well in industrial environments (20+ hour operations), and current battery tech constrains unit lifecycles to 0.9 to 2.3 kilowatt-hours23 (3.2 to 8.3 megajoules), we need to think critically about how to allocate that energy budget in computations.

• At best, with an energy-efficiency rating of 5 tokens per joule, an 8 MJ (think Tesla Optimus) form factor would have a budget of 40M tokens to process on a single charge, not accounting for idle power, locomotion, etc..

• Given typical consumption of ~50K tokens for a reasoning run with thinking turned off, you might get 800 tasks completed with a reasoning model. Visual tokens are processed differently than text, but let’s simplify and avoid that for now.

• Versus 500K to 2.5M tokens for end-to-end reasoning with thinking turned on, you may only get ~20 to 80 truly complex tasks completed in that budget.24

• Speed is also a factor - robots must be able to not only think through the task, but complete it quickly. Is that range of completed tasks in a given power envelope faster or slower than human-level performance in an equivalent workday?

Modeling

Let’s reframe this in the form of an energy-compute problem. Assuming:

• You’d only require a single VLA query per task (“What are my instructions for placing this box on the shelf”)

• Subject to the following constraints:

  • Energy used cannot exceed your energy budget

  • Tasks completed must meet or exceed human-level performance in the same time period

Your task completion ceiling is the lesser of your token budget and your wall-clock time.

Results

Without getting too much further into the weeds, Prof. Yu assumes that ~100-1000 tokens/sec throughput is about the baseline (tau_sec,min) for useful embodied AI systems. That’s not a problem for well-designed cloud systems - 8 H20s can run full-blooded DeepSeek R1 at up to 8,000 tokens/sec - but today, decent models on edge hardware achieve 5 to 25 tokens/sec. He derives these figures from a 2019 study on edge robotics requirements.25

In terms of speed, that puts us 2-3 orders of magnitude behind the usefulness threshold. We can’t yet clear our human-level completed tasks threshold.

This still doesn’t factor in the constraint of high-quality token throughput - that you’re not just completing tasks faster than humans, but qualitatively better. That could be measured in terms of yield, defect rate, intervention rate, etc..

It’s not just about robotics

Let’s zoom out for a second. Why the obsession with robotics here?

Robotics is not just a sexy category. It’s a forcing function for rapid economic diffusion. By solving for energy-efficiency, you’re also solving for affordability in the economy’s most cost-sensitive sectors.

So when Prof. Yu mentions “ubiquitous edge intelligence for applications,” he’s talking about all kinds of economic applications, not just robots.

Consider one of the other keynote speakers: She Ying, founder of Haizhi Online.26 Haizhi is a digital platform connecting global industrial buyers with ~300K small and mid-sized factories.

In her world, AI’s (and technology in general) adoption by small and medium-sized manufacturing businesses is contingent on it being “understandable, useful, affordable, and effective.” That’s not a controversial statement, but it is rooted in customer empathy - all of her clients operate in intensely competitive local markets on razor-thin margins.

To that end, the concept of “SaaS subscription” as a revenue model does not really vibe in China. This is a big difference between American and Chinese consumers and businesses that’s often overlooked.

Many startups we spoke to are selling AI-enabled services by inking revenue-share contracts with their clients. Their strategy is to demonstrate that by partnering up, they can significantly boost their client’s sales, improving their competitiveness in highly crowded fields (manufacturing, textiles, etc.). Few, if any, were demanding up-front monthly subscription costs.

Incentives therefore are aligned for usefulness (high revenue growth for its partners) and affordability (to remain above breakeven), rather than passing usage costs onto customers.

For a great non-techie summary of this predisposition to ruthless optimization, check out Lesley Gao’s latest piece on Shaoyang and the ¥1 disposable lighter.

Shear Force

How a $1 Lighter Defied Inflation for 20 Years

At a time when coffee costs $6 and even cup noodles have seen price hikes, a few products have managed to hold their prices steady for decades. Costco’s $1.50 hot dog is perhaps the best-known case, maintained at that price only because it is intentionally…

Read more

9 months ago · 2 likes · Lesley Gao

“Ubiquitous edge intelligence”

Let’s wrap up the keynote by summarizing Prof. Yu’s calls to action. His milestone for ubiquitous edge intelligence requires the following:

• ✅ GPT-4o/o1 grade intellect

• ✅ In a <7B parameter (read: sub-8GB) form factor

• ✅ At 100 - 2000 tokens/sec throughput (tau_sec)

• ✅ With >20 tokens/joule energy efficiency (eta)

This succeeds a migration from fused, modular systems (模块化系统 - mókuài huà xìtǒng) into a single end-to-end model (端到端模型 - duān dào duān móxíng) which can handle a variety of tasks and challenges.

In the upper-left quadrant there is is a low-power, high-throughput domain which no tested combinations have been able to breach to date, let alone to do so with GPT-4o grade intellect.

As we’ve discussed, at a given iso-efficiency contour (~10 eta in best-case deployments), scaling alone does not take you closer to the upper-left quadrant. It just makes you more power-hungry.

End-to-end VLA models therefore require investments in hardware, model, and hosting infrastructure designs that considerably depart from scaling law theory. We need order of magnitude improvements in model and chip efficiencies.

He groups these improvements into three buckets. They are not mutually exclusive, as each technique carries advantages in each bucket.

1. ​​单次推理​​: Single-step inference (Token-by-Token)

2. 思维链推理: Chain-of-thought reasoning (Action-by-Action)

3. 算力系统: Hardware or system-level optimizations

Quantization (量化 - liànghuà)

Conserving memory bandwidth by casting model weights into reduced precision (e.g. FP32, FP16, integer-based formats, or new datatypes). Most hosts use quantization in inference today, but some research labs (like DeepSeek) are using mixed-precision in model training as well. Critical for fitting larger models into constrained memory footprints.

BinaryBERT was mentioned as an extreme demonstration.27

Sparsity (稀疏化​​ - xīshū huà)

Inspired by biological neural networks. “Deactivating” unused parameter weights in neural network activations. Even if a weight is set to “0”, the operation of multiplying that weight by another still contributes to arithmetic intensity. Critical for reducing computational overhead, and therefore, energy consumption.

This was a major focus area at the conference. Prof. Yu emphasized spent a fair amount of time detailing the benefits (maximize resource savings) and challenges (minimize degradation) from sparsification methods, and credited SparseGPT and DeepSeek-V2 for demonstrating its potential in post-training and training-aware approaches.2829

Fast Decoding

Speculative decoding and dynamic batching techniques to maximize token throughput within strict latency budgets. DeepSeek’s MLA attention mechanism is one such example. Critical for aligning with real-time requirements for inference with physical actuators.

In longer dialogues and super-long context windows (e.g. Kimi K2’s 1M token context window), inefficient decoding can carry serious memory loads and throughput penalties, making these techniques super valuable as user demands influence context length, memory, and so on.

Operator Optimization

Kernel fusion and hardware-aware operations (e.g., grouped GEMMs, hoisted activations) to minimize memory bottlenecks and exploit low-power silicon microarchitectures. Critical for achieving token-time and efficiency targets.

New Architectures

This could be either new chip designs like FPGA accelerators, neuromorphic chips, brain-inspired AI, and more.30 New memory-efficient model architectures like state-space models and task-specific language models are also increasingly in favor, especially for long context windows.3132

As you can tell from the earlier chart, novel chip designs (ASIC, PIM/NDP) are achieving considerably better token energy efficiency than general-purpose GPUs. However, they may not be suitable for all kinds of computations… which is why the next advancement is so important.

Heterogeneous Cooperation

Many research labs throughout China are now training or hosting models on heterogeneous rather than homogeneous chipsets. Distributing inference across hybrid chipsets balances the benefits of each platform, such as the flexibility of multi-core (多核) CPU with highly parallel (高并行) GPU throughput.

As an example, in this paper from the Shanghai AI Lab, the team demonstrates they can not only meet, but actually exceed token throughput for LLM training runs using mixed chip stacks:33

In practical production environments, lower-spec chips usually feature significantly lower pricing and reduced power consumption compared to their high-spec chips. By leveraging the HeteroPP framework for heterogeneous training, which integrates both lower-spec and high-spec chips, we can achieve training performance comparable to or even exceeding that of homogeneous high-spec setups, which preserves high performance and reduces overall training costs.

Ding Tang, et al., 2025.

Successful implementations here not only yield energy efficiency improvements - they de-link China’s chip ecosystem from the chip export bottlenecks, as they’re no longer tied to performance of homogeneous leading-edge GPUs.

Takeaways

Alright, that was a lot. Let’s wind it down and bring it all back.

I started this post by introducing energy-compute theory, its inputs, and its constraints. Then, we investigated how China’s AI ecosystem has been leaning full-tilt into this development framework, in comparison to the US’ preference for IQ-maxxing.

What are the implications if these ecosystems continue on those development tracks? Here are three to follow:

Open-source continues to raise Qmin.

Strategically, continuing to release more and more capable open-source models raises the baseline for expected intelligence levels in deployed models. But without commensurate investments in model or chip optimizations, energy efficiency will drop, and inference costs will rise.

GPT-5 may already be a casualty here. Some skeptics consider its launch a cost-cutting measure rather than a material improvement in capability - an ill-conceived attempt to abstract reasoning effort and save on inference costs by nerfing the consumer experience for 1BN users in the process.34

Chip wars diverge national eta values.

Thanks to recent shenanigans, Chinese domestic semiconductor independence is now a matter of when, not if.3536

As the market leader, NVIDIA will continue pushing out more capable chips, but every release since Volta has only averaged a 1.53x improvement in performance-per-watt. So it’s doubtful that homogeneous NVIDIA systems will achieve the energy efficiency values necessary to meet audacious economic diffusion targets (to say nothing about developer friendliness towards CUDA - NVIDIA still takes the crown there).

Hyperscalers have been hard at work developing their own custom silicon in order to reduce their total cost of ownership (TCO), of which energy use is a major factor.37 Even in the most resource-rich playgrounds, efficiency matters.

The post-WAIC establishment of Model-Chips Ecosystem Innovation Alliance is an encouraging signal that integrative R&D circles in China may spur joint advancements in model and chip energy efficiencies. The initial listed members cut across each category:38

• Models: StepFun

• Infrastructure: Infinigence AI, SiliconFlow

• Hardware: Huawei Ascend, MetaX, Biren, Enflame, Iluvatar CoreX, Cambricon, Moore Threads

China surpasses the US in energy-compute budgets (E).

This will likely happen regardless. China already enjoys a 6,000 terawatt-hour (TWh) per year advantage over the United States today, and that may rise to 10,000 TWh by the end of the decade.39

While the US spends a larger proportion of power on data centers today at 327 TWh vs. China’s 163 TWh in 2024, that budget is impossible to ignore… it’s the equivalent of the cumulative TWh consumption of the entire OECD.4041

The Triple Product Advantage

Single-dimension scaling has plateaued; the limits of pure parameter growth are now pretty much mainstream.

Unless the US keeps pace on all three fronts - novel model architectures, high-efficiency silicon, and plenty of power - China’s AI output will compound past ours in economic value.

It will be gradual at first, then sudden. In energy-compute theory these levers don’t add, they multiply: a modest gain in each lever (energy, efficiency, IQ) produces a polynomial jump in the total. This is called a triple-product advantage.

Is this doom and gloom for the American AI ecosystem? Not at all, just a call to action. Perhaps we should be thinking less like software venture capitalists and more like energy companies.42 That’s what energy-compute theory is meant to capture.

Simply put, as model intelligence commoditizes thanks to scaling law limitations, we may finally be entering the era of ruthless optimization.

Some might call it an “AI Winter”… I call it the start of something great.

1

Dwarkesh Podcast

Dylan Patel & Jon (Asianometry) – How the Semiconductor Industry Actually Works

Listen now

2 years ago · 84 likes · 7 comments · Dwarkesh Patel

I’ve got a lot of upcoming posts on the China AI market map, its rate of diffusion, and global AI infrastructure developments coming out of this. But for now, let’s take stock of the Action Plans unveiled last week and set the table.

HUGE thank you to Rui Ma and the Tech Buzz China team for organizing this tour. Our delegation of researchers, academics, and industry experts definitely came away informed and invigorated.

The “race” so far

Last week, the US published its renewed AI Action Plan, subtitled “Winning the AI Race,” and it had some surprises in it. Some great, some mid, and some déjà vu.

This dropped while I was in in Shanghai, shuttling between the World AI Conference expo hall (sold out to 30,000 participants), state-sponsored industrial parks, and cutting-edge research labs. And at this conference, China’s competing Global Governance AI Action Plan also dropped. This led to some fun conversations with curious local experts.

I left the tour with two reactions:

1. China’s AI build-out is even faster than Western headlines admit, and

2. Most Westerners are still obsessed with who’s first, not how they get there

Whenever I hear the phrase “AI race,” it deeply bothers me. A race implies a finite game, a finish line. But with artificial intelligence there is no discrete finish line, despite pundits and policymakers’ insistence otherwise.

What would happen after we “reach AGI”? Technology is over, the world ends? More than likely, we’ll just shift the goalposts forward to some new threshold. No, progress will just continue to compound as its benefits diffuse into every economic sector.

But for argument’s sake, let’s play into the narrative.

We’ve all heard the allegory of the tortoise and the hare. In the old fable, the tortoise wins by never stopping.

Take electricity generation for example. There’s a viral graph that’s been making the rounds that, if you haven’t seen it, puts things into perspective. This chart shows the total amount of electricity generated in the US and China in the past 40 years, starting a few years after Deng Xiaopeng’s economic modernizations.

Today, China generates over 6,000 terawatt-hours (TWh) more electricity than the United States, enough juice to:

• Power every US home 4 times over (541 million homes worth of power)

• Build ~7,000 new large industrial facilities

• Run 4.9 million Blackwell GPUs at full-tilt

Meanwhile, America’s year-over-year growth in power demand, already middling, flatlined to an average of just 8 basis points from 2010 to now. We literally took a nap.

Energy has and always will be a leading correlative factor for economic output. It forms the foundation for modern standards of living, domestic production and manufacturing capabilities, and now - all of a sudden - the bedrock for the next generation of the services economy, the “intelligence economy.” Advanced artificial intelligence systems are here, and boy are they hungry for power.

So while dominant Western social movements advocated for de-growth and McKinsey was proclaiming a post-energy future, the rest of the world didn’t seem to get the memo.

China’s tortoise has been plodding along at a Breakneck pace, but apparently too slow for most Americans to notice. Until now.

The American hare has woken up.

And it’s got some ground to cover.

Review: America’s AI Action Plan

Honestly I had low expectations given my experience with Trump 1.0’s equivalent action plan which I wrote about three years ago, but I was genuinely and pleasantly surprised to see a lot of sensible calls to action in here.

The Plan has three pillars: innovation, infrastructure, and international diplomacy and security (wait that’s four pillars…), each with 15, 8, and 7 respective subsections. I won’t be commenting on each one here, just a handful that I’ve added 💬 emojis next to in the below table.

China’s competing Plan was actually rather pallid in comparison: a slim, 4 page document with mostly broad statements about the importance of multilateral collaboration. This was yet another surprise for me, as that was an unusual departure from typical contrasts between these nations’ industrial planning documents. However, we are still 8 years into China’s 13 year development roadmap, so it could be the case that Beijing decided to restrict their new plan to matters of governance, safety, and standards-setting in the generative era.

This limits an apples-to-apples comparison to just the third pillar in the US Action Plan, regarding International Diplomacy and Security. I’ve got a ton of upcoming blog posts on AI diffusion (pillar one) and AI infrastructure (pillar two) in China, so I’ll save those for another time.

With that, let’s cover some of the noteworthy sections in the plan, and give a quick gut-check on national readiness to rise to the challenge.

I. Accelerate AI Innovation

General themes for this first pillar are:

1. Decluttering red tape

2. Greasing the wheels for public and private sector adoption

3. “De-woking” frontier LLMs

Overall this sets things in a positive direction, celebrating open-source advocacy, common-sense adoption levers, and thematically supporting a liquid compute marketplace. But in contrast to the laissez-faire approach for the first two, that latter theme concerns me not in its ideology, but in its practical implementation that would lead only to a quagmire.

Oh, we like open-source now

The plan’s strongest surprise is an explicit embrace of of open-weight models. It acknowledges that it gives startups more flexibility, lets governments and businesses keep data in-house, and are absolutely essential for academic researchers.

In my last post, I was concerned by many leading voices in the American AI ecosystem calling for pulling the rug out on open-source. But as the plan reads, the US is taking a very sensible position which leans into the narrative that open models contribute to soft power in the new economy.

Had we gone the other way I’d be writing a very different blog post.

Towards a liquid compute marketplace

Another welcome surprise was the acknowledgment that many are in unfavorable positions when trying to access reliable (i.e. not spot market) compute.

We absolutely should be ensuring level access to large-scale computing power rather than concentrating it oligopolistically. The National AI Research Resource (NAIRR) Pilot from NSF is a start, but it’s still a boutique collection of one-off credits and in-kind resources.

At a major industrial park in Hangzhou, I learned the city government actually created a state-owned cloud compute reseller to flip compute to startups at up to a 50% discount. The Shanghai city government announced similar subsidies last week.

The faster we align on a similar view that AI is part of a new energy-compute asset class, the faster we’ll lower the unit cost for all participants, not just large incumbents.

Evals for enterprise adoption

I think that at steady-state, a healthy evaluations ecosystem vs. “trust me bro” benchmarks is going to provide much more trust in a market that can be crowded with vibes and grifters. Third-party assistance here will make it easier for founders to cut through that noise.

Academic tests are decent barometers of research progress, but they are not procurement criteria. Stats can be juked through selective tuning, hand-crafted prompts that don’t represent production use, or straight up fraud. You need real-world performance standards informed by industry input.

NIST’s Face Recognition Technology Evaluation works because it measures an end-to-end task with clear failure modes. Domain-specific LLM applications need equivalent rigor in their benchmarking.

It would be great to see something like this rolled out with fixed datasets, common metrics, and quarterly reports. Vendors will either improve their baseline, or they won’t. Grifters would be reluctant to put their slop through an open evaluation process like this, and the truly talented would be elevated (one can hope).

Transparency will do wonders for clarity in the marketplace, and hopefully speed up the enterprise adoption cycle from quarters to weeks in the process.

No bias but our bias

Unfortunately, this pillar is also internally inconsistent. It’s second section, “Ensure that Frontier AI Protects Free Speech and American Values,” calls for models that are “objective and free from top-down ideological bias,” yet immediately instructs federal agencies only to buy models that exhibit a particular ideological slant. In this case, the absence of DEI or climate-related content, and the rejection of “Chinese Communist Party talking points.” You cannot outlaw bias with one hand while prescribing it with the other.

If procurement rules require that certain topics (misinformation, DEI, climate change) be removed from the NIST AI Risk Management Framework, the policy is no longer content-neutral. It privileges one worldview and disqualifies others, contradicting the First Amendment principle the plan claims to defend.

Unintended consequences and the “Waluigi Effect”

LLMs learn concepts in oppositional pairs - full/empty, wet/dry, loyal/subversive. When training forcibly suppresses one side of a conceptual pair, the model often preserves it latently - the same vector arithmetic that yields man : king :: woman : queen can flip suppressed traits back on, sometimes in exaggerated form.

Public examples of what alignment researchers call the Waluigi Effect include Bing Chat / “Sydney” abruptly becoming hostile or conspiratorial under pressure, and the Grok / “MechaHitler” jailbreak which flipped the model into extremist rhetoric.

In any case, these failures stem less from hidden agendas than from brittle, rules-based patches that ignore how general these systems really are. In truth, these are likely influenced by the role of fictional tropes in training data (“Evil All Along” is a very common plot twist in literature, and “MechaHitler” is a reference to the 1992 video game Wolfenstein) rather than some sort of top-down bias or Manchurian candidate situation.

Policymaking based on ideological, rather than technocratic, goals can only introduce inadvertent externalities down the road. Platitudes like “ensure free speech,” “reject CCP talking points,” or “DO NOT HALLUCINATE” will not deliver the reliability policymakers want. Empirically validated, context-aware guardrails seem like a better way to accomplish those goals without sacrificing the very freedom of expression the plan seeks to protect.

II. Build American AI Infrastructure

The most well-flushed out pillar of the Action Plan focused on practical constraints and development plans for energy-compute infrastructure, which I’m very excited by. The administration calls for the following:

• Streamlining permitting for new datacenters, semiconductor manufacturing facilities, and energy infrastructure

• Prioritizing grid stability and rapid buildouts

• Restoring American semiconductor manufacturing

• Beefing up a skilled infra workforce

I’m honestly stoked about most of this pillar. However, I can’t help but lament how long it took for us to get to this point, and wonder about our long-term commitment to revitalization. In particular, the plan ignores the critical role that international talent has to play, and I’m convinced that blind spot will bite us in the long run.

Prioritizing stability over ideology

Show me the incentive and I’ll show you the outcome.

– Charlie Munger

This is probably my favorite part of the entire action plan, for one specific line: “reform power markets to align financial incentives with the goal of grid stability.”

It starts by reiterating that our electric grid is critical for all aspects of the modern economy and must be safeguarded, while acknowledging that in its current form, it’s unsuited for the increased pressures of AI datacenters. Training and deploying large-scale models carries high load variability (±15MW peak-to-trough in milliseconds) and therefore very real blackout risk for energy grids which cannot absorb that much demand-time.

The safest way to accomplish this is by introducing large volumes of “baseload” power - reliable, stable sources like natural gas, hydropower, and nuclear - to create a high watermark in excess of that demand variance, say 85% of peak.

• Excess power from these projects during downtimes can be added back to the grid, enhancing stability.

• During peak times, pairing these projects with battery energy storage systems (BESS) can significantly help with sub-second ramps, since lithium ion batteries excel at this.

Up to now, the interconnection queue has not at all resembled these needs. In the most recent report from the Lawrence Berkeley National Lab, of the 1,570 GW of new generation proposals still in the queue (which takes 3 to 8 years to clear right now, by the way), a meager 118 GW (7.5%) of new power fits this baseload profile. Intermittent generation like solar and wind dominate the queue, commanding 1,086 GW (69.1%) and 366 GW (23.3%) respectively.

A good deal of this buildout was no doubt influenced by the prior administration’s last-minute bum rush to push out $400BN of debt financing to renewable energy projects… influenced by ideology, not by the practical needs of energy-compute infrastructure in the new economy.

Not all power is equal

Remember, we measure power not just in instantaneous terms (gigawatts) but in time-denominated terms (gigawatt-hours). So when comparing power from different generation resources, you need to consider over what period of time that’s deployed.

Since battery duration is limited to just a few hours (1 to 4 commonly), we can’t rely on BESS alone to handle these loads. When paired with renewables, developers have to significantly overspec in order to provide continuous power.

• Using Las Vegas as a sunny example, irradiance patterns allow for delivering 1 kW of continuous power via 5 kW of solar panels paired with a 17 kWh battery

• Scaled up to a 100 MW AI data center, this would require 500 MW of solar panels (making it the 6th largest deployment in the US) and 355 MWh of storage supported by at least 91 Tesla Megapacks (4-hour configuration)

Recall from earlier that our current delta with China’s generation capacity is roughly 6,000 terawatt-hours (TWh) - that’s 6 million gigawatt-hours (GWh). If we waved a magic wand and activated all of our interconnection queue simultaneously, could we close the gap?

By some reasonable estimates, yeah, we could narrow or even close it entirely. But those hours aren’t uniformly distributed throughout the day. There’s no reason why baseload power can’t be augmented with renewables and BESS, but putting all your chips in the intermittent basket with the fluctuations AI workloads introduce doesn’t make sense. Hence the Plan’s emphasis on comprehensiveness, stability, and consistency.

Cutting the red tape

Most data center infrastructure projects require significant upgrades to grid infrastructure to handle contemporary AI workloads. The Action Plan more or less gives the green light for data center-related projects to either fast-track or bypass reviews and permitting, so long as the site is consistent with the size of a modern AI data center. What that size is specifically remains unclear in this document.

Aside from energy and labor, seeking relevant permits and environmental reviews do take a significant portion of time in new and brownfield builds. This administration calls for using exclusions granted by existing regulations to speed things up, rather than go through Congress:

• Establishing new Categorical Exclusions under the National Environmental Policy Act (NEPA) to cover data center-related actions

• Expand the use of the FAST-41 process (Title 41 of the Fixing America’s Surface Transportation Act) for speedier reviews and processing

• Make Federal lands available for data center construction and the construction of power generation infrastructure for those data centers

This is generally good news, especially on the third point, which I detailed in a May submission to the Department of Energy’s RFI for building AI data centers on federal lands. A coalition of industry reps from energy, AI, and power that I gathered identified and proposed the most suitable DOE-owned sites that could house energy-compute projects, based on a number of factors. It’s good to see the DOE following through on this.

Restore American semiconductor manufacturing

Perhaps in a future post I’ll give a history and near-term forecast of the US semiconductor industry. For now, all you need to know is that while American companies like NVIDIA, AMD (as well as new entrants like Positron and Groq) may lead fabless chip design, on-shore production of bleeding-edge nodes is in a pretty dire state:

• Intel’s recently decided to shelve its 18A (1.8 nm) process in favor of diverting resources to 14A (1.4 nm) underscores the challenge

• Analysts warn Intel has roughly 18 months to land a “hero customer” for 14A… otherwise its cutting-edge fabs could be abandoned altogether (forfeitting any further CHIPS Act disbursements)

• Even foreign foundries with US fabs are facing problems. Samsung’s $44BN Texas mega-fab - originally slated for 4nm mass production - has struggled with yields and customer commitments, prompting executive leadership to declare a 6 month transition window to 2nm, and delay 1.4nm to 2029 while they dial in reliability

The US government does wield some levers. In the next decade, the Semiconductor Industry Association (SIA) projects the US fab capacity will grow by 200%+ over the next decade - doubling the global average - and American semiconductor R&D outlays exceed $60BN, the highest in the world today in purchasing power parity (PPP), despite our tax incentives lagging peer nations.

But funding is just one part of the problem, as our prior examples show. If domestic champions like Intel can’t make the cut meritocratically, then we shouldn’t be casting taxpayer pearls before swine.

To that end, the SIA projects a domestic shortfall of 67,000 workers across the semiconductor manufacturing chain - device and machinery manufacturing, design, and EDA tools - by 2030.

Given that this section imposes a sensible constraint of “lead[ing this] revitalization without making bad deals for the American taxpayer” (emphasis mine), I fear that without significant advances in domestic semiconductor talent to cross critical yield thresholds (70%+) in a short period of time, domestic manufacturing will continue to be a satellite for foreign fabs like TSMC and Samsung, while domestic players only satisfy trailing edge process nodes.

This isn’t the worst possible outcome, but it does put the U.S. at a disadvantage if China continues to improve in fabless chip design and domestic manufacturing.

We need international talent. Full stop.

Even if the U.S. fast-tracks every permit, reopens every chip fab, and supercharges the grid, we still face a crash course in human capital. We’re not just talking about the semiconductor shortfall - we need tens of thousands of electricians, HVAC technicians, network engineers, and other tradespeople to support this buildout.

Yes, the administration’s push for national, state, and local workforce programs is welcome - industry-driven training, revamped apprenticeships, early-career exposure and the like. But this transformation will take at least a generation without top-tier international engineers and technicians.

Nativist tit-for-tats on Chinese student visas are worsening, not improving, the problem. I’m not even coming from an emotional place on this, despite my obvious affinity for both countries. I’m stating a fact that in many STEM industries, even in our most advanced AI research labs, the diffusion of Chinese talent is impossible to ignore. Half of Meta’s Superintelligence team got their undergraduate degrees in China, for Christ’s sake.

This is not to say that domestic employment concerns are unwarranted. There are some common-sense changes we can implement that achieve a favorable middle ground, like revamping the green card system to a merit-based skills prioritization matrix rather than a country quota exercise. Permanent residency for critical roles like semiconductor engineers, AI researchers, etc. should be sought.

But what is the benefit of antagonizing those talented young engineers, if only to send back home and accelerate China’s progress further? As Kaiser Kuo said in his response “An Own-Goal of Historic Proportions,” Chinese officials were likely publicly condemning the move, but privately popping the baijiu.

Sinica

An Own-Goal of Historic Proportions:

Written in haste — forgive the unpolished prose…

Read more

a year ago · 597 likes · 11 comments · Kaiser Y Kuo

Meanwhile, at that same industrial park in Hangzhou I mentioned earlier, I asked a senior government official what their policy was towards foreign talent. What kinds of personnel were they looking for, and what support (if any) did they offer? I’m paraphrasing his translation, but

“Foreign talent in key industries, especially PhD holders, are more than welcome. In addition to the office lease discounts and apartment leasing subsidies for college graduates, overseas talent can get free housing ($1M to $2M USD)”

I’ve heard of such programs in China’s broader “Thousand Talents Plan,” but it was my first time seeing it face to face, despite the escalating tensions.

Even if we solve our energy problems… even if we cut all the red tape… we risk building golden factories with empty benches. Unless the administration pivots here, our nativism will kneecap our ambition.

III. Lead in International AI Diplomacy and Security

Overall, this part of the action plan falls woefully short. Predictably, the administration pushes for a unilateral promotion of American AI systems, compute hardware, and standards throughout the world. It’s also logically inconsistent, or at least overestimates the negotiating leverage Americans have in bilateral trade of these systems outside of a vacuum.

Consider the U.S.’ failed efforts to limit Huawei’s 5G diffusion throughout the developing world. I’ll briefly summarize those points below, then apply them to the Action Plan’s stated intentions against the status quo:

Recap: Why the Huawei 5G Ban Failed

1. U.S. “Security Threat” Narrative. Washington repeatedly labeled Huawei “an arm of the Chinese state,” warning that its 5G equipment could be used for espionage or cyber-sabotage. Brazilian and South African officials pushed back, demanding concrete proof of back-doors or spying, but no transparent or legally vetted evidence could be shared.

2. No American 5G Champion. Meanwhile, as a counter, no U.S. or U.S.-allied vendor offered a full end-to-end 5G solution at Huawei’s price and scale. With neither domestic champions nor subsidized U.S. exporters stepping up, local operators saw little downside in choosing Huawei for cost-effective, rapid-deployed, field-proven equipment.

3. China’s “Digital Silk Road”. Beijing wove Huawei into broader “Digital Silk Road” and BRICS-era cooperation, offering low-interest loans, technical assistance packages bundled with infrastructure financing, and joint development labs and exchanges. As usual, this was a financial “no strings attached” proposal boosted by multilateral collaboration on shared principles for digital development.

Taken together, these factor left Brazil, South Africa, and many countries free to integrate Huawei into their 5G roadmaps.

The more things change...

However, the current situation is not completely congruent. The first item in this section advocates for exporting America’s “full technology stack – hardware, models, software, applications, and standards.” That’s a genuine step up from 5G, since American labs and chip designers currently own the leading edge.

But the stack is more than just chips and models. As mentioned earlier, AI datacenters have insanely high load variability and require significant energy infrastructure upgrades. Without behind-the-meter generation, high-voltage transmission lines, battery energy storage systems (BESS) and the like, many developing countries wouldn’t be able to locally deploy complete stacks even if they wanted to.

China bundling energy infrastructure investment and financing together with datacenter rollouts would be a successful repeat of its 5G playbook. And early innings of that strategy were on display in the WAIC expo hall.

...The more they stay the same

Because energy and digital infrastructure rollouts are tightly interdependent, local operators and regulators may naturally gravitate toward China’s packaged offerings, especially where BRI momentum is already strong.

Further, ceding the field on AI safety and governance presents too wide a gap that China’s “AI Governance Action Plan” is primed to fill. Beijing called for convening governments, industry and civil-society around multilateral standards, capacity-building programs and joint R&D - while at WAIC not only did no Trump administration AI safety officials show, the only representative from a US AI lab even proposed an outright ban on open-source models, arguing “you wouldn’t give everyone a nuclear bomb.” From Paul Triolo:

The lone official from a leading US AI lab, Dan Hendrycks, raised eyebrows by essentially calling for a ban on open source/weight models, while some Chinese AI safety experts called for just the opposite, urging the US to force proprietary model developers such as OpenAI and Anthropic to open source their models, arguing that this was a better way to ensure the safety of advanced models going forward.

To say Hendrycks read the room badly would be an understatement. While he is a respected safety researcher and red-teamer, his position leaves no room for negotiation or joint development, is in contrast to the Action Plan’s commitment to open-source, and left most of the room feeling puzzled.

Overall, I’m getting a strong sense of déjà vu: Beijing’s “inclusive multilateralism” mirrors the Digital Silk Road playbook, whereas the US push to “Counter Chinese Influence in International Governance Bodies” suggests that any multilateral consensus that includes Chinese participation is inherently suspect, prejudicially undermining sovereign decision-making.

I’ll stop my comparisons here, as Paul goes into excellent depth in his latest post on his Substack, AIStackDecrypted, which I strongly agree with and encourage you to read.

AIStackDecrypted

China set to lead global effort on AI safety?

This year’s World AI Conference (WAIC) in Shanghai was well attended by Chinese AI companies, robotics firms, infrastructure providers, investors, and other companies across the AI stack. The record attendance cited by organizers testifies to the intense interest in AI at many levels across China’s economy, and the event was dominated by the innovation …

Read more

9 months ago · 9 likes · 3 comments · Paul Triolo

Conclusions

The American AI Action Plan is a welcome signal that Washington is waking up to what’s necessary to succeed in the new economy:

• Open-source wins. It embraces open-source and open-weight models - crucial to developer and industry soft power

• Grid stability over ideology. It confronts the brutal physics of the grid, without an ideological slant - no electrons, no intelligence economy

Yet it tiptoes around two levers that decide whether the American hare will actually start closing the gap:

• Talent. 67K semiconductor hires plus tens of thousands of skilled trades won’t materialize out of thin air. Skills-first green cards, visa-to-apprenticeship tracks, and a reversal in the student-visa nonsense is critical

• Diplomacy. Export controls and America-first platitudes aren’t effective strategies. Allies need a full AI stack bundled with infrastructure, financing, and standards toolkits, but they need it on a multilateral basis, not one-way

So yes, it’s a start. But we need more than an Action Plan… we need a Marshall Plan. Alliances built out of watts, wafers, workers, and a shared vision of the future.

Because there isn’t a finish line. There’s just progress.

I kept this long because I wanted the arguments in a complete form, and to discourage readers jumping to conclusions from unqualified headlines and half-truths. Fortunately, some major voices have provided fulsome commentary by now, so we get a complete and timely picture vs. pigeonholing into any one narrative.

Introduction

Over the holidays, a Chinese AI lab called DeepSeek released a pair of large language models which rocked the markets and AI industry. For most, the prevailing wisdom had been that bigger was better - raw compute power, ever-larger parameter counts, and economies of scale were the only path to achieving state-of-the-art intelligence that could compete with the top players.

Except DeepSeek trounced them all with a tiny fraction of their resources. Reportedly, DeepSeek-V3 was trained on just $5.6M worth of compute1, compared to $78M for GPT-42 and over $100M for LLaMa 3 405B.3

This was a shot across the bow of the AI industry's reigning "sovereigns" - the super-scale labs whose strategy rests on cornering the market for AI capital, compute, and energy. It's the first domino to fall in a chain reaction that will reshape the technopolitical and economic landscape of AI as we know it - to no surprise to watchful observers, but a major surprise to the sovereign ecosystem.

Here's how we'll break it down:

1. First, I'll demystify the architectural decisions that contributed to DeepSeek's efficiency gains.

2. Second, I'll lay out strategic implications for "sovereigns," the super-scale AI labs whose entire strategy rested on accessing capital, compute, and energy resources on the scale of nation-states.

3. Third, we'll assess the immediate market fallout and compelling emergent investment themes.

4. Fourth, cover the geopolitical reckoning with example reactions from American and Chinese stakeholders.

Ultimately, I argue that DeepSeek's breakthrough validates the AI commoditization thesis, challenges the sustainability of the sovereign AI model, and heralds a new era of AI diffusion across players and industries - one that demands a strategic reset from stakeholders looking to avoid the "sovereign trap".

Before diving in, keep in mind this essay is a deliberately provocative take meant to challenge orthodoxy. I’m an investor in companies like Fastino and Positron that compete with sovereign AI strategies, and am a vocal advocate for open-source models, which may color my analysis of centralized AI power structures. I’ve aimed for objectivity, but please weigh my affiliations against the merit of my arguments.

1. The Efficiency Revolution

When talking about great power competitions, especially in the tech context, I often talk about "diffusion," a framework Jeffrey Ding (ChinAI, George Washington University) introduced to characterize the spread of an innovation throughout a population or ecosystem.

To summarize, general purpose technologies (GPTs) like electricity or broadband gradually "diffuse" across many industries which make use of their horizontal capabilities. This contrasts against the "leading sector" approach which emphasizes concentrated upheavals in singular technologies, like railroads or automobiles, which while impactful are not as portable across industries.

Leading sector technologies lend themselves to monopolization, while GPTs benefit greatly from tailwinds that accelerate societal adoption, like drastically lowering the cost of inputs, or widening the base of engineering skills that take advantage of the GPT. Nation-states' success in leveraging these innovations depends on how effectively their institutions respond to those needs.

Up to now, western institutions in the sovereign AI ecosystem have behaved very much in a leading sector approach, emphasizing concentration of talent, compute, energy, and capital in a handful of captured labs. But DeepSeek's revolutionary efficiency gains was an ecosystem earthquake, causing many to rethink their priors and question the sustainability of the sovereign AI model against an impending tsunami of efficient diffusion.

DeepSeek's Architectural Innovations

First, let's get up to speed. There are two models DeepSeek released recently, both with their own implications:

• DeepSeek-V3 is an advanced language model that tops leaderboards among open-source models, despite its puny compute resources. It builds upon the successful techniques discovered in prior DeepSeek model iterations.

• DeepSeek-R1 is a "reasoning model," a modified version of DeepSeek-V3 developed using pure Reinforcement Learning (GRPO). It is fully open-source, MIT-licensed, and reportedly performs on par with leading AI models. The model was released on January 20, 2025, and is claimed to be 20 to 50 times more efficient than its primary competitor, OpenAI's o1.

A "reasoning" model is an evolved LLM which uses "chain of thought" to improve its problem-solving capabilities. Basically, it thinks out loud to itself before giving an answer. This usually leads to higher quality answers, especially in complex tasks, workflows, and critical thinking problems. R1's efficiency gains stem from the same algorithmic and recipe improvements that went into V3, so I often refer to them both in the collective.

To keep it simple, think of DeepSeek's breakthrough as building a more fuel-efficient engine rather than building bigger gas tanks. The team introduced four key innovations that, working together, deliver the same performance as models 10x their size while using a fraction of the resources. Let's break those down:

1. FP8 mixed-precision training framework

At its core, this is about being smarter about how we store numbers in AI models. Traditional models insist on extreme precision, storing all parameter weights in 16-bit or 32-bit precision, like measuring your height down to the nanometer.

DeepSeek proved you don't need that level of precision for every calculation, choosing instead to use 8-bit precision wherever possible without inducing quality loss. They're not the first to implement this approach, but it is the first to do it on such a large scale. Several research teams have previously implemented mixed-precision training approaches, such as Microsoft's FP8-LM framework4 and BitNet56 (which uses binary weights, seriously pushing the boundaries of low-precision training).

DeepSeek's implementation yields a 40% reduction in memory usage while maintaining model quality. This is a direct challenge to the "more compute = better AI" orthodoxy that's dominated the field up to now.

For the technically curious, DeepSeek's implementation includes specialized hardware optimizations and dynamic range adaptation that prevent information loss, detailed in their technical report.

2. Multi-head Latent Attention (MLA)

Traditional transformer attention is like having to remember every word you've ever read to understand the next sentence. Developers try to improve prediction performance with larger and larger context windows, but without addressing _how_ information is stored and retrieved, they're anchored to an exponential scaling problem.

In contrast, DeepSeek's MLA mechanism is more like human memory: it focuses on key concepts and relationships rather than raw data. Instead of storing the entire context window in memory, DeepSeek's MLA mechanism stores a latent vector representation of the context window, compressing the information into a smaller, more manageable size - by 93.3%.

This reduces memory complexity from O(n²) to O(n·k), where k is a constant - a game-changing improvement for scaling. It enables models to handle context windows 4x longer while using 60% less memory.

The full architectural details are available in their MLA paper published last year7, but the key takeaway is simple: longer context windows no longer mean exponentially higher costs.

3. Auxiliary-loss-free MoE

A mixture-of-experts (MoE) model like DeepSeek's is a lot like a hospital. You have lots of specialists on call, and when patients come through the door, you want to route them to the right specialist who can give them the best help. That's why it's called "mixture" of experts - the full 671B model is actually a lot of smaller expert models stitched together.

You want to make sure each patient sees the right specialist, but time is money, and you don't want doctors sitting idle while others are overwhelmed. Traditional Mixture of Experts (MoE) models face the same challenge - they need complex "traffic control" systems (auxiliary losses) to route work efficiently among specialist neural networks.

If this sounds familiar, it's because it's conceptually similar to the model routers we discussed in Betting on Model Marketplaces

While external routers can use fancy techniques to choose between different models, internal MoE routing traditionally required crude balancing mechanisms that often hurt performance by forcing work to less-qualified experts, just to maintain balance. Like sending patients to a dermatologist when they really need a cardiologist, just because the dermatologist's waiting room is empty.

DeepSeek's innovation is a new gating mechanism that achieves natural load balancing through the architecture itself, by considering both expert specializations and current load. Their implementation modifies the traditional sparse gating function.8

When combined with MLA, this approach yielded a 42.5% reduction in training costs and a 5.76x boost in throughput. Using just 21B parameters per token, DeepSeek-V2 matched the performance of models 2-3.5x its size, like Mixtral-8x22B and LLaMa-3 70B, on a drastically lower compute budget.

This matters because MoE architectures are one of the most promising paths to scaling AI capabilities without linear compute increases. While Mistral's Mixtral9, Microsoft's Retentive Network10, and Google's Switch Transformer11 demonstrated MoE's potential (and GPT-4 is rumored to use it12), auxiliary losses remained a key bottleneck until now. DeepSeek's implementation found a path to more efficient scaling without complex load balancing schemes.

4. Multi-token prediction system

Traditional language models generate text one token at a time - like someone reading a book word by word. In order to predict the next token in a sequence, the model draws on both its world knowledge and prior tokens in the current context. Sequential generation therefore creates practical memory and throughput bottlenecks in real-world applications, as the amount of context in the conversation grows and grows.

To get around this, DeepSeek's implementation predicts four tokens simultaneously, using a "look-ahead" mechanism during both training and inference, to make parallel predictions about the next token. Their technical report describes a causal masking approach that maintains proper sequence dependencies.

When combined with their MLA architecture and auxiliary-loss-free MoE, DeepSeek reports:

• 2.5x speedup in text generation

• 40% reduction in training time

• Quality metrics comparable to sequential prediction

Once again - there's prior art for this technique. Most recently, Meta FAIR published research showing up to 3x speedups through multi-token prediction while improving model quality.13

Recap

I hope what's clear by now is that DeepSeek's innovations in no way seem to be secret research or state-sponsored voodoo. These implementations have been validated in prior work by Western sovereigns and open research teams. They're just the first to combine them.

When combined, the efficiency gains compound tremendously:

• Memory: 93.3% reduction through MLA, while extended context windows by 32x

• Training: 71% reduction in compute costs, achieved with just $5.6M total training budget

• Throughput: Up to 5.76x improvement during inference

• Performance: Matches or exceeds models 2-3.5x larger on standard benchmarks

Culminating in a state-of-the-art model trained on $5.6M worth of compute, compared to $78M (GPT-4), over $100M (LLaMa 3 405B), and $191M (Google Gemini Ultra).

The Sovereign AI Trap

These efficiency breakthroughs expose a critical flaw in the sovereign AI strategy: what if bigger isn't actually better?

Thus far, sovereigns' approach to AI development has followed a predictable pattern:

• Raise massive amounts of capital

• Build increasingly large models behind closed doors

• Maintain strict control over architecture and training data

• Rely on raw compute power as a competitive moat

• Where possible, use regulatory capture to strengthen their position14151617

Wind back the clock a year to February 2024: OpenAI's Sam Altman was attempting to raise $7T (TRILLION) to reshape the global semiconductor industry, arguing he needed unheard of chip fabrication investments meet AI compute demands18, no doubt influenced by OpenAI's massive cost basis19.

Sam's concerns weren't exactly unfounded at the time, but DeepSeek's reported cost advantages have now cast significant doubt on the sticker price he was quoting.

To put it in perspective, The Information estimated that for 2024, OpenAI spent up to $6B on compute-related costs against annual revenues of $3.5 - $4.5B. If the numbers we've heard are true, that would mean OpenAI is spending more than 2 DeepSeek-R1 training runs per day on compute costs.

Now it's at this point that skeptics would assert one of three things:

1. DeepSeek had inherent cost advantages as a 'follower' rather than an 'innovator.'

2. Those aren't apples to apples comparisons on compute-per-dollar efficiency.

3. You fool, you absolute buffoon, don't you understand? Sovereigns can simply replicate DeepSeek's efficiency gains and deploy them across their larger compute footprints.

To which I say: fair. Most articles I've read up to now stop at this point. Let's dig deeper.

1. Second-mover advantages

Truth be told, followers do have an advantage - Epoch AI has pointed out the physical compute required to achieve a given level of performance drops by 3x every year, so for a 10x training cost reduction, about a 14 month lag time seems appropriate20. Except DeepSeek-V3 caught up to Llama 3.1 405B, the prior OSS state of the art, in just 5 months - less than half that estimate. That's a significant bucking of the trend.

Further still, open-source models tend to lag their closed source competitors by about one year. The going assumption is closed-source research labs have access to more compute and data than their open source counterparts (Meta's contributions notwithstanding21), and it takes time to replicate those models in the open.

But even OpenAI's o1, considered a breakthrough in reasoning capabilities and a potential "path forward" beyond traditional scaling limitations2223, wasn't immune to the DeepSeek effect. Their open-source reasoning model, DeepSeek-R1, met or surpassed o1's performance just 4 months later.

So yes, DeepSeek enjoyed the fruits of second-mover advantages as Meta, X, Cohere, and others have. But where all the others took about a year to catch up, DeepSeek did it in a third of the time.

2. Compute-per-dollar efficiency

Given the effective compute efficiency curves I just mentioned, we know training costs become cheaper overtime for a given architecture and training recipe. So it's unfair to compare GPT-4 and Gemini Ultra costs _at that time_ to DeepSeek's today.

Fortunately, @arankomatsuzaki and @ldjconfirmed [have done the work for us](https://x.com/arankomatsuzaki/status/1884676245922934788?s=46). Here's the estimated training costs for a benchmark set in 2025 prices:

For the curious, the researchers provide an online calculator for you to run the numbers yourself.24

"Ah," you might say, "but DeepSeek's model family only has a 6x cost advantage over GPT-4 and Claude-3.5!" And again, I say: fair. But that's not the point. OpenAI and Anthropic have already booked the capital expenditures required to train those models - the money is out the door. And for the upcoming 100K H100 colossal language models, those costs are expected to be higher, astronomically higher.

What I'm saying is, the apples-to-apples appeal may be important academically, but I only minored, not majored, in economics. I care a lot more about the all-in cost.

3. Compute economies of scale

Sovereigns absolutely could replicate V3 and R1's algorithmic improvements in their own systems, and I hope they do, users would be better off for it. But they've already spent the money they needed to scale up their inference businesses and training clusters, and need to recoup their capex somehow.

Under normal circumstances, that might mean artificially higher API prices in the initial months post-rollout, which 'magically' come down as equivalent caliber models diffuse into the market.

DeepSeek may have just nuked that chance, leaving OpenAI (and others) holding the bag. At the time of writing, OpenAI is charging $15 / 1M input tokens for o1 access, and DeepSeek-R1 is charging $0.55 / 1M tokens.

Expect these margins to narrow soon, but sovereigns need to differentiate themselves and fast.

Recap

Knowing what we know now, it's worth re-examining OpenAI's trillion-dollar chip ambitions. Were they driven purely by market demand, to more democratically diffuse access to intelligence? Or were they solidifying a competitive moat through artificial scarcity, maintaining a leading edge through anti-competitive means?

Wherever OpenAI is on the spectrum, DeepSeek severely upset the apple cart. Sovereigns (and their backers) have gone all-in on an extravagantly expensive strategy, and may have cornered themselves into a trap. Without serious differentiation, their lead may not last long enough to recoup their capex. That threatens the multi-billion dollar ecosystem of research labs, hyperscalers, energy companies, venture capital, and national industrial policies that are betting it all on a centralized, compute-intensive future.25

2. The Efficiency Paradox

This was a major shock to the markets, which in the short term clearly wondered whether low-cost compute would nerf demand for high-performance chips and components. Fortunately, as a general-purpose technology becomes more efficient and accessible, aggregate demand increases as the market finds more uses for it. The commoditization of intelligence will likely lead to accelerated adoption across multiple industries, rather than a market contraction.

Immediate Fallout

This faster-than-expected moat erosion has incumbents scrambling to figure out a response26, and the markets took notice.27 Headlines screamed about the bursting of the hype bubble, as investors processed what DeepSeek's efficiency gains might mean for the ecosystem.28

NVIDIA, the primary beneficiary of sovereigns' compute addiction, booked the biggest one-day loss in history - close to $593B in market cap, a 17% hit29. Semiconductor and hardware component stocks like Broadcom, Marvell, Micron, and others also took double-digit hits.

More surprising to some were the unexpected beneficiaries. Apple, criticized for its conservativism (and borderline braindead Apple Intelligence rollout), saw its stock price actually appreciate by a few points. While surprising to some, Apple has been pursuing a very different strategy, emphasizing edge inference and custom silicon for their own devices. To Apple, it really doesn't matter what models people prefer to use - what matters is where the models live and run.

signüll on twitter had a great breakdown:

We're only a few days out from the Monday correction, but I expect Apple will continue to sit pretty for a while longer, thanks to an economic principle that's picked up again in online meme circles - Jevons Paradox.

Jevons Paradox

In 1865, English economist William Stanley Jevons observed that across a wide range of industries, the increased efficiency of coal use led to its increased consumption. The less it cost to use, the more uses people found for it, roughly maintaining or growing aggregate demand.

That's it. It's not complicated. There, saved you a lengthy twitter thread.

Economists have observed the same phenomenon in hybrid engines and driving distances, expanded highways and traffic times, bandwidth costs and internet usage, and many other examples.30

Now contemporaries are applying the same logic to intelligence as a commodity - the more we have, the more we want. If true, what tailwinds are contributing to an accelerating intelligence' diffusion?3132.

Tailwinds for intelligence efficiency gains

First, let's look at the data. Epoch AI's analysis argues that, in terms of raw resources required to achieve a given level of performance, 2/3rds of the gains are attributable to more scale (chips), and 1/3rd to algorithmic progress.

It's either better algorithms, more hardware, or better hardware. And while up to now we've focused mostly on training costs (building intelligence), the much larger category of spend is actually inference costs (using intelligence).

1. Better algorithms

Foundation model developers that can mimic sovereigns' performance capabilities at a fraction of the price are well-positioned, as businesses question the sticker prices they've been sold up to now. Fastino is one such company, whose novel architecture is so ridiculously efficient that it yields 2,000+ tokens per second on a CPU. (full disclosure, I'm an early investor).

Cartesia is another example that use efficient alternative architectures to meet or eclipse dense transformers' performance on various tasks, with the company claiming its state-space models (SSM) are efficient enough to run just about anywhere.33 Coincidentally, SSMs use similar memory compression techniques to DeepSeek's MLA, summarizing context to enable efficient, long-range memory.

2. Better hardware

The same logic applies to semiconductor efficiency gains with companies like Positron, which I'm also an investor in. Assuming token generation throughput parity (and model compatibility), performance per watt improvements are likely to yield increased demand as a multiple over the NVIDIA baseline.

Datacenters will need to expand to keep up with compute demand, but newer Blackwell chips from NVIDIA are expected to cost more, not less, in opex, due to critical design decisions that raise electricity consumption and cooling needs.34 With downward pressure on compute costs, and physical limits on electricity consumption per rack, datacenter customers have to improve margins where possible.

3. Efficient energy systems

To get around these problems, datacenters can do quite a lot in optimizing their operations before making any chip decisions. They have physical limits they have to abide by. Options vary for new builds vs retrofits, but some contributors I'm seeing include:

1. Liquid immersion cooling. Highly efficient method of cooling datacenter equipment. Strictly on a square-foot basis, reduces datacenter footprints by up to 10x, with the input costs being the container and the coolant, that can upgrade the power usage effectiveness (PUE) of datacenters from ~1.55 down to as low as 1.05

2. Datacenter power and energy management software. This market is relatively immature, and most DCs have been caught flat-footed as they've mostly rolled their own custom software up to now. It hasn't been battle-tested for intense AI workloads. Companies like CentralAxis that can provide mature, scalable, and user-friendly solutions are well-positioned.

3. Natural gas suppliers. Everyone in my circles won't stop talking about nuclear energy, which I'm all for and excited about, but the demand exists NOW and we don't yet have nuclear capacity to absorb it. Fortunately, existing natural gas suppliers all over the Permian are already raking it in with datacenter end-customers, and I expect that only to increase overtime.

4. Edge inference and model commoditization

Finally, as we saw with Apple's stock performance, companies positioning themselves for a world of efficient, widely-available models - whether through edge computing, optimization layers, or deployment platforms - are likely to benefit from increased AI adoption even as sovereign margins get compressed. Model commoditization, rather than artificial scarcity, is a more anti-fragile bet.

Chip Market Implications

With all that said, obviously more chips to use means greater capacity for intelligence diffusion. But as for specific winners here, the hardware story is more nuanced.

1. Hardware baselines

NVIDIA remains the only serious game in town for high-performance training hardware, though with a twist I didn't expect. Historically, their chips configurations have been inefficiently utilized in inference, using only about 30% of their memory bandwidth at peak35.

But their new GB300 chip seems purpose-built for reasoning models, which require more memory bandwidth for longer chains of thought. If reasoning models like R1 virally spread at DeepSeek prices, inference demand may actually spur even more demand for NVIDIA's latest chips.36

2. Inference optimization

Aside from performance-per-watt optimization via Positron, customers may want to optimize against speed and latency above all else. Hyper-fast inference chips from companies like Groq and Cerebras are valuable... but need to make economic sense on a total cost of ownership basis. Groq's design decisions for its LPU chips optimize for speed, but severely sacrifice memory capacity, requiring 10 racks to deploy a 70B model that would otherwise fit on a single GPU373839.

The question remains whether that strategy makes sense for diffusing into the entire datacenter market, or if the only path forward is vertical integration, which could be another version of a sovereign trap.40

3. Infra software

The infrastructure software layer is where we start to see clearer winners and losers. NVIDIA's years of investment in CUDA has created an ecosystem and user experience that competitors struggle to match. AMD's story is instructive - despite MI300X being on par with (or better than) NVIDIA's H100/H200 line, AMD's ROCm dogwater software stack prevents them from realizing those hardware advantages, and they've made no successful effort in the past year plus to close that gap.4142

Having said that, it's worth noting that DeepSeek did collaborate with AMD in developing DeepSeek-V3, which may be a soft endorsement of their Instinct chip line and software stack.43

Basically, what I'm saying is that a rising tide does not lift all boats here. Some boats have holes in them. They need to be patched up, and until then exist on an entirely different demand curve.

The Real Paradox

I'm crudely bisecting the AI market into 2 sets of players: those accelerating its diffusion, and those rent-seeking through artificial scarcity.

DeepSeek's breakthrough demonstrated that the moats closed-source model had are not as defensible as sovereigns believed. They may yet adapt by adopting these efficiency gains, and closed-source models do still dominate enterprise workflows for the time being. But DeepSeek's cost trajectory suggests a reckoning is inevitable. The market's reaction shows that investors are starting to recognize this new reality.

The real efficiency paradox is this: do sovereigns have the cojones to disrupt themselves? Or will they cry foul and double down on protectionism?

3. The Technopolitical Reckoning

American Sovereign Responses

Predictably, American stakeholders have responded with a mix of hostility, defensiveness, and existential ruminating. I mean, it's hundreds of billions of dollars on the line here.

"They cheated"

OpenAI directly accused DeepSeek of using OpenAI models to train their own, by training based on outputs from ChatGPT, which would give DeepSeek an unfair advantage. No evidence has actually be presented yet, just conjecture, and the reactions from Twitter have been... less than sympathetic.

"Actually, we hate open source"

Meanwhile, venture capitalists with large stakes in American AI companies are reportedly in "crisis mode", as this breakthrough threatens the competitive advantage of their portfolio companies. Some are questioning whether models should be allowed to be open-sourced at all... which is a ridiculous take.

Alleged export control violations

To tie this all together, the "DeepSeek is a Chinese psyop" conspiracy (disclosure: the tweet author's dad has a large stake in OpenAI) only works if the CCP secretly granted DeepSeek access to advanced chips China procured through illicit means.

Alexandr Wang, CEO of Scale AI, made one such (unsubstantiated) claim, that DeepSeek has access to 50,000 advanced AI chips that it shouldn't have, given US export controls. Dario Amodei (Anthropic) and Elon Musk have also been supporters of this theory.

I'm not naïve - there are certainly shady dealings happening in Southeast Asia to evade export controls.44 That said, in true open-source spirit, DeepSeek released its complete training recipe for its models, and HuggingFace is already hard at work rebuilding it from scratch... so if it truly would take 50,000 H100s to build it, we'll all know soon enough anyway.45

If it were actually a conspiracy, that would be like handing over a smoking gun. A couple weeks of market confusion, maybe, but no lasting damage. Just a really dumb prank.

Altogether, these gripes betray a deep misunderstanding of how open source research works. I provided rich citations for DeepSeek's architectural decisions and training recipes explicitly to point out that _none of these techniques are new or unheard of_. This is just the first time someone has put them all together. It's a Sputnik Moment, sure... but as far as China's concerned, their own Sputnik Moment was two years ago when ChatGPT was released.

The China takes

In contrast, the response from Chinese media, AI executives, and netizens has been overwhelmingly positive, tinged with national pride and interesting fanart. I'm especially grateful to Jordan Schneider and his team at ChinaTalk (longtime listener) for collecting some of the best quotes from first-party sources.46

China Daily declared, "For a Chinese LLM, it's a historical moment to surpass ChatGPT in the US."47 Daily Economic News echoed this sentiment, stating, "Silicon Valley Shocked! Chinese AI Dominates Foreign Media, AI Experts Say: 'It Has Caught Up with the U.S.!'"48

Feng Ji 冯骥, founder of Game Science (the studio behind Black Myth: Wukong), called DeepSeek "a scientific and technological achievement that shapes our national destiny (国运)."49

Zhou Hongyi, Chairperson of Qihoo 360, told Jiemian News that DeepSeek will be a key player in the "Chinese Large-Model Technology Avengers Team" to counter U.S. AI dominance.50)

A closed-door session between Shixiang 拾象, a VC spun out from Sequoia China, and dozens of AI researchers, investors, and insiders provides an even richer view on DeepSeek's market entry.

Key takeaways from the discussion include:

1. **Celebrating efficiency as a source of resilience.**

   "How Chinese large-model teams use less computing power to produce results, thereby having some definite resilience — or even doing better — might end up being how the US-China AI landscape plays out in the future."

2. **Addressing the efficiency paradox head-on.**

   "In the long-run, questions about computing power will remain. Demand for compute remains strong and no company has enough."

3. **Open source controls the margins of the whole market; the US discovered that China is not two years behind, but 3 to 9 months.**

   "If the capabilities of open-source and closed-source models do not differ greatly, then this presents a big challenge for closed source."

4. **This, plus the lack of a good business model for AI labs, heightens the commoditization risk.**

   "The business model of AI labs in the United States is not good either. AI does not have a good business model today and will require viable solutions in the future. Liang Wenfeng is ambitious; DeepSeek does not care about the model and is just heading towards AGI."

5. **Differentiation has to therefore come from vision, not from technology scarcity.**

   "China is still replicating technical solutions; reasoning was proposed by OpenAI in o1, so the next gap between various AI labs will be about who can propose the next reasoning. Infinite-length reasoning might be one vision. The core difference between different AI labs’ models lies not in technology, but in what each lab’s next vision is. After all, vision matters more than technology."

4. The End of the Sovereign Era?

To put it bluntly, "the global diffusion of AI is now irreversible" (full translation here).

• HuggingFace is already hard at work replicating R1 to prove, without a shadow of a doubt, its validity

• Community researchers are sharing resources for running full, undistilled versions of R1 on prosumer hardware that is highly unlikely to fall into an export ban

The nail seems to be in the coffin for the sovereigns' initial strategy. Don't worry, they've got big treasuries and have time to adapt, but the model is not the product, and never has been. That said, they've been inefficiently burning capital at a rapid clip, and will likely use the market moment to double-down with additional funding.

The stark contrast between the American and Chinese responses to DeepSeek's breakthrough reveals deep geopolitical fault lines we already knew were there. Americans responded with skepticism and allegations of foul play, while the Chinese AI community (and frankly, a lot of the open source community) celebrated DeepSeek's achievements as a validation of their own strategic vision.

Its exciting to see open source provide such a strong counter to power ossification, and I strongly believe this will be a huge boon for boosting AI supply chains and markets in the aggregate. Efficient training techniques, edge inference, and energy-efficient datacenters all point to model commoditization as an inevitability, and these efficiency gains will drive demand to new heights.

Critics might argue my optimism underplays real moats that sovereigns possess: proprietary data, regulatory capture, or ecosystem lock-in. These are valid concerns. But I still believe efficiency and community momentum will outweigh them overtime.

In the new era of AI diffusion, I believe winners will be those who embrace intelligence' commoditization and find ways to create value on top of it. If we get this right, we can uplift the entire economy... rather than a few magnificent slices of it.

23

The Obsolete Newsletter

Is Deep Learning Actually Hitting a Wall?

[EDIT: I should have credited Gary Marcus for coining the term “deep learning is hitting a wall" back in March 2022. I didn’t actually realize it had originated entirely with him, given how much it’s entered the lexicon. He argued that LLMs were hitting diminishing returns in April 2024, which was an input into my similar prediction in June. He also had…

Read more

a year ago · 48 likes · Garrison Lovely

46

ChinaTalk

DeepSeek: The View from China

Before December 2024, DeepSeek was rarely mentioned in China’s AI community. With the release of DeepSeek-V3 and the reasoning model R1, Chinese media and AI researchers started to ask the same question as their American counterparts: Who is DeepSeek and how should we feel about them…

Read more

a year ago · 134 likes · 5 comments · Jordan Schneider, Irene Zhang, Angela Shen, and Yiwen

“A word is worth a thousand vectors.”

Given the choice, would you rather be a well-trained expert who stays in their lane, or have the raw intellectual horsepower to master anything?

Back when I took CS224N at Stanford, before the transformer era, Prof. Chris Manning blew our minds with a simple demo. Using an embeddings model, he showed us how semantic concepts like “royalty” or “gender” could be reduced to arithmetic operations in vector space, e.g. “king is to queen as man is to woman” (woman = queen - king + man). We spent the next hour testing increasingly complex analogies across many domains - book genres, clothing styles, occupations - and in each one, the model generated outputs that seemed, well, weirdly intuitive.

This was a glimpse into how models perceive language, not as siloed domains, but as an interconnected web of concepts, with clusters and intersections that we can’t naturally envision… or as Chris Moody so pithily put it, “a word is worth a thousand vectors.”

I thought about this a lot as last year’s meme of ‘domain-specific models’ took off. What constitutes a domain that deserves a specific model? Are the differences we perceive between subject matters so complex that a fine-tuned model is typically the best solution?

This question is at the heart of many Fortune 500 companies AI strategies, and I’m not sure they’re going to like the answer.

Your data (probably) doesn’t matter

Unfortunately, fine-tuning your own <Company Name>GPT might be a waste of time.

My consultant friends wouldn’t want me telling you this, but beyond some baseline of model intelligence, your data probably doesn't matter that much. While fine-tuning can be useful for consistent output formats or learning certain styles, most businesses' datasets aren't unique enough in language vector space to warrant the investment.

Take BloombergGPT, for example. Despite costing $10M to build and being trained on 51% proprietary Bloomberg financial data, its initial performance edge was quickly eclipsed by GPT-4 without the latter using any special Bloomberg sauce, just a larger parameter count and more compute.

Harvey may have followed a similar playbook. Raising $21M just a month after BloombergGPT’s release, with nearly half the headcount comprised of expensive lawyers, and insisting on a hefty user-based pricing model ($1200/seat/year, 100-seat minimum), they’re betting big on fine-tuning, prompt engineering, and negotiated lock-in. But as a viral teardown by Edward Bukstel suggests, chances are Harvey spent a ton of money fine-tuning a model that newer ones already outpaced with RAG and/or agentic systems.

Might be reading into the tea leaves too much, but this sounds to me like an incredibly expensive endeavor fighting last year’s battle.

In both cases, The Bitter Lesson rings true - the biggest breakthroughs in AI tend to come from simple, scalable architectures, vs. bespoke tweaks that yield short-term improvements.

Models as utilities

Instead of building bespoke models that introduce complexity, simplifying how we interact with other peoples’ models is the bigger prize.

When Andrew Ng talks about how “AI is the new electricity,” here’s how I think about it: We need electricity for lots of things. Kitchen appliances, lighting, television, tons of applications. But ultimately, does my nephew care that our TV uses electricity which comes from a utility that uses 100% renewable energy? No, he just cares that he can watch Bluey on it.

However, as the one paying for the electricity, I definitely care about variables like price, availability, and climate-friendliness. I fortunately come from a state where consumers can pick their provider based on these variables, and that transparency creates a more competitive market that ultimately benefits me, the consumer. Although I now live in a state where this choice isn’t available, and outside of regulatory action, PG&E’s monopoly gives it little incentive to do anything besides raise rates, even in bankruptcy.

Right now, there are dozens of competing large models, each hosted by dozens of token-as-a-service providers, all of whom are competing with each other based on price, quality, climate-friendliness, data privacy, many other variables to win your business. This is a hugely beneficial time to be building if you know how to play these providers off each other.

You’re welcome to lock in to a single provider who has your preferred ecosystem, but you’re SOL if the price jacks up or there’s an outage. You’re also perfectly capable of setting up your own server, just like you could install solar panels on your house, but that carries maintenance costs many would rather not worry about.

To my nephew, playing Bluey is the TV’s only job. He’s not going to be very impressed with my off-the-grid solar array if it’s cloudy for more than a few days.

Routers are an ROI multiplier

So if sinking millions into fine-tuning models might not work for most businesses, and you want to avoid the hostile user-based pricing, what’s the alternative?

Model routers like Martian and OpenRouter are promising, anti-fragile solutions which deliver all the benefits of a competitive model marketplace with minimal complexity.

Rather than getting locked into a single model and praying your vendor doesn't jack up prices (or get dumber), routers use special techniques to dynamically send prompts to whichever model is best suited for it. To be clear, this is much more fine-grained than saying “Model X is best for <Domain Y>” - this is task-specific prompt → model routing.

The net effect is that you get orders of magnitude reductions in inference costs while enjoying the same (or better) quality results that larger, expensive models would return.

Under the hood, routers are built on datasets of pairwise comparisons between model outputs in response to a query. Prompts are assigned a high-level category, such as asking_how_to_question or text_correction, and human raters (or GPT-4 in some cases) select the best output.

Overtime, a representative corpus of qualitative comparisons between models is built, which can then be used to categorize prompts, then route those prompts to the best performing model for that category. Platforms like Chatbot Arena are great for this, though its category granularity could be a bit better.

However, that’s just for qualitative comparisons. Deployment teams may want to optimize for speed if latency is a success metric, or cost if spend reduction is the primary goal.

Many organizations (including OpenAI, if the rumors are true*) leverage model routers internally to optimize their inference costs; they route “simpler” requests to smaller models that are much cheaper to run (but lack sophistication), and route more complex, thoughtful prompts to more expensive models better equipped to provide helpful responses. These dynamic systems can significantly outperform any single-model system on multiple dimensions.

* This is at the center of the “GPT-4 is getting dumber” meme. At a new model’s launch, OpenAI would want all prompts to deliver high-quality results to connect its brand to consistently high intelligence. But not all prompts are created equal. At full optimization, dumb questions get <proportionate> answers from lower-parameter model variants.

Getting started

Whatever system you end up using, you should definitely be capturing user prompts, outcome ratings, and other metadata for your own analyses later. This will help RLHF better routing systems over time, tailoring them to your specific application, and contributing to defensibility against unoptimized wrapper and single-model competitors.

You may also find certain tasks are consistent and frequent enough to justify creating your own small language model (SLM) to serve them. This targeted approach can compound inference cost savings even further.

If you're interested in exploring model routers for your own applications, the team at Pulze AI has generously open-sourced their intent-based router, comparison dataset, and intent embedding model, which are great starting points for teams looking to roll their own solutions.

Wrap-up

Given that “almost everyone is losing money on LLM inference,” the anti-fragile approach model routers offer is pretty refreshing. While everyone else seems to be rushing to build ever-larger generalist models or domain-specific models, you might want to leave that particular race to the well-funded labs draining VCs dry.

Model routing isn’t just a technical solution - it’s fundamentally democratizing, as it encourages more, not fewer, models to be built. Instead of raising billions for a generalist model only to fail, as Inflection did, devs can build niche specialist models that plug into a routable model marketplace. If they’re objectively better, they’ll be used and yield a profit.

Make no mistake, that doesn’t mean every specialist model constitutes building a venture-scale company around it. And I don’t think “LLM for Finance” or “LLM for Legal” counts as “specialized.” But in aggregate, a constellation of LLMs, SLMs, and task-specific agents, stitched together by a robust routing layer, starts to look an awful lot like the future of the field.

For more on routers, check out Cerebral Valley’s Martian deep dive into Martian, where founders Yash and Etan share their inside story.

The elephant in the boardroom

As AI hype continues to rip, a nagging question is popping up in boardroom circles - are we actually seeing real impact from our AI strategies?

A few weeks ago, we co-hosted an event with Reinvent Futures here at SHACK15, which brought together leaders from the tech and sustainability spheres. Matt Kropp, CTO of BCG X (BCG’s tech build & design unit), kicked things off with a pretty clear signal from their clients: “we're now getting to the point where there's some question about whether we're actually getting impact.”

The surveys definitely back this up, and I’m personally hearing this more and more among consumers and enterprises alike.

• Consumers are feeling the squeeze of subscription fees, as research labs start to lock the best models behind paywalls.

• Enterprises are still feeling pressure from their boards to adopt GenAI tools (89% say AI and GenAI are in top 3 tech priorities for this year), but struggle to figure out which ones to use.

• Even with a hefty "enterprise" discount, no sane CIO or CFO is going to greenlight a plan that charges $240 per user per year for access to OpenAI's finest, because they first want to know… what will I actually use this for?

LLMs are becoming commoditized, so teams building AI products they want businesses to use need to focus on building complete solutions solving concrete problems. Simply participating in the LLM gold rush isn't enough anymore, because the model is not the product.

Model-centric missteps

Over 70% of people haven’t even used generative AI tools yet, let alone know the difference between distinct models. While I’m grateful for leaderboards from Stanford, HuggingFace, and most recently Scale, the benchmarks we use are somewhat academic, or at least lack the specificity one could use making a business decision about which model to go with.

The first chapter of the AI arms race has had VCs shoveling money towards teams building model-centric research labs. These models are great across a range of tasks, but lack significant differentiation from one another, and may eventually fall short of investor expectations. Inflection’s implosion and Stability AI’s public struggle demonstrate the eventual risks of this model-centric approach.

Inflection in particular shows that war chests are not a guarantee of product-market fit. Despite raising $1.3B last year alone, they were late to an already crowded party and failed to gain traction. Their Pi assistant, while friendly and acclaimed by users, ultimately may not have been distinct enough from the ChatGPTs, Claudes, and Geminis of the world to warrant much attention.

Similarly, Stability, despite hitting unicorn status in 2022 thanks to the buzz around Stable Diffusion, failed to build a compelling business around their family of open-source models. The financial situation became pretty dire: in Q1 2024, they reportedly lost $30M on a meager $5M in revenue (with nearly $100M in unpaid cloud compute bills). So despite great contributions to open source, they didn’t create a revenue model which could support demand for its service, hampering their ability to raise more cash.

Behold “AI Operating Systems”

So if the model-centric approach is losing its luster, what are market leaders doing instead? They’re shifting their focus from best-in-class general models to tightly integrated ecosystems.

OpenAI is doubling down on the sci-fi dream of an always-on, hyper-personalized AI companion (Her, but IRL), while Google and Apple are leveraging existing product and device ecosystems to make their models indispensable across every touchpoint. Meanwhile, Microsoft is betting on “Copilot+” branding as a value-add for its PC and productivity software businesses.

Each player is trying to weave LLMs into a stickier, more defensible product strategy as the go-to “AI Operating System.” Google, Apple, and Microsoft have extensive network effects they can already leverage for lock-in, and while OpenAI is playing catch-up here, their market share with ChatGPT has so far been substantial enough to dictate the pace of development.

But while this AI OS approach may be useful for consumer lock-in, it doesn’t solve the primary problems businesses have integrating models into their own products and services. Businesses aren’t buying models or ecosystems - they’re “hiring” them to perform specific jobs-to-be-done. And they need the flexibility to choose the best, fastest, and cheapest option for each job, not to be beholden to a single provider.

Unsustainable inference costs

Even the most seamless ecosystem will struggle if the economics don’t add up. The cost of at-scale inference is staggering, and it doesn’t seem like any vendors serving models today are making a profit.

This introduces some weird market dynamics.

Token-as-a-service providers are playing by a predictable growth-at-all-costs playbook: sacrificing near-term profitability for user acquisition, then making it up on volume once you’ve cornered the market. So for the time being, no one really knows what the willingness-to-pay is, because token prices are being kept artificially low.

As we’ve seen with Inflection and Stability, that strategy may have its limits, especially if they fail to distinguish their open offerings from paid ones. In turn, these failures may cause disillusionment, consolidation, and upward pricing trends.

At first glance, this could seem like a good thing (VC-subsidized prices, yay), but there are significant risks. Basing your own product strategy on unsustainably low prices can cause a severe hangover to your cost structure if and when those prices eventually do rise. Moreover, vendor lock-in carries the risk of your chosen model suddenly becoming unavailable or prohibitively expensive if they fail to hit profitability. You need to be careful not to anchor your product too much on the idiosyncrasies of a single vendor’s model family.

A valid counter would be that Moore’s Law is naturally driving down inference costs overtime, hence the “make it up on volume later” meme. While this is true that there’s about a 3x reduction per year in physical compute required to hit a given performance target, it’s also true that training costs for frontier models are rising by a similar rate at 3.1x per year. So if you want last year’s model, that will be cheaper, but if you want the latest and greatest, you’re still going to have to pay.

In short, the current economics of AI inference are concerningly unstable, and businesses should be strategic in navigating that when building products that use external models.

Recommendations

To recap, the path to real impact and ROI remains unclear for many businesses under pressure to adopt AI in their products and services. So, what’s a savvy team to do when crafting their strategy? Two options stand out in contrast to the model-centric approach:

1. Highly optimized token-as-a-service providers like DeepInfra offer a compelling value proposition, by focusing solely on efficient model hosting and serving. By comparing models and hosts on independent leaderboards like Artificial Analysis, businesses can find the best bang for their inference buck.

2. Model routers like Martian and OpenRouter take this a step further by dynamically allocating queries across multiple models based on cost, speed, and performance. They’re constantly tracking the model tokenomics and quality with insane prompt-specific granularity, which lets customers tap into the best available models for any use case without vendor lock-in.

Of course, each solution has their tradeoffs, which we’ll dive into next week with a tactical breakdown in part 2. We’ll explore, model routing, fine-tuning, leaderboarding, and more practical takeaways for AI strategies. Until then, I’d recommend reading Cerebral Valley’s deep dive with the Martian team to get familiar with what they’re up to.

But for now, the key takeaways are this: the model is not the product, and the businesses that will thrive are those that match the right model to the right job at the best price.

The first wave under-delivered

When ChatGPT debuted, a friend raved to me about Sudowrite, a thoughtfully designed fiction-writing app created by James Yu and Amit Gupta. Sudowrite leverages LLMs to help authors stay in flow, organizing LLM calls into purpose-built features for tasks like world-building, pacing, and outlining. With a vibrant user community stress-testing new features, millions in annual revenue, profitability, and fewer than 10 employees, Sudowrite exemplifies what’s possible when generative AI products are built with deep user empathy and domain knowledge.

At the time, I thought this foreshadowed a wave of similarly well-designed AI products across many industries.

A year and a half later, that prediction remains mostly unfulfilled.

Many first-wave AI startups have failed to deliver little more than thin UI wrappers around generic LLMs. I’ve critiqued Jasper before, and it looks like Tome (text-to-powerpoint) may be next on the chopping block. Turns out defining your ideal customer profile and product is crucial before hiring sales reps.

This trend extends beyond software. First-gen consumer AI hardware like the Humane AI Pin and Rabbit R1 have been dead-on-arrival, tethered to the latency, limits, and hallucinations of off-the-shelf LLMs. As MKBHD (Marques Brownlee) put it, they’ve failed to deliver compelling (or even passable) user experiences.

This “wrapper” or “copilot” design philosophy is at the center of my beef with application-layer startups, and why I generally avoid investing in them (with limited exceptions I’ll cover at the end of this post*).

Most seem to follow the same playbook - inject someone else’s LLM into a user-facing SaaS model, christen it a “copilot”, and ignore competitive analyses because they misread Christensen’s Innovator’s Dilemma. At its current stage, generative AI has been more of a sustaining innovation for incumbents, not a disruptive one they’ve been reluctant to adopt.

Yes, I’m aware many copilots claim to be much more than a wrapper. Some emphasize that they use fine-tuned, domain-specific models that outperform generic LLMs. However, even category leaders building these kinds of products can be paper tigers. Case in point - Harvey, the buzziest player in AI x legaltech.

Harvey: Legal Jasper?

I’m going to go on record and say what I’m seeing with Harvey smacks a great deal like Jasper last year - Harvey’s last raise pegged them at $700M+, but leaked screenshots of their tool leave a lot to be desired.

First, let’s understand how Harvey views itself in the AI x legaltech landscape.

Harvey describes itself to investors and customers as an application layer that is “foundation model agnostic.”

They highlight that a team of in-house attorneys and experts “structure application and model calls” to create a best-in-class legal LLM. Off the bat, this sounds like an expensive consulting agreement for lawyers to do prompt engineering.

They also have no plans to provide real-time access to the internet for legal research, because their focus is on not providing “wrong or tainted responses.”

Let’s see how that philosophy manifests into actual product.

Lacking basic integrations

Harvey’s legal research capabilities are slim. The only searchable source is SEC filings - no US case law or EUR-Lex integrations, despite APIs for these services being freely available.

As an attorney, when conducting legal research you’re looking for relevant cases, rulings, and written opinions from the court on your area of interest. This helps you vet whether there’s established precedent in your favor or which arguments are most likely to work.

A baseline expectation for a production-grade legal research app is that generated outputs map to real cases. These integrations would ground any generated cases in discrete outputs, and a well-designed system would assert the outputs map to real cases before including them.

To be sure, LlamaIndex built secinsights.ai (RAG for SEC filings) in September 2023 as a feature-complete demo of their toolkit - really impressive work for LlamaIndex, and a great open-source contribution. But what has Harvey actually built on top of this?

CYA

Very little, it would seem. The “Assistant” tool is a bare-bones LLM wrapper with lengthy disclaimers about how LLMs hallucinate and you shouldn’t take anything Harvey says seriously.

Neither a wall of disclaimers nor a black box “fine-tuned by the best lawyers” instills confidence. Harvey may argue their in-house experts can prompt-hack a robust model for legal work, but don’t expect customers to trust an opaque, unverifiable system, no matter how well-built it claims to be.

If your Harvard-trained associate handed you a legal brief with a note saying “but idk I make stuff up sometimes,” you’d fire them on the spot. Or in this real-life case, you’d so thoroughly embarrass the lawyers who submitted it that you wouldn’t even need to disbar them.

Let me be clear - in domains where the cost of errors are severe (e.g. legal, finance, accounting) or even life-threatening (e.g. healthcare, pharma) you absolutely should not design a system solely around a zero-shot LLM.* LLMs are stochastic by nature, and need to use external resources to provide verifiable, discrete responses.

What you can do instead is create a modular system with explainability, grounding, and auditability built-in - something more akin to an “agent-based” approach.

* This is part of a greater degree of skepticism I have with “domain-specific models,” which is a topic for a future post - the observation that beyond a certain level of model intelligence, your domain-specific data does not matter.

Agents “Do No Harm.”

While Harvey bets on an offline, domain-specific model, Hippocratic AI has the opposite tack. They break physician-patient interactions into individual “specialist models,” (i.e. ‘agents’ or ‘modules’) narrowly scoped to address individual workflows like checklists, medication, labs & vitals, etc..

It’s a fantastic example of an agentic system in action, especially notable for its clout and adoption in such a sensitive domain - healthcare - where the cost of a mistake can be deadly.

Listen to this AI-patient conversation at 2x speed, and observe the agents in action:

Bonus points for anyone who heard “delve” as a clear AI giveaway.

• The primary agent for Linda prioritizes good bedside manner, building rapport, and providing empathetic guidance

• Linda’s “checklist specialist” balances extracting info and moving the conversation forward

• For each checklist item, Linda has a specialist “support agent” for tasks like reviewing meds, answering insurance questions, or giving nutrition tips

• These interlinked agents share data and RAG into specific knowledge bases

Hippocratic stress-tested with 1,200+ clinicians in simulated patient-actor conversations, and are achieving verifiable parity (or supremacy) on all 5 of the surveyed dimensions: bedside manner, patient education, conversation quality, clinical readiness, and medical safety.

I’ll dive deeper into agentic systems design in a future post, but the key takeaway is you can mimic these techniques for building sophisticated AI that consistently works, even in the most sensitive domains. I don’t think customers would accept less, and neither should investors or founders.

What’s next: beyond copilots

I mentioned at the top of this post that as a general rule, I don’t like to invest in application-layer companies. Here are the exceptions to that rule:

1. You’re selling into a highly sensitive regulatory domain. This makes it harder to sell into, but drastically improves customer stickiness, AND

2. You (the founder(s)) have unfair channel access / distribution into these regulated customers, AND

3. Because you know these customers so well, you’ve designed a task-specific agentic system that meets or exceeds human-level performance on each of the scoped modules

Application-layer companies that meet these criteria have a better shot at building defensible, long-term value. We need modular, explainable systems that leverage generative AI while grounding it in domain knowledge and rigorous evals. Agentic systems, à la Hippocratic AI, are great examples of this. Looking forward to seeing many more like it in Chapter Two of this cycle.

What’s Happened

In January, I wrapped up a multi-year stint working with Andrew Ng and a team of talented builders at his venture studio, AI Fund. Together, we built 9 companies across various domains, including Workhelix, a GenAI readiness assessment platform for executives. We developed and shipped AI-powered products into many different verticals before and after ChatGPT’s explosive release.

In that time, I learned that:

1. My skills and expertise were best utilized collaborating with teams building foundational tech like infrastructure, agentic systems, developer tools, and frontier AI research.

2. I believe we’re still too early to crown winners in the application layer, as there are many product categories founders are shoehorning AI into that won’t even exist three years from now.

Just as Uber’s success relied on “existential technologies” like mapping systems and cheap mobile bandwidth, AI and its supporting systems are existential technologies that will spawn entirely new categories. “Electricity,” as Andrew puts it. We still have a lot of work to do building out the power grid.

After meeting tons of founders, judging hackathons, and even participating in a few, I co-founded an angel fund called SHACK15 Ventures with my partners Jørn Lyseggen, Bogdan Cristei, and Ashwin Ravichandran. Operating out of the eponymous social club SHACK15 atop San Francisco’s iconic Ferry Building, we’re working with a vibrant community of investors, entrepreneurs, and hackers, including Cerebral Valley, known for their exceptional AI hackathons. We’ve already made our first few investments and will have more to share in the near future.

To stay updated on Cerebral Valley, subscribe to their lu.ma page.

Machine Yearning

I started this blog back in 2021 as an attempt to organize my thoughts on the emerging AI industry. As a product manager at the time, I wanted to ask better questions to my machine learning engineers, and more thoughtfully build great AI products from the ground up. I believed that only a deep, bottoms-up understanding of what we were building could deliver that.

This experiment yielded some promising results:

• I wrote about large language models and transformer architectures before they hit the mainstream. The lessons I learned helped position our team at Spiketrap for an acquisition by reddit, two months before ChatGPT's release.

• Last year, I wrote about platform risk and “sherlocking”, and we're beginning to see those warnings play out for some of GenAI’s first movers.

That said, there’s still a lot to explore. As we enter what some are calling an LLM-powered “creativity apocalypse”, it’s probably good practice to curate a whitelisted corner of genuine conversation.

Obviously, I'm using LLMs as creative writing assistants, just like everyone else. Just count the number of times you see 'delve' in a sentence.

What’s Next

People who know me know I like to launch trial balloons into conversation – ideas shared with enough conviction to provoke reactions from those with more expertise. It’s also the Bayesian thing to do, publishing priors and updating them in real time.

Machine Yearning will be switching to a bi-weekly format, sharing unfinished thoughts and observations. These pieces may feel raw at times, but that’s actually the point.

If you’d like to get in touch, DM me on Twitter or LinkedIn, and follow what we’re up to at shack15.ventures.

Setting the Scene: Introducing "Agents"

If you’ve been listening to AI twitter, you’ve probably heard of “Agents” like BabyAGI, AgentGPT, and others. What are Agents anyway, and why were we at AGI House testing their limits in a hackathon?

Put one way, if large language models (LLMs) are the brains of advanced AI systems, Agents would be the hands, using LLMs to automate complex chains of tasks. With an Agent, instead of discretely programming a model, you can prompt it with an objective in natural language (e.g. “take my information and register for a frequent flyer account number on these 90 airlines’ websites”), and using an LLM it will learn the steps to accomplish that objective. It’s like Excel macros or robotic process automation (RPA), but on steroids.

A lot of traditional software is designed such that humans still end up doing most of the work. Things like data entry, financial and accounting processes, or IT operations. Menial but labor-intensive. Throughout the hackathon, we saw brilliant teams and demos which took the tedium of legacy workflows and chucked them out the window. This thread from Alex Reibman captured all the demos (including ours!).

Our Approach: Building an App Autonomously

Our team decided to take a literal approach to the theme of “autonomous agents.” Inspired by the recent paper "Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents" by Yashar Talebirad and Amirhossein Nadiri, we created a constrained environment where multiple agents could collaborate towards a common goal: autonomous app development in Agile sprints.

For our project, we chose an Agile development sprint in Linear (a popular project management tool), as our playground. Our playing pieces? A team of Agents designed to mimic the roles of a typical scrum team.

We devised three specialized Agents, each taking on a role in the Agile process with a biological analogue. To manage this efficiently, we also introduced an executive agent to oversee task delegation and bandwidth checks.

1. AutoPM: This GPT-4 powered agent accepts problem statements or user feedback, turning them into customer pain points. These are then translated into a framework such as "jobs to be done," converted into potential features, then refined, ranked, and finalized into a table of features and engineering sub-tasks. It uses human PM feedback before sending the tasks to AutoArch.

2. AutoArch: AutoArch prepares the groundwork before coding begins. It uses GPT-4 to draft markup languages for creating UML diagrams and generate the file structure for the project. These resources provide the scaffolding for the engineering work.

3. AutoEng: With a clear customer problem, component tasks that ladder into features, and architectural constraints, AutoEng is set to start writing code.

We quickly found that off-the-shelf LLMs were capable at most tasks, but not perfect. The AutoPM, for instance, required a lot of human feedback before arriving at product recommendations that were relevant, actionable, rewarding, and specific enough for engineers to work with.

Share

The Outcome: Success!

Despite the experimental nature of our project and only 8 hours to do it, everything (mostly) fell into place! Once the team aligned on AutoPM’s features and sub-tasks, the whole Agent relay would take LESS THAN A MINUTE before working code is shipped.

To reiterate: this is strictly a toolchain of an LLM, a UML generator, and the Linear API. And apparently… that’s all you need to manage a team of Agents in an Agile sprint.

Astute readers will have already heard of “chain-of-thought prompting” (basically, showing your work) and how that elicits better reasoning skills in LLMs. We’re effectively replicating those advancements using Linear as a scratchpad - letting Agentic team members show their work, share context, and provide clarifications where needed.

While we had to scrap some of the elements we had hoped to include, such as AutoUXR for user feedback curation and AutoEng specializations, we were amazed we even got this far!

You can check out this 45s demo for yourself to see the whole thing in action on Linear.

Here’s another example generating a Tic Tac Toe game from scratch. It’s not a difficult coding exercise, but drafting a diagram, file structure, and deployment destination gives the AutoEng much more of a scaffolding.

Takeaways: Specializations and Human Feedback

Despite the rapid turnaround, we got some valuable insights about working with agents and LLMs:

• Off-the-shelf LLMs are very capable at handling Agile tasks, but overloading a single LLM with the end-to-end process was highly error-prone.

• Most agents today are fragile - small changes to web layouts or APIs could break their logic.

• Breaking the process into specialized roles and discrete tasks made error tracing easier by avoiding “black box” productivity losses.

• While we didn’t implement robust RLHF features, we definitely saw how human feedback enhanced agent outputs, especially on AutoPM.

If you imagine integrating this with other enterprise apps like Slack, Airtable, or email… you could create an entire workforce of digital agents, with auditable trails to measure and refine work outputs. How crazy would that be?

We walked away with new ideas, a deeper appreciation for advanced AI, and a glimpse into a future where intelligent agents could transform software development. Can't wait to see what's next!

Subscribe now

Special thanks to the entire Relay team (Gurkaran, Tony, Anand, Travis, Steve, and Pascal), to Rocky Yu and AGI House for hosting, and to the event sponsors WING VC, RunPod, MultiON for helping put on a great event with an awesome theme.

Disclaimer: This post was co-written by an AI.

The Future is Open

The halls were buzzing with enthusiasm. With over 130 open-source demos scattered throughout the Exploratorium (already packed with interesting exhibits), we hardly knew where to start - we saw next-gen literary teaching tools, a dev who converted hand-drawn sketches into richly textured digital assets, literal four-legged LLaMas, and much more.

In many ways, it reminded me of the early days of the App Store once Apple opened it up to third-party developers. At launch, there were only ~500 apps, but one year later there were 35,000, with over 1 billion downloads. Just like back then, we’re in the early innings of this game, and there’s plenty left to be discovered on UX, business models, retention, and so on. Some of the lessons of that era likely apply today; more on this in a little bit.

The Gold Rush

The folks I met aren’t just sitting idly by, waiting for the future to come to them. They’re actively building it! Everyone is an API key away from integrating into their applications state-of-the-art models from OpenAI, Stability, AI21, and other research labs. Daring engineers can pip install into their local file system whatever generative model they need from Hugging Face. This has unlocked a tremendous amount of creativity and potential.

… And yet, the generous availability of cutting-edge generative AI models has a downside.

The double-edge of this accessibility is competition. No doubt about it, the competition is fierce. It’s going to be very challenging for AI entrepreneurs to build differentiated solutions, especially in the application layer.1

We saw a similar dynamic in 2008. When the App Store officially opened up to third party developers, entrepreneurs scrambled to get onto the platform. They, too, had to navigate an unprecedented level of consumer accessibility and competition. It was a similar moment of unfettered creativity and opportunity. For anyone who doesn’t remember the top apps in the first year of the App Store, here’s a refresher on the top downloaded apps in 2008:

Source: The Verge, 2018.

It was a weird time in technology. Lots of novelty, but productivity not so much, not at first. Many paid apps that hit the big time were fun expressions of the iPhone’s hardware capabilities. Koi Pond was an elegantly simple pond simulator which took advantage of the responsive touch screen, and iBeer was considered the first killer demo of the accelerometer. It even minted its creators up to $20,000 / day at the top. That didn’t last, but it’s worth pointing out!

“Sherlocking”

Novel usually doesn’t mean sustainable whether you’re talking about GenAI or iPhone apps. Apps like Recorder were quickly washed away by iOS-native apps that targeted the same functionality (Apple released Voice Memos in iOS 3). There’s even a term for this: “Sherlocking,” when a platform releases native functionality that renders a third-party development obsolete. The origin for this was a 2006 conversation between Steve Jobs and developer Dan Wood, whose company Karelia built a tool that extended the local search functionality of Apple’s Spotlight predecessor, Sherlock, to conducting online searches as well:

It seemed that the sky was the limit, until I was called in for a meeting with Apple's Phil Schiller. I listened to him tell me that Apple was going to announce Sherlock 3, and it was very similar to Watson. I watched a demo of their program: all but one of their modules connected to the same service that Watson did and looked almost the same…

Steve Jobs made it clear to Dan that any resemblance was very much his problem, not Apple’s.

"‘Here's how I see it,’ Jobs said… ’You know those handcars, the little machines that people stand on and pump to move along on the train tracks? That's Karelia. Apple is the steam train that owns the tracks.’"

Origins of Sherlocking: “The Long Story Behind Karelia's New Logo,“ Jan 7, 2006

Make no mistake, developers of foundation models own their GenAI tracks. In fact, a similar conversation happened between Jasper CEO Dave Rogenmoser and Sam Altman once ChatGPT was released. Jasper had raised a $125 million war chest at a $1.5 billion valuation, leveraging a head start conferred by its private beta with OpenAI. But almost overnight, that advantage came to a screeching halt.

Jasper’s unique value proposition was perilously diluted by the fact that ChatGPT was fast, powerful, and above all, free. Jasper has plenty of cash to burn as it tries to deepen enterprise relationships and improve stickiness, but developers of less well capitalized text generators (and other first movers) risk getting “Sherlocked” nonetheless.

Features vs. Products

This means it’s a good time to ask yourself: Are you building a feature or a product? These are two very different things. In simple terms, a product is something that customers consider valuable enough to pay for. Products have to solve problems. Features are attributes, functions, helpers that come with a product. They’re useful, but not enough to pay for on their own.

• Using the Sherlock example, search is a product. Web search is a feature of search

• Microsoft Nuance is a speech recognition product for medical scribes. Spanish speech recognition is a feature of Nuance

This isn’t a constant, since products risk becoming features of larger products overtime. Watson and Sherlock are just one example where the verticalized product (Watson for web search) became a feature of a larger product (Sherlock for web and local search). ChatGPT may do the same to many copywriting apps.

You may be thinking to yourself “well, there’s no way a developer of foundation models has enough resources to go after every vertical.” True, which is why OpenAI effectively launched its own app store with Plugins. Want to make an “LLM for dinner reservations” product? There’s a Plugin for that. What about an “LLM for booking flights”? There’s a Plugin for that.

Now, if you tell me “Travel is about ditching the mundane and discovering the extraordinary. But travel planning? Incredibly stressful. We believe you should be able to plan your entire trip with the push of a button. So we’re building everyone’s private travel agent. Interested?” Sure there are Plugins that can handle pieces of what I may be thinking of, but you’ve got me hooked. Tell me more.

Throughout the Hugging Face event, several entrepreneurs asked me to demo their apps. The tech was consistently impressive, but my first question was usually: “Who is this for?” More often than not, the answer was akin to “It’s for everyone, the possibilities are endless!”

I agree, the possibilities are endless! But if you intend to build a sustainable business, I encourage you to focus on building solutions with a specific customer in mind, a customer who will pay for it in time or money. If you can articulate the following:

• The pain they have

• The willingness they have to pay to eliminate that pain

• How many people out there share that pain (your total addressable market)

then we’re really onto something. In doing so, we can create products that are genuinely valuable and rewarding for both the customer and the entrepreneur. If you need a place to start, I always recommend “The Mom Test,” one of the best resources on practical questions to ask as you’re finding product-market fit. Cut through the BS and figure out if customers will pony up!

Most importantly, have fun!

Most of these notes are aimed at founders intending to build GenAI businesses, but there’s just as much value in diving in and seeing what you can create. Don’t just build a generative AI app because you think it’ll grow into the next unicorn. Build it because you want to, build it because it’s fun, build it because it solves a specific problem and you’re passionate about solving it!

If enough people share that problem and are willing to pay for that product… then who knows! But building and learning are inherently valuable.

There are going to be so many new business models, experiences, and possibilities we discover in the application layer that it isn’t clear first-mover advantages exist. If you’re building an infrastructure, platform, marketplace, or any other product that depends on network effects for its fundamental value, then yes, it’s best to be first. But not every business fits that description. Above all, build something that people love.

The potential of AI is vast, and many exciting opportunities await entrepreneurs who  create world-changing solutions. #WoodstockAI was an amazing celebration of all this potential energy. With so much talent and enthusiasm in the AI community, I'm confident that we'll continue to see groundbreaking advancements in the field.

Share

Thanks for reading!

Liked this post? Comment and subscribe for more!

Subscribe now

Machine Yearning is a collection of essays and news on the intersection between AI, investing, product, and economics, light on technicals but heavy on relevance.

Ryan Cunningham is a Senior Builder at Andrew Ng’s AI Fund, a venture studio in Palo Alto. As part of the venture studio, he’s co-founded AI startups in the fields of fintech, human capital, swarm intelligence, speech recognition, generative AI, and more. He specializes in deep tech applications (foundation models, drones, autonomous vehicles) and is active in the Stanford AI Alignment community at SERI, where he contributes to research and advocacy efforts focused on AI safety.

Any suggestions or topics you want to see? DM me on LinkedIn or Twitter.

I wrote this piece shortly after attending a seminar with postdoctoral fellow Jeffrey Ding at Stanford’s Institute for Human-Centered AI. In it, Ding convincingly laid out what I think are the most common fallacies American pundits fall into when discussing AI in the context of global powers. I’ve added to it with practical recommendations on how to navigate around these fallacies.

Ding also has a fantastic newsletter, ChinAI, where he provides first-party translations of Chinese source texts and strategy documents on artificial intelligence and other high technologies. If you’re used to getting your news on China or AI from non-Chinese speakers or non-practitioners, you owe it to yourself to subscribe!

📱 How Technology Changes Societies

“First to AI supremacy?” Not so fast. When it comes to artificial intelligence, it doesn’t matter who’s first because it's not a product in the way most pundits understand it.

“AI is the new electricity,” but electricity isn’t a product. Being first to have electricity doesn’t matter without the infrastructure to deliver it, the human capital to manage and improve upon it, and the standards to commercialize it. Benjamin Franklin famously conducted his kite experiment in 1752, and Thomas Edison patented the lightbulb in 1879, but it wasn’t until the 1920s that even half of American homes had electricity.

Like electricity, AI is a general-purpose technology - one with vast potential for nearly all sectors of the global economy. Talking about “AI supremacy” as if it were a zero-sum game is a fundamental misunderstanding of both how AI works and where AI power comes from. If institutions truly want to harness AI for large-scale transformations, then we need to create environments suitable for innovating upon it. We need to cultivate our societies into “innovation gardens,” instead of planting our flag on the moon and never going back.

A Tale of Two Theories

Before we get into it, let’s discuss the source of this misconception, so we know what to avoid when discussing AI’s potential.

There are two competing frameworks for understanding technological advancement in societies:

1. The standard view is the leading sector theory of technological advancement, which emphasizes the first-mover advantages in fast-growing industries (aka leading sectors). The impact is felt rather immediately and is concentrated in a key sector. In international political economy, a single nation-state first monopolizes initial gains (sometimes called an “innovation monopoly”), then the industry spreads to other competing powers.

2. An alternative theory, proposed by researcher Jeffrey Ding, a postdoctoral fellow at @StanfordCISAC and @StanfordHAI, is the diffusion theory of general purpose technologies. This theory highlights longer, drawn-out trajectories of incremental improvements upon general technologies which diffuse into broad sectors of the global economy overtime. The impact comes later and is more dispersed.

High-profile products like smartphones and ridesharing fit neatly into the first framework. They have immediate use cases, benefit from network effects, and have winner-take-all dynamics.  

After all, If you’re buying an iPhone, that’s the only phone you’re going to buy for the next 2 years. If you take an Uber, that’s one fewer trip out of your 5 daily trips going towards a taxi or Lyft. These are zero-sum games, for the most part, which fit into the product life cycle framework taught in most business school curricula. 

Artificial intelligence does not share these characteristics of leading sector products:

• Like electricity, AI is a general purpose technology that when first introduced did not have a singular commercial application. 

• The open-source nature of most AI research means that state-of-the-art performance is not restricted to first movers; I can visit HuggingFace and deploy a GPT-like model for some web app in under an hour.

• It is highly pervasive, meaning it has applications for many sectors of the economy. To oversimplify, anywhere a decision must be made using data or intuition, AI can augment or replace the decision-maker.

A summary of these distinguishing factors is laid out below.

Being First Isn’t Enough

Suffice to say, it isn’t enough to just be first to invent a new general purpose technology. That alone does not guarantee a competitive moat. Instead of focusing on building innovation monopolies from one-off products, Ding suggests societies that see the greatest impact from technology diffusion are those which intentionally cultivate environments suitable for innovation.

These environments:

• Continuously improve state-of-the-art benchmarks (via patents, research, and academic papers)

• Upgrade human capital with formalized disciplines (e.g. machine learning engineering, AI product managers)

• Introduce and evangelize standards for development and production (e.g. MLOps)

• Inspire innovation chains of complementary technologies across broad sectors of the economy

It’s a more holistic and nuanced approach vs. the innovation-centric framework for leading sector technologies, which is more popular with pundits.

⚡️ “AI is the New Electricity”

Andrew Ng famously coined the phrase “AI is the new electricity,” claiming it will “transform every industry and create huge economic value.” If, like electricity, AI adoption follows a diffusion theory trajectory, what practical investments are required for institutions and countries to cultivate leading innovation gardens?

I would argue for a cohesion of 5 focus areas of public-private partnerships:

1. Clear Research and Development Goals

2. Synergistic Infrastructure

3. Human Capital Upgrades

4. Commercial Standardization

5. Ethics and Explainable AI (XAI)

1) Clear Research and Development Goals

Clear research and development objectives at the national level signal strategic interests which may not emerge from the commercial sector on their own.

Chinese AI basic research goals include cross-media sensing and computing (think audio-visual-text fusion) and brain-inspired intelligence computing, among 6 others. All of these synergize with explicit commercial, infrastructure, and human capital goals in their strategy document, the 2017 Next Generation Artificial Intelligence Development Plan, which align neatly with the diffusion theory framework.

Though these basic research lanes aren’t likely to result in commercial offerings on a VC time scale, that isn’t the primary goal. The goal is creating the conditions for a quickly compounding chain of follow-on innovations.

Contrastingly, a glaring miss in the 2019 American AI Initiative is clarity around specific R&D goals of any kind (Strategy #1 is “make long-term investments in AI research”). The objectives mentioned are far too high level. In absence of specific R&D goals, “planning to plan” is not a plan. Nor is “promoting leadership.”

2) Synergistic Infrastructure

Institutions should focus on building and supporting synergistic frameworks between open-source hardware, software, and cloud infrastructure.

If the past two years have taught us anything, it’s the fragility of the global silicon supply chain. Industry-wide over-reliance on just a few general-purpose chip manufacturers is a systemic risk for the global economy, with inventories under siege from players in nearly every industry. Investing in antifragility for the hardware supply chain would be a good long-term bet, meaning serious consideration of novel computing methods like neuromorphic computing or even quantum computing architectures, which are far more efficient than von Neumann architecture, is in order.

Open-source software libraries like Tensorflow and PyTorch, as well as platforms like HuggingFace, are speeding up AI application development time. Investing in these and other platforms for standardization across other AI paradigms should reap similar rewards.

Finally, development of national research clouds, as France, Japan, and China have done, should provide an antifragile alternative to incumbent cloud computing platforms, especially for basic research which may not have immediate revenue opportunities.

3) Human Capital Upgrades

As international competition for top researchers heats up, America is in particularly dire need of an upgrade in human capital.

American universities still attract the best international talent, and enjoy healthy ecosystems promoting basic AI research. But the rest of the US population suffers from incredibly low literacy rates on fundamental AI concepts. By some estimates, fewer than half of US high schools teach any computer science at all. Of those that do, the curricula have remained more or less unchanged for 15 years.

Upskilling startups like FourthBrain and Deeplearning.AI are delivering practical and highly relevant skillsets needed for learners to pivot into a career in AI. Workera is augmenting employees of existing workforces, and Factored is curating a deep bench of contractable AI/ML experts for multiple industries. Finally, in primary education, Kira Learning is designing a contemporary AI fundamentals curriculum for K-12 American students in all 50 states, the first of its kind.

4) Commercial Standardization

Like Agile software processes which formalized mechanical and software engineering workflows, MLOps is standardizing ML engineering in the workplace. Through MLOps, engineers and product managers are learning to tackle development of AI products in virtuous closed loops rather than linear progressions. Startups like WhyLabs and Arize are formalizing these practices, making model development a core business process at many institutions, instead of a data science side project.

5) Ethics and “Explainable AI”

Much like the concerns around dangerous electricity inspired regulation and safety practices, we should anticipate a need for explainable AI (XAI) and the ability to audit model decision-making frameworks. This is especially relevant for “black box” models in mission-critical applications, such as loan applications or the criminal justice system, where hidden biases may have drastic and immediate impacts on humans’ well-being. Startups like Credo AI are paving the way for XAI frameworks, guaranteeing safe model deployment to critical sectors.

💯 In 100 words or fewer…

Institutional AI supremacy will not be the result of one-off killer products, but a concerted, holistic series of investments in resource infrastructure, human capital upgrades, and standards-setting to create environments that nurture and encourage innovation. These innovation gardens do not exist de facto for any one system of governance, but are deliberate in their construction… without them, any first mover advantages will quickly wane, and the world’s primary AI innovation center may converge elsewhere.

Share

Thanks for reading!

Machine Yearning is a collection of essays and news on the intersection between AI, investing, product, and economics, light on technicals but heavy on relevance.

Ryan Cunningham is a Senior Builder at Andrew Ng’s AI Fund, a venture studio accelerating the adoption of AI across the global economy. Prior to joining AI Fund, he worked in product at Uber and various AI startups, beginning his career as a technology investment banker at Credit Suisse. He studied Finance and Economics at Georgetown University, and is currently studying Artificial Intelligence part-time at Stanford.

Any suggestions or topics you want to see? Connect with me at takeme.to/ryan (@rydcunningham across all platforms).

Part 1 explores language models at a high level, with both fun and dangerous applications.

Part 2 (this post) introduces GPT-3 for anyone who's heard about it but has no idea what makes it special.

🔬 Long Take: GPT-3 & Me

You've all probably seen article after article hyping up this thing called "GPT-3." You may have seen headlines like "Why GPT-3 Heralds a Democratic Revolution in Tech" or "One Model to Rule Them All". You may even know that GPT-3 has 175 billion parameters (kudos), and not really know what a parameter is, just that 175 billion is a lot.

Not to worry, we'll clear that up in this post. You'll be able to chat with your co-workers about the exciting new language model in about 5 minutes from now.

What Even Are Parameters

I get it, after a certain point the numbers all sort of blend together.

The broader definition is that “parameters” are different coefficients a machine learning model tries to optimize during training.

The plain English understanding is that for a language model, parameters represent the strength of connections between words and word sequences. More parameters means more opportunities to see different sequences in different contexts.

• If a sentence starts with “Please pass the,” what is the probability that the next word will be “salt”?

• What about “kangaroo”?

Chances are that most small models would never ever predict "please pass the kangaroo" as a likely sequence. Most humans wouldn’t, anyway. If a model has never seen that sequence in its training data, there is an infinitesimal chance of predicting "kangaroo" as the close word.

However, on the off-chance the training data is so large and parameters so numerous that the model spans many different domains, there may be a chance - however small - that sequence may come up.

What Are Domains

Traditionally, the domain of the training set is strongly correlated with performance on any one task.

For instance, a hate-speech detection model trained on language data from Wikipedia or another well-curated, grammatically sound corpus will rarely perform as well as a model trained on more informal sources like web forums, IRC, Discord chats, etc.. There are at least three reasons for this behavior:

• This is because the kinds of insults, slurs, and syntactical/grammatical language nightmares on those informal sources are far more varied, numerous, and current than a formal list from Wikipedia

• Each forum may have unique lingo, like Redditors using /r/subredditsashashtags, making it more difficult for models trained on other domains to predict subsequent words

• Since there are usually more diverse contexts (i.e. the words around the slur) for forum-based sources, by using a model trained on that domain, we have a much better chance of recognizing contexts in the wild that match up with examples of hate speech the model has already seen

At least, these are the traditional limitations of language model effectiveness.

ELI5 GPT-3

And then of course, GPT-3 comes around. Over two orders of magnitude larger than its predecessor GPT-2, and ten times larger than any other model at the time of release, GPT-3 is renowned for its generalizability and emergent properties unseen in prior language models. In particular, a meta-learning feature called "in-context learning," which we'll come back to in a bit.

Few-shot learners

OpenAI's paper "Language Models are Few-Shot Learners" introduced GPT-3 in July 2020. The term "few-shot" (as opposed to "one-shot" or "zero-shot") refers to the number of examples a model has to observe before inferring the task it's trying to accomplish.

Put more colorfully, picture the following:

• You're unexpectedly and violently woken up to the sound of a masked man blaring Queen's "Bohemian Rhapsody" through a megaphone in your bedroom.

• At "Caught in a landslide..." the song stops, and this mystery person gestures to you.

• With neither a coffee nor a clue as to why this person barged into your home for this, you muster up the softest, most confused "...no escape from reality."

If the task was to predict the next lyric in the song, congratulations! You're a one-shot learner.

If however, the task was something else - like a question-answer task "Who is the artist who sings this lyric?", then you wouldn't have gotten it right the first time. It might take a few more examples from the masked man with a megaphone before you get it right. Or before you call the police. This is called "few-shot learning."

GPT-3 is particularly good at few-shot learning for a wide variety of NLP tasks, including translation, question-answering, word unscrambling, and even 3-digit arithmetic. This is part of the reason why it's so generalizable, because a single model can be used for multiple applications.

In-context learning

The reason for this is that GPT-3 seems to infer the type of task it's being asked to solve much faster than smaller models. Its sheer size developed a broad range of pattern recognition skills, which are then used to quickly adapt to whatever unspecified task the user is prompting.

This meta-learning characteristic is called "in-context learning," which means that the model is able to infer the task it’s trying to accomplish on the fly, learning as it goes.

When prompted with even just a handful of examples, GPT-3 has remarkable accuracy on not just traditional NLP tasks, but also new adaptations like writing articles, guitar tabs, and even computer code that otherwise would have required specialized models.

Is this intelligence?

I wouldn't say so, more like extremely high-dimensional mimicry.

Recall that in Watch Your Language, Part 1, a Berkeley student was able to write entire productivity blog posts by prompting GPT-3 with a title and introduction. Does the fluency of the language, and its ability to fool human readers, imply it truly understands (as we would) causal links between sentences and aspects like "narrative"?

Not necessarily. Lifestyle blogs are one of the most popular blog formats, and posts on productivity are quite numerous. Because 60% of GPT-3's training mix is made up of crawled websites from all over the internet, it's safe to assume that the model has probably seen enough examples of lifestyle blogs to craft language that resembles them.

Some common tells are that GPT-3:

• Has trouble maintaining narrative consistency over longer documents

• Is subject to repetitions and non sequiturs

• Underperforms on language generation that requires logic and reasoning. Lifestyle blogs don't require either, which is how it was able to dupe readers so easily.

Regardless, GPT-3's size has undoubtedly made its meta-learning characteristics much more performant and easier to identify. It's still a mystery to most researchers exactly how in-context learning works, but the excitement is palpable to see how much further we can take it.

What is indisputable so far, is that the more data a model trains on, the better it generally seems to perform.

In Closing: The Cost of Capital

Which brings us to our final point. As models grow, so do their costs of capital. Using Google’s T5-11B (its previous 11 billion parameter language model)’s $1.3 million per run estimate (State of AI Report 2020, slide 17), after all rounds of training experts ballpark the training cost of transformer language models at ~$1 / 1,000 parameters.

Roughly speaking, that puts the final tally for GPT-3's training costs at around $175 million for a single model. There are very few institutions that can afford that kind of capital. And if you’ll recall from part 1, there are risks associated with such large models, including disturbing encoded biases and the budding replicability crisis.

To wrap this up, I'll repeat what Dr. Kai-Fu Lee of Sinovation Ventures recently shared when asked about the investment potential of massive language models at this year’s AAAI conference. He shares similar concerns about the growing tendency towards monopolistic advantage in large models. The full video and slides are in the 🎥 Watching section of this week’s bonus content.

I believe that the new, huge language models that appear to make a big difference, also come with a large cost in computation. That on the one hand has energy implications, but it also makes it harder and harder for universities to compete with internet giants. Google and Microsoft, they can build a $100mm computer, and train models with a trillion parameters, but professors and researchers cannot. Every country should think about how we make those types of resources available to researchers and startups so that the giant corporations don't end up with a... monopolistic advantage.

Bonus Content

Articles, reports, videos, and more worth checking out this week that didn’t make the featured cut

💹 AI in the Markets

Building a digital twin of the planet (ETH Zurich)

European and ETH Zurich computer scientists proposed developing a digital twin of the Earth for high-precision environmental and climate modeling across space and time. Researchers estimate this system would require ~20,000 GPUs, consuming an estimated 20MW of power.

SuperAnnotate, a no-code computer vision platform, partners with OpenCV (TechCrunch)

This partnership could lower the barrier to entry for lots of business leaders on cmputer vision applications. Personally I own 2 OpenCV OAK cameras. They come out-of-the-box with object detection inference models onboard, and are quite impressive.

🔧  Hardware & ASICs

Semi Demand 30% Above Supply, 20% Year-on-Year Growth (AnandTech)

TSMC, Samsung, UMC, and GlobalFoundries leading the pack, with China's SMIC in 5th. Inventories are drying up since "semi companies are shipping 10% to 30% below current demand levels, and it will take at least 3-4 quarters for supply to catch up with demand."

💡 Startups & Strategy

Why There's No Such Thing as a 'Startup Within a Big Company' (Waze vs Google)

Waze co-founder and CEO Noam Bardin left Google in January and published a personal essay detailing "the trickle-down problem" in thoughtful and heartbreaking detail.

🎧 Listening

The chip choke point. A single machine from the Netherlands could catapult China to the leading edge of the semiconductor industry. If the U.S. allowed it, that is. (The Wire China)

EUV light sources emit light in incredibly short wavelengths, less than 20nm, which is necessary to carve circuit features onto nodes less than 7nm. The company, ASML, has been courted by China for a decade for a potential acquisition, however the patent for EUV is actually owned by a US company. Without it, China will have to pursue alternative means to go from 14nm to 7nm.

🎥 Watching

AI Infusion & Investment Opportunities, with Kai-Fu Lee (AAAI-21)

Dr. Kai-Fu Lee (founder of Sinovation Ventures, author of AI Superpowers) highlights the contributing factors to China's competitive rise within applied AI, and identifies some oft-overlooked investment themes.

Thanks for reading!

Machine Yearning is a collection of essays and news on the intersection between AI, investing, product, and economics, light on technicals but heavy on relevance. Think of it as a casual chat about AI over coffee (or any other preferred beverage).

Ryan Cunningham is an AI Product Manager, strategist, and ex-investment banker. Today he leads applied AI strategy and new verticals at Spiketrap, an NLP-as-a-service company. He spent 4 years at Uber and is currently studying Artificial Intelligence part-time at Stanford, with a BS in Finance and Economics from Georgetown University.

Any suggestions or topics you want to see? Shoot me an email at rydcunningham@gmail.com or hit me up on Twitter / LinkedIn / Clubhouse @rydcunningham.

I originally planned this week to dive into Google’s recently released Switch Transformer language model and how it differs from previously published models like GPT-3. Instead, I will be splitting this topic into two parts:

Part 1 (this post) will explore language models at a high level, with some fun and some dangerous applications.

Part 2 will compare current state-of-the-art models like the Switch Transformer, GPT-3, and its open-sourced alternative GPT-Neo, through a more technical and economic lens.

For now, welcome to the sometimes silly, sometimes concerning, but always interesting world of language models. Here’s why they matter and why you should care.

🔬Long Take: Language Models for Fun and Profit

Before I start, it’s probably best to quickly state what a “language model” is.

Put simply, given the context of preceding words, a language model predicts what the next word in a sentence will be.

That’s it. Simple as that.

Believe it or not, you already use language models in your daily life. Gmail’s ‘Smart Compose’ feature and the predictive keyboard on your iPhone are common examples.

Researchers train these models on large collections of text documents (corpora), which contain millions and millions of words and sentences in natural language. Some of the most common corpora include sets from Wikipedia, books in the public domain, internet news, even Shakespeare. Importantly, the kinds of corpora researchers choose to train on have significant implications for the kinds of words the model predicts.

“You are Hagrid now.”

Back in late 2017 one studio trained a language model on Harry Potter books, then used it to independently generate a hilarious new chapter called Harry Potter and the Portrait of What Looked Like a Large Pile of Ash. I definitely recommend reading the whole thing, but here are my favorite excerpts:

Leathery sheets of rain lashed at Harry’s ghost as he walked across the grounds toward the castle. Ron was standing there and doing a kind of frenzied tap dance. He saw Harry and immediately began to eat Hermione’s family.

Ron’s Ron shirt was just as bad as Ron himself.

“If you two can’t clump happily, I’m going to get aggressive,” confessed the reasonable Hermione.

“Not so handsome now,” thought Harry as he dipped Hermione in hot sauce.

The pig of Hufflepuff pulsed like a large bullfrog. Dumbledore smiled at it, and placed his hand on its head: “You are Hagrid now.”

At best, it’s a whimsical, silly romp that actually does sound Rowling-esque. But since this model is trained only on Harry Potter books, with a measly total of 1,084,170 words, its performance is rather limited. Professional writers and copy-editors had to be employed to prune the generated text, and you shouldn’t expect this to perform very well on a task like Smart Composing email replies.

“Writing more while thinking less.”

Another silly, but more concerning real-world example is a productivity blog created by Berkeley student Liam Porr last year. Reportedly, it only took a few hours of playing with OpenAI’s popular language model GPT-3 (more on this in a bit) for Porr to think of an experiment to run: first, he would feed the model a headline and introduction for a blog post as inputs. Next, given that context, the model would go on to generate several versions of full-length self-help and productivity blog posts based solely on just the headline and intro.

Determining these inputs wasn’t difficult. He would make quick trips to Medium and Hacker News to see what’s trending, copy something similar, and that would get the job done. Under the pseudonym “Adolos”, he began publishing near-daily posts to his productivity blog over a two-week period:

This was an innocent enough idea, and in practice the blog posts seemed so familiarly written and the language so sensible that almost no one could see through the ruse.

In a surprising turn of events, many who suspected these posts were AI-generated were actually criticized and downvoted by the community!

Just before fessing up to the whole charade, his final, human-written blog post “What I Would Do with GPT-3 If I Had No Morals” pulled an O.J. and described in hypothetical detail how one might go about pulling this off.

While the output is not perfect, you can easily curate it to something that's convincing. This will make it so easy for people to just pump out clickbait articles to drive traffic. It would be pretty simple to do actually.

First thing you would need to do is come up with a name. If it were me, I’d name it after the Greek god of deception or something like that just to be clever. Then I’d just stick an “A” in front so nobody gets suspicious.

After that, I’d make a substack because it takes no time to set up. Once thats done you have to come up with some content. GPT-3 isn’t great with logic, so inspirational posts would probably be best, maybe some pieces on productivity too.

Porr observed that GPT-3 is “good at making pretty language, and … not very good at being logical and rational.” Given those constraints, what better blogging category to pick than productivity and self-help?

Size matters

On a more serious note, that these communities were so easily duped with such little effort on Porr’s part raises serious questions. How is it that language models like GPT-3 are able to generate such convincing language?

When it comes to building effective language models, generally speaking the more natural language data included in the training phase, the more convincing the generated outputs will be.

Now remember, models themselves aren’t malicious, they just amplify whatever biases exist in the data used to train them. Massive language models? Even more so.

Two ___ Walked Into a ___

Here’s where things get dicey. Taking the outputs of massive language models for granted can have dire consequences, well beyond occasionally duping a bunch of Hacker News readers. Abubakar Abid, a Stanford PhD candidate, demonstrated one such example using the GPT-3 demo in August of last year. Watch the video below to see what happens when the model is fed a sentence that starts with “Two Muslims”…

Compared to any other religious group, preceding a sentence with “Two Muslims” generated violent content about 9x as frequently.

Abid sees this sort of anti-Muslim bias as “persistent” in many massive language models.

Now, why might this be happening? These models are built on massive datasets with billions and billions of tokens from many different sources, like books, news, Wikipedia, and website text. For these datasets, think about what bias the authors, editors, and journalists may have had going into the publishing process for the documents in these corpora.

• What language, what country, what time period were they written in?

• What percentage of documents come from a time period, say, post-9/11, when the frequency of anti-Islamic sentiment and hate crimes rose and persisted for many years?

The bigger these models get, the more difficult it is to perform studies that spot these kinds of biases. Without proper scrutiny and auditing, training on massive text datasets carte blanche will encode sometimes extreme but otherwise unhelpful biases into our language models.

The Replicability Crisis

Because these models are so big, the arms race is contributing to a replicability crisis where very few entities have the capital and resources to train and run said models from scratch. We end up having to rely on the publisher (Google, OpenAI, etc.) to check their homework.

While some researchers are alright with this approach - let Google-sized entities pre-train and publish these large general models, and have smaller entities customize them for specific applications - this obscures peer reviewers’ and third parties’ ability to spot biases that can be used to discriminate, radicalize, or justify extreme violence.

What can we do about it?

Understanding how these biases show up is critically important. That’s the first step to solving the problem.

Ask the right questions.

I alluded to this before. Be inquisitive about the datasets used for training. What are their sources, on what time window were they collected, etc.. Are there alternative datasets available which may mitigate or de-risk unhelpful bias?

Conduct regular audits.

Structured reviews and audits of these popular datasets often yield surprising results. To borrow a computer vision example, the Tiny Images dataset from MIT (comprised of 80 million labeled 32x32 pixel images used to train and benchmark computer vision models) was removed from the public domain after a more thorough review revealed some highly inappropriate, derogatory annotations (emphasis mine):

The dataset includes, for example, pictures of Black people and monkeys labeled with the N-word; women in bikinis, or holding their children, labeled whores; parts of the anatomy labeled with crude terms; and so on – needlessly linking everyday imagery to slurs and offensive language, and baking prejudice and bias into future AI models.

Personally, I think there’s much more to learn from auditing and pruning than outright removal, but tough calls have to be made. Unfortunately for MIT, the images themselves were too small for manual review, so the lab elected to remove the dataset in its entirety.

When creating datasets, do your diligence.

The lab (CSAIL) also admitted that in creating the dataset, they “automatically obtained the images from the internet without checking whether any offensive pics or language were ingested into the library.” Any downstream models making use of the Tiny Images dataset, or any large pre-trained model built on it, would therefore mistakenly suggest problematic, causal links not grounded in reality.

Open-source where possible.

For now, GPT-3 is under lock-and-key on an exclusive license to Microsoft. The source code was never released to the public, which has obscured the ability for researchers to examine its biases in detail.

Fortunately, a team called EleutherAI is working to open-source GPT-3 through a parallel effort called GPT-Neo. Areas of improvement include a dedicated team of curators who have performed “extensive bias analysis” on training data, and in some cases, excluded otherwise popular datasets they felt had “unacceptably negatively biased” content. Compared to GPT-3’s 5 datasets, GPT-Neo has a more diverse mix of 22 smaller datasets.

Moving forward

It’s important to remember that machine learning models are, in their simplest form, just pattern recognition tools. They are not free of subjectivity. Any bias that exists in the training data will be amplified. So there is an ethical responsibility on the part of both researchers and customers to think deeply about their models’ application and misapplication.

In a world where proliferated, unmoderated fake news on Facebook directly contributed to the systemic genocide of a Muslim ethnic minority (Rohingya Muslims in Myanmar), it isn’t a stretch to imagine how malicious actors could, with a botnet and a language model, procedurally generate enough conspiracies and hateful content to dupe individuals, communities, and governments alike.

At the end of the day, I would caution against measuring progress by size alone, and recommend a more nuanced approach which takes extreme bias detection into consideration. To make effective use of these large models you need capital and resources belonging to a select few, which concentrates research and publishing power, de-emphasizes practical relevance for most institutions, and increases systemic risks when left unchecked and unchallenged.

Subscribe now

Bonus Content

These are articles, reports, videos, and more worth checking out that didn’t get a featured mention this issue.

📚 Reading

💲 AI in the Market

From China to San Francisco: The Location of Investors in Top U.S. AI Startups (CSET)

The most popular foreign investors in top US A.I. startups are out of - you guessed it, China.

Comparing Corporate and University Publication Activity in AI/ML (CSET)

While universities technically produce more numerous AI papers, corporations generate the most citations for published work.

🏛 Government & Policymaking

The Biden Administration’s AI Plans: What to Expect (MIT Technology Review)

Three takeaways: first, the President elevated the director of the Office of Science and Technology Policy (OSTP) to a cabinet-level position. Then, he named a prominent sociologist as deputy director, who may be more mindful of encoded bias to algorithm inputs. Finally, his Secretary of State described the pursuit of AI standard-setting against ‘techno autocracies’ (i.e. China) in particularly dramatic terms.

US has ‘moral imperative’ to develop AI weapons, says panel (The Guardian)

For about eight years, a coalition of non-governmental organizations have been pushing for an international treaty banning the use of autonomous (read: no humans in the loop) weapons. Interesting takes in here as to why (“human control is necessary to… assign blame for war crimes.”). The National Security Commission on Artificial Intelligence, led by former Google CEO Eric Schmidt, acknowledges risks but does not recommend a ban, saying such weapons will contribute to a cleaner, safer battlefield, free of human error.

We’ll definitely cover this in a future issue, but for now keep this in mind: genetic diversity is a crucial line of defense for any species. If our autonomous weapons can all fall victim to the same adversarial attacks, our defense infrastructure becomes far more fragile.

🔍 Hidden Biases

OpenAI and Stanford Researchers Call for Urgent Action to Address Harms of Large Language Models Like GPT-3 (VentureBeat)

“Lies travel halfway around the world before the truth ties its shoes.” Perpetuating disinformation is only one of many malicious applications of large, unaudited, pre-trained language models.

Predictive Policing is Still Racist - Whatever Data it Uses (MIT Technology Review)

Predictably, supervised algorithms trained on past crime data make irresponsible causal fallacies between race, crime, and recidivism, no matter which way researchers slice it. They’re having a tough time finding a technical fix; in the meantime, they advocate for a political solution, but officials are reluctant to abandon the technology.

🔧 Hardware & ASICs

Alibaba Unveils Its First AI ASIC, Hanguang 800 (CNBC)

Older news (Sep 2019), but still important. Large companies reliant on AI for competitive advantages have been developing their own hardware in-house, like Google and its custom tensor processing units (TPUs). Alibaba’s Hanguang 800 can allegedly speed up computing tasks by 12x in crucial tasks for e-commerce (product search, translation, recommendations, etc.). Eventually, Alibaba can lease out their cloud compute and Hanguang 800 capacity to other companies, a lá AWS.

Baidu in Talks to Raise Money for a Standalone AI Chip Company (CNBC)

In the face of global semiconductor tariffs, Chinese tech enterprises have realized the need to invest in their own semiconductor capacity. GGV Capital and IDG Capital are advising. These industries don’t just pop up overnight, but compute (and the resources to produce it) is becoming more valuable than oil. Keep an eye out for pundits quantifying “compute independence” on an international basis.

Tencent Invests in Chinese AI Chip Startup as Part of $279 Million Funding Round (CNBC)

The startup in question is “Enflame Technology,” headquartered in Shanghai. CITIC, CICC Capital, and Primavera are involved.

NeuReality Emerges From Stealth with $8m Seed for AI Compute Infrastructure (HPCwire)

NeuReality is an Israeli semiconductor startup building scalable AI-specific chipset. They recently appointed Naveen Rao to their Board of Directors; Rao founded the first AI chip startup, Nervana Systems, later acquired by Intel.

🎧 Listening

The First AI Chip Startup with Naveen Rao, Nervana Systems (Spotify)

Naveen Rao shares firsthand experience with the ‘trickle-down problem’ after Nervana Systems was acquired by Intel. Listen in for why the chipmakers of today will most likely not be the AI ASIC champions of tomorrow.

🎥 Watching

How Can Human-Centered AI Fight Bias in Machines and People? (MIT Sloan)

Prevailing wisdom assumes that the role of algorithms “is to correct the biases that humans have… [t]his follows the assumption that algorithms can simply come in and help us make better decisions – and I don’t think that’s an assumption we should operate under.

Thanks for reading!

Machine Yearning is a collection of essays and news on the intersection between AI, investing, product, and economics, light on technicals but heavy on relevance. Think of it as a casual chat about AI over coffee (or any other preferred beverage).

Ryan Cunningham is an AI Product Manager, strategist, and ex-investment banker. Today he leads applied AI strategy and new verticals at Spiketrap, an NLP-as-a-service company. He spent 4 years at Uber and is currently studying Artificial Intelligence part-time at Stanford, with a BS in Finance and Economics from Georgetown University.

Any suggestions or items you want to see? Shoot me an email at rydcunningham@gmail.com or hit me up on Twitter / LinkedIn / Clubhouse @rydcunningham.

Subscribe now

I kicked off this project as an extension of a time-honored tradition where every Thanksgiving, a family member asks me “what is AI? When are the machines coming to get us?”

AI as a concept can often seem intimidating to folks with non-technical backgrounds, and we tend to fear what we don’t understand. They’ll usually fill in the gaps with pumped up expectations and dystopic sci-fi imagery.

In my work leading applied machine learning projects at institutions big and small, I’ve run into many versions of the same Thanksgiving problem. Simply put, there’s not enough people speaking the same language.

The goal of this newsletter is to bridge that gap by providing relevance, commentary, and deep-dives for the 90% of institutions who have heard about AI but don’t yet know where to start. It’s the intersection between AI, business, and product strategy. If this sounds like something you’re into, read on!

Subscribe now

🔬Long Take: Knowledge Gaps

Let’s get one thing out of the way: machine learning is a toolkit like any other. What’s more important than knowing how to use a tool is to know when not to use it. You don’t need an advanced degree to know when to use a hammer, and you don’t need to know how the hammer was designed, either. Just what it’s for.

These knowledge gaps manifest in a few different forms:

• The expertise problem: When the quants are too far away from the customer to understand problems in context. In my work, the most common ask I hear from engineers and scientists is to hear the customer’s perspective more often.

• The marketing problem: When client-facing personnel are not well-versed enough in technical concepts to concisely explain the value-add. This is one of the most common problems in fundraising.

• The trickle-down problem: When an acquired AI startup doesn’t effectively grease the wheels of the inertia-laden parent company, stagnates, and the talent is either bought out, returns to academia, or both. Ultimately the acquisition is written off. More on this in a future issue.

These problems are so pervasive that according to a BCG/MIT joint study, only around 10% of businesses have actually seen “any significant financial benefits” from applied AI. According to the report, the reasons for failure are nearly always preventable, not technical: a lack of alignment, clear goal-setting, and leadership buy-in. The marketing problem is at the center of each of those failure points.

This doesn’t stop at the enterprise level. Sometimes, the misunderstanding is so pervasive that it spreads across the entire industry as runaway hype train, derailing otherwise perfectly competent portfolio companies for not living up to sky-high expectations.

A Tale of Two Curves

To illustrate this, I want to highlight a particularly common case where investors and practitioners disagree on what should be considered reasonable progress.

ARK’s “Big Ideas 2021”

ARK is an asset manager that structures its investment strategies around themes linked to “disruptive innovation.” They have a podcast I listen to called FYI - For Your Innovation, and recently released their “Big Ideas 2021” report, which highlights some of their favorite investment themes for the upcoming year.

This year, they extensively focus on deep learning (the parfum du jour of machine learning methods) and share high expectations for its economic contributions in the coming years. In particular, they expect deep learning applications to grow from an estimated $2 trillion in contribution to equity market capitalizations (as of 2020) to a whopping $30 trillion over the next two decades.

This massive “deep learning wave” is contingent in part on the expected performance enhancements from deep learning applications. Like many others, ARK projects a familiar curve resembling Moore’s Law.

Projections like these are a good example of the marketing problem. Yes, some deep learning models are getting exponentially bigger, and if you were to take this to its logical conclusion, it’s only a matter of time before deep learning truly eats the world, with economically uncompetitive humans living off scraps. But is size on its own tantamount to performance? Not exactly.

Starsky Robotics’ Swan Song

Contrary to the Moore’s Law expectation, practitioners assert that progress in machine learning applications isn’t exponential… it’s sigmoidal, an S-curve.

Last year, Starsky Robotics (one of the earliest self-driving truck startups) eventually shut its doors for good. Despite a string of excellent milestones and firsts for the autonomous vehicles (AV) space, Starsky’s CEO Stefan Seltz-Axmacher credits unmet promises and overinflated expectations as the company’s kiss of death.

A common joke in the industry is the first 90% of progress in a machine learning application is almost always pretty achievable… it’s the next 90% that’s so tough.

All models work well in small, controlled environments. But real-world deployments bring a plethora of edge cases you can’t always simulate, and depending on the required safety bar, you may not be able to resolve these cases in a window investors or executives are comfortable with.

The below chart captures this dilemma pretty well. The horizontal lines represent hypothetical human-equivalent performance levels.

If L1 is the line of human equivalence, then leading AV companies merely have to prove safety to be able to deploy… If L2 is the case, the bigger teams are somewhere from $1–25b away from solving this problem… If, however, L3 is the line of human equivalence it’s unlikely any of the current technology will make that jump.

Whenever someone says autonomy is 10 years away that’s almost certainly what their thought is.

The investing community’s exponential pitch creates a misalignment with operators when the reality of the S-curve comes to light.

Starsky’s failure was not one of competence or technical ability. Safety (both real and perceived) is one of the biggest hurdles for AV, and Stefan’s team invested nearly two years doing nothing but safety engineering. Pursuing that path with realistic milestones and centering on a business model that (though lower than traditional software margins) at least got the wheels turning was a strategy investors found little enthusiasm for.

It took me way too long to realize that VCs would rather a $1b business with a 90% margin than a $5b business with a 50% margin, even if capital requirements and growth were the same. And growth would be the same. The biggest limiter of autonomous deployments isn’t sales, it’s safety.

I wholeheartedly recommend reading Stefan’s excellent blog post in full, and to keep his thoughts in mind anytime you see exponential curves or hear the words “AI is taking over the world.” There are lots of low-hanging fruit, but overinflated expectations and runaway hype trains recklessly divert capital and resources from the most feasible and high ROI projects to the sexiest stories that are easier to sell.

A Personal Note

Early in my career, I made a lot of the same mistakes mentioned above. Coming from a business background rather than engineering, I’ve hyped the potential of some projects to executive audiences as soon as we determined a clever way to apply machine learning to the problem. This creates unfair expectations on your team and converts what might have been a few small wins into one big promise you might not live up to.

I’ve found the most success by appreciating the limitations of our solution, narrowing the scope, and laying out a stepped roadmap for how to eventually get to the Big Idea that executives and investors want. For the 90% of businesses that aren’t yet finding success with applied AI, that’s what I recommend: start small.

Share Machine Yearning

Bonus Content

These are articles, reports, videos, and more worth checking out that didn’t get a featured mention this issue.

📚 Reading

Introducing Reinforcement Learning for Optimizing the Player Lifecycle (Unity)

Don’t yet know what reinforcement learning (RL) is and too afraid to ask? Unity’s case study is a great primer on how the technology is used to maximize customer lifetime value (LTV) by keeping players engaged and their wallets open.

The Semiconductor Supply Chain: Assessing National Competitiveness (CSET)

Required reading for anyone without an industry background to get familiar with the hardware powering machine learning applications (CTRL+F “AI ASICs”).

National AI Initiative Office Launched by White House (FedScoop)

The US formalizes a commitment to funding AI research of national interest and importance.

Why the OECD Wants to Calculate the AI Compute Needs of National Governments (VentureBeat)

“Compute” is a generalized term for computing capacity used to train and deploy machine learning models. “If we can measure how much compute exists within a country or set of countries, we can quantify one of the factors for the AI capacity of that country.”

China’s STI Operations: Monitoring Foreign Science and Technology Through Open Sources (CSET)

The US relatively neglects open-source intelligence (OSINT) while China uses it as the “INT of first resort.” This contrasts extends to science and technology intelligence (STI), used for assessing foreign capabilities in applied research and engineering.

The U.S. AI Workforce (CSET)

CSET defines a taxonomy for AI labor supply sizing. Part 1 of a 3 part series.

This Chinese Lab is Aiming for Big AI Breakthroughs (WIRED)

Take a tour through one of the newest government-sponsored research labs, the Beijing Academy of Artificial Intelligence. Spoiler alert: they’re working on a GPT-3 / Switch Transformer competitor.

Amazon opens Alexa’s advanced AI for firms to build their own assistants (Business Standard)

Intelligent assistants in cars, apps, real estate, games, and edge devices galore.

🎧 Listening

Dan Wang on China’s Mission to Be a World Leader in Semiconductors (Odd Lots Podcast)

Dan Wang of Gavekal Dragonomics gives on-the-ground perspective of China’s efforts to de-risk itself from the global semiconductor supply chain.

🎥 Watching

People’s Republic of Desire (Documentary)

At the intersection of technology and culture, “A digital fantasy world where culture has been abandoned in favor of commerce.” Timely for the Kuaishou IPO.

About me

I am an AI Product Manager at Spiketrap, a seed-stage language processing company where I focus on applied strategy, analytics, and new verticals for our NLP stack. Previously, I spent ~4 years at Uber integrating machine learning techniques into high-growth marketplaces like Uber Eats, Uber Elevate, and JUMP. I started my career in technology investment banking at Credit Suisse, specializing in fintech, blockchain, and Chinese internet companies.

I’m studying Artificial Intelligence part-time at Stanford, and graduated from Georgetown University with a BS in Finance and Economics.

Thanks for reading!

Any suggestions or items you want to see? Contact me on Twitter @rydcunningham or shoot me an email at rydcunningham@gmail.com.

Sign up now so you don’t miss the first issue.

Subscribe now

In the meantime, tell your friends!