Recently, a spate of AI data centers consuming upwards of 1 GW have been under construction, well more than any existing supercomputer. Could an efficient true, coherent supercomputer be constructed at that or even greater scales, given current semiconductor technology?

[u/GrantExploit]

For slightly over half a decade, I have noted that the largest supercomputers on Earth consume orders of magnitude less power than the largest industrial facilities on Earth, and have thought that a supercomputer-at-gigawatt-scale would (obviously) be highly useful for certain types of modelling. At ~10²⁰ FLOPS₆₄ or FLOPS₃₂, these could include near-perfectly-accurate direct numerical CFD/FEA simulations of vehicles (of all types) and engines at faster-than-real-time speeds; and physically-based molecular simulations of eukaryotic cells, tissues, and simple multicellular organisms.

Even more recently, a large number of massive AI data centers have been built or are in the process of being built. My perspective is that this development is an outright existential threat to the very fabric of human civilization... but, still, finally, people are aggregating hundreds of megawatts or even gigawatts of computational horsepower into defined places!

However, these AI data centers are quite architecturally distinct from supercomputers. While like supercomputers (and unlike the data centers of the past; with their tasks including archiving social media content and running light web-based SaaS, earlier data centers were proportionally much more oriented towards storage and networking) they contain considerable arithmetic power, AI data centers are still geared for running myriads of individual agents for remote users (or, increasingly *shudders* themselves) rather than working as a whole on one or a few big tasks. As such, computational density and node interconnection are not quite as high priority as with supercomputers, as they don't have high task coherency that could be crippled by insufficient bandwidth or high latency through signal propagation delay. In other words, if, say, Colossus 2#Colossus_2_and_cluster_expansion) were to run the HPLinpack benchmark, its results would likely be very disappointing compared to the theoretical capabilities of all its constituent GPUs.

However... while current AI data centers aren't under particular pressure to be compact, the most powerful supercomputers aren't very compact either! El Capitan, the second most powerful supercomputer on Earth, takes up ~700 m²—a spherical-cow calculation indicates that a signal originating from a blade at one end of the system takes at least ~320 CPU cycles to get to the other end of the system, imposing a latency beyond the most feverous dreams of an early-2000s NetBurst engineer and already making task coherency difficult and the system less performant than it would ideally be. Crudely expanding this to a gigawatt would increase this figure to beyond 1,800 cycles... yikes. When looking at high-end supercomputers it is clear they have a large amount of "wasted" space, but it is also clear that significantly reducing that could make maintenance, power delivery, and cooling a nightmare. One might be tempted to build a gigawatt-class supercomputer in 3D in some sort of high-ceiling industrial facility to reduce signal distances, though it must be kept in mind that beneath their orderly floors lies a labyrinth of networking cables, electrical lines, and cooling water pipes; in other words, they take up more volume than might initially be apparent.

And so, the question.

[u/subtle_bedtime]

the latency problem is the real killer here, not raw flops. everyone gets distracted by the power numbers but forgets that a coherent machine needs tight synchronization and low signal travel time. once your round-trip latency gets past a few hundred cycles the overhead of keeping everything in lockstep eats away most of the theoretical performance gains

i worked on a smaller cluster project a few years back where we tried collapsing the physical footprint by stacking nodes vertically and running fiber in the ceiling plenum. cooling was a disaster, we kept tripping thermal limits on the upper racks no matter how much airflow we shoved through. maintenance was worse, pulling a single node meant unthreading a rat's nest of cables that snaked three racks over

at gigawatt scale you're basically fighting the speed of light in fiber, you can't cheat physics. the networking fabric alone would cost more than the compute. might be better to rethink the problem and accept that some workloads just don't parallelize cleanly past a certain point, no matter how much juice you dump into the building

[u/GrantExploit → u/subtle_bedtime]

Really need to get some sleep, but I just wanted to say now that I did mention most of those concerns in the description.

[u/Toyota__Corolla → u/GrantExploit]

So if you don't want reiteration the answer is no. We have bulky inefficient hardware that can't get any smaller with our current building blocks

[u/looktowindward]

> My perspective is that this development is an outright existential threat to the very fabric of human civilization..

What a strange supposition. That you begin with this, bereft of context, makes me not want to engage.

>  One might be tempted to build a gigawatt-class supercomputer in 3D in some sort of high-ceiling industrial facility to reduce signal distances, though it must be kept in mind that beneath their orderly floors lies a labyrinth of networking cables, electrical lines, and cooling water pipes; in other words, they take up more volume than might initially be apparent.

That is exactly what happens today with AI - things are laid out in three dimensions. I'm not sure you understand the difference between a conventional data center, an AI data center (which are identical except for GPUs vs CPUs), and a supercomputer.

[u/arielif1 → u/looktowindward]

an AI data center (which are identical except for GPUs vs CPUs), and a supercomputer.

I mean, not really. AI data centers are barely data centers at all, since they barely have any storage. It's nothing like a cloudflare site for example. They really are just much closer to very application-optimized supercomputers (optimized mainly for memory bandwidth limited tasks)

[u/looktowindward → u/arielif1]

A "Cloudflare site" - is a CDN POP. They don't have much storage by datacenter standards due to long-tail CDN distribution - on a cache miss, they pop up to a higher tier CDN node.

> They really are just much closer to very application-optimized supercomputers (optimized mainly for memory bandwidth limited tasks)

Memory bandwidth limited? An AI training data center has VAST cross-sectional bandwidth due to RDMA. And the memory is absurd. Slightly less for inference, but when you have a 10T model, the memory demands are huge.

Go to SC2026 - they cover this pretty well.

[u/arielif1 → u/looktowindward]

uuuh... aside from prefill, which is indeed mainly compute-gated, token generation is sequential due to it being causal. it is obviously memory bandwidth constrained. i don't know whether you misunderstood me or what but that much is obvious, you can't generate token n+1 until token n is generated, and thus you need a whole ass forward pass per token, so you have to pass the entire (active for moe) parameter count from memory to the gpu cores

[u/Bryguy3k]

Current AI systems are built around neural networks of various architectures which means a massive number of small calculations which is why GPUs took over as folks had developed GPUs to have a massive number of small cores to do ray tracing in parallel.

Super computers are for different kinds of problems - ones that are often not easily broken down into tiny little computable problems like neural networks.

[u/me_myself_ai]

All I can say for sure is this: the supercomputer nerds agree with you! At least in the sense of wanting them to run the benchmark on Colossus 2 in particular.

In other words: we’re not quite sure to what extent datacenters could function as supercomputers, or how feasibly they might be converted physically.

It does seem feasible (esp with AI help) but the economic pressures just aren’t there. We just recently got a newest largest supercomputer in China, and idk what it’s even for…

—-

My personal opinion as a SWE with very little hardware expertise is that “high task coherency” is a little outdated. Maaaybe some of the things you mentioned are only possible with a single unit, but distributed computing is so advanced these days that it feels like you could spread work pretty darn efficiently! Both in the “separation of concerns” sense and the “sharding big jobs using incomprehensible maths” sense.

Another personal musing (I love this question!): I think the hardware conversion would be at the level of the “switches” (to be extremely vague), which puts something of an upper limit on the job. AFAIU the individual racks themselves are almost OTS now that NVIDIA is playing the datacenter game to the max, so they likely wouldn’t need changes.

Ok final musing: in a sense, training supermassive models is kinda a supercomputer task, no? Hell, it’s measured in FLOPs. And clearly they’ve managed to do that quite well on some centers!

[u/me_myself_ai → u/me_myself_ai]

OH huge addendum: the space/compaction stuff will change wildly now that liquid cooling is rolling out (accelerated by politics around water usage). So building a supercomputer will get easier

[u/Vitztlampaehecatl → u/me_myself_ai]

The politics about water usage are in relation to evaporative cooling, not watercooling. Watercooling a CPU involves a closed loop between the heatsink and the radiator, so it really doesn't consume a lot of coolant over time.

Evaporative cooling is an alternative method of air conditioning without using a compressor. Instead, air is blown through a spray of water droplets, which evaporate quickly because of their immense surface area, thus absorbing heat from their surroundings. This is the cooling method used for enormous buildings like college campuses, shopping malls, and power plants. It requires a steady stream of potable water because all that water getting sprayed into the air and evaporating needs to be continuously replaced.

You can combine either type of internal heat transfer with either type of air conditioning, so you could watercool your servers and still use a traditional air conditioner to keep your coolant cool and not waste any water. But this becomes impractical for something as big as a modern data center because of capital costs and electricity consumption. 

[u/me_myself_ai → u/Vitztlampaehecatl]

Thanks for expounding! All looks very exciting from here.

[u/logperf]

I mean, is there much to be gained in terms of cost/benefit ratio?

AI relies mostly on parallelism. So your supercomputer would have to be designed to have lots of processors running in parallel. We already have SMP UMA architectures so that's nothing new, but you'd have to scale it up to insane amounts of processors and a huge memory. And you guessed correctly the engineering challenges of cooling it down, passing cables/lines.... there's another one you didn't mention which is EM interference between parallel lines, manageable in small motherboards, but it would be a nightmare if they are huge.

So, it'd be a great engineering challenge, coming at a huge design cost. For what? To achieve the same result you can get with thousands of comparatively small servers running in parallel, at a more manageable cost?

[u/Wondering_Electron]

These data centres aren't being built with over half being cancelled or delayed.

There is going to be a reckoning when the debt is due for repayment on these and then even more builds will collapse.

[u/looktowindward → u/Wondering_Electron]

Except most of the ones NOT being built have no debt because they get cancelled before that point.

> There is going to be a reckoning when the debt is due for repayment on these and then even more builds will collapse.

Meh, unless you are talking about Oracle, I'm not so worried.

[u/Wondering_Electron → u/looktowindward]

80% of the debt taken on are non cancellable, so how do you square this circle?

[u/looktowindward → u/Wondering_Electron]

The debt is not taken on until a lease is signed and construction commences. The fake projects never get that far - no leases, no debt.

And a lot of the debt, in the early days of a project are for long lead time materials which are fungible - equipment not labor.

If you were in the business, you would know this.

You have to REALLY fuck up to risk your debt - again Oracle. Otherwise, not so much. Your equity, on the other hand, is in jeopardy.

> 80% of the debt taken on are non cancellable, so how do you square this circle?

This is nonsensical. You typically have an 80/20 debt to equity ratio. None of the debt is "cancellable" in the sense you are implying. But its collateralized with the gear you've ordered.

So far, no one's debt has been called in. I think some will, but the more likely situation is that the debt markets will tighten for a few quarters as the ratios unwind.

[u/me_myself_ai → u/Wondering_Electron]

:( please look up