r/LocalLLaMA • u/chemist_slime • 20h ago
News Any takers for local llm? NVIDIA’s CUDA Moat Is Accelerating RTX Spark Laptop Sales, Higher 128GB Unified Memory Options Belonging To Surface Laptop Ultra Already Out Of Stock
https://wccftech.com/nvidia-cuda-moat-rtx-spark-laptop-sales-surface-laptop-ultra-128gb-out-of-stock/Cheapest one I see at MC is the asus pro-art version for 6499. https://www.microcenter.com/product/715662/asus-proart-p14-14-wqxga-120hz-oled-laptop
97
u/Formal-Exam-8767 20h ago
CUDA is not a moat when it comes to LLM inference. Memory bandwidth is the moat.
32
u/More-Curious816 19h ago
with frontier models, no company has a software moat, people currently porting kernels and writing custom engines for Vulkan and metal on AMD, Intel and apple hardware. some make CPU only solution, clean the dust out of old hardware or even try his luck with raspberry pi. all you need is a dedicated person, willing to learn, and using AI to guide him.
and I agree with you. bandwidth is the new moat.
3
u/tossit97531 13h ago edited 5h ago
I've been saying it for weeks once I realized it: we don't pay for GB, we pay for GB/s. All this "128 GB UMA!" means shit if it's
LPDDR5low-bandwidth.2
u/More-Curious816 12h ago
LPDDR5 actually a very good solution, it's designed to be flexible to be mold into your hardware demands. Apple for example, they already have achieved 1.2 TB/s (1,200 GB/s). that just proves that AMD and NVIDIA are cheap with configuring it to have higher bandwidth.
1
15
u/vtkayaker 18h ago
I mean, even the frontier labs are increasingly saying, "ChatGPT/Claude/DeepSeek, please port this GPU kernel to the new experimental hardware. Make no mistakes." And by checking the old output against the new output, they can mostly verify it (except for rounding errors on different CPUs). This is also why there are suddenly 30 new inference engines optimized for specific model/GPU combos.
Nvidia has high memory bandwidth and fp8/fp4. The AI Pro R9700 is fantastic card, but it's got about a third the memory bandwidth of a 5090, and it doesn't have native hardware smaller than fp16.
And, sure, fp4 usually costs quality compared to a good 4-bit integer quant, but sometimes you just want speed, you know? And you can train models for native fp4, as several labs have done.
8
u/valarauca14 16h ago
Next generation of AMD & Intel cards have MXFP4/FP8 native, which will close that gap to an extent.
The "real" advantage NVIDIA has is the PTX byte able to represent that the result of an operation is sent to memory not registers, which is good for a lot of matmul/register pressure heavy work, and AFAIT it has native ops to do this.
3
u/colbyshores 15h ago edited 15h ago
The thing is that once you have ported one LLM to a architecture(wither that be Nvidia, AMD, Vulkan, Intel, or whatever like Huawei), that matrix kernel is largely transferable for other LLMs. It's why getting a new model on to that hardware takes days.
5
u/Sleepnotdeading 18h ago
I feel like you’re conflating moat with bottleneck.
3
u/More-Curious816 15h ago
Moat is always what the company has to solve a bottleneck, basically an advantage over competitors.
CUDA was a moat when the hardware playing field had leveled with AMD, its software stack gave Nvidia an advantage over AMD’s dogshit software stack. Today, that software problem is no longer difficult, hardware engineering has become the new moat. Bandwidth is embedded in the hardware ecosystem, not just a software-level constraint, and so far Nvidia has the best hardware.
2
u/Etroarl55 11h ago
CUDA is still a moat for stuff outside LLMs. Video gen and image gen and increasingly popular topology is still basically either CUDA only or optimized heavily for CUDA. Amd will work, just nowhere near as fast or support.
1
u/HeavenlyAllspotter 2h ago
what kind of topology?
1
u/Etroarl55 2h ago
It’s a term I think the 3d modeling ai space hijacked over at topologyai.
Basically ai 3d modeling.
2
u/DueAnalysis2 18h ago
How important is CUDA for something like fine tuning?
1
u/More-Curious816 15h ago
not that important, if you have the knowledge or use another person script, you can do it on other hardware brand. but Nvidia has both cuda and the extreme bandwidth and compute, which is why people still use it. if AMD or APPLE made a chip that as powerful as Nvidia’s top of the line GPUs, it will be a major market pivot to that hardware for training and fine-tuning. Nvidia’s still has the market because they offer the best hardware for data centers.
1
-3
u/BumbleSlob 19h ago
ITT: ignoring the memory bandwidth of sparks which are only good because of cuda
2
22
u/thefuckevengoingonan 19h ago
do people nor realize they can just buy a server and connect from any cheap laptop? that way you aren't out $7000 when you leave you laptop on the train
7
6
12
6
u/Last_Bad_2687 17h ago
I have the strix halo framework desktop and pre-ordered Medusa Halo. Gufo/Halogen make it almost as good as a DGX spark
1
6
u/HighSeasArchivist 17h ago
For a corporate/government setting Windows on Arm is still a scary thought. We are looking to try again, again, again with Windows on Arm, and maybe this time Microsoft has got their shit together. Right now we are planning on getting a few Surfaces to test, but I want to try a 192GB Gorgon Halo when they come out as well.
1
u/This_Maintenance_834 16h ago
a decent open weight model is all you need.
if Alibaba can release a refresh of qwen3.8-flash-next, it will be a killer app of RTX Spark.
As is, it runs on single GB10, with not too slow token generation and prefill speed, if they can train the model to be less verbose, that will be a very good solution for local deployment.
every month new model coming out, i won’t surprised some labs release one for this type of ram footprint.
2
u/HighSeasArchivist 16h ago
I run Qwen at home, but at work it has to be US only. Gemma4 31B is our default, and hopefully Gemma5 will be out soon. I haven't played Llama4 Scout, but maybe being MoE it would run well on a unified laptop.
2
1
u/More-Curious816 19h ago
bad hardware to get, even if you are a developer, it has no ecosystem and you will develop for a niche, if you need it for pure inference and "need" Nvidia, get the DGX Spark or consumer GPU.
1
1
u/Kindly-Bank-416 14h ago
at best its a dgx spark in laptop form factor. I'd rather have the SFF form factor.
1
u/unjustifiably_angry 11h ago edited 10h ago
I can't wait until we're out of this era with hardware that has half the VRAM bandwidth of an entry-level GPU. It would probably be worth investigating how much gain might be possible from running a draft model on an external GPU.
0
0
-1
u/Correct-Call3004 19h ago
that 128gb would be sweet for loading bigger uncensored models without constant swapping during long sessions
0
u/follimath 18h ago
Is the RTX Spark’s SoC RAM upgradeable?
Doubt it. It’s beyond me why anyone would pay for this over the DGX Spark.
40
u/feelspeaceman 20h ago
Too passable, so no.
I would take Bosgame M5 for $3000 running Q38FN at 50-60t/s decode 1300-1800t/s prefill using gufo/halogen/strixite all days over this.
A laptop for local inference is like the worst choice unless under extreme contrain like space limit, portability or company forcing NVIDIA hardware, it's not ideal for long terms.