Rendered at 19:48:14 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
YuechenLi 1 days ago [-]
Nvidia's biggest advantage in AI has never been only their hardware performance but how entrenched their software is in ML research that flowed down stream. However, if you've actually used CUDA C/C++, it's pretty one of the worst software development ecosystem imaginable: you get all the footgun of regular C++, plus GPU compute pretending to be C++ and but doesn't actually behave like C++ because CPU and GPU compute are fundamentally different, and the only reason people put up with it is because Vulkan and HIP C/C++ are even worse.
Google's limitation is that they still don't offer TPUs in a PCI-E card/dev board that people can plug in to their PC for local development and sane low level API to develop against, instead you have to go through their cloud and their full software stack which greatly limits ecosystem growth. The minute that Google figures that out, that's when Nvidia's dominance would be challenged.
bri3d 16 hours ago [-]
The CUDA runtime coming with a gazillion reasonably decent kernels (DNN, BLAS, CUTLASS) and a concurrency system (NCCL) is a big deal; especially in the “early days” very few researchers or development runtimes were even writing their own kernels or dealing with CUDA C++ extensively, they were wrapping the ones NVidia gave them.
I do agree that it’s really not great, and I also have never been a strong believer in the CUDA moat overall; as the need for GPUs moves from research to production (inference), companies are plenty willing to build software from scratch anyway (and we see this with AMD GPUs being in plenty high demand in the datacenter and enthusiast market now).
galaxyLogic 14 hours ago [-]
What I don't quite get is why can't they use AI to translate CUDA programs into more open architectures like AMD ROCm?
AI is supposed have solved the "coding problem". But shouldn't translating a program from one platform to another be an even easier, more mechanical, task for the AI?
bri3d 4 hours ago [-]
No, because the two platforms often don't share the same underlying kernels; this has been one of the main issues and complaints with ROCm/MIOpen since the start, although they are catching up slowly.
This is actually a corollary to the point I was making about "CUDA" usually also including a ton of the included kernels and not just referring to a crappy programming environment; translating mid-level C that does math between two runtimes wouldn't be hard for an LLM, but translating "doBigDNNThingNVidiaGaveMeInAKernel()" to "doBigDNNThingByHandBecauseAMDDoesntSupportIt()" isn't a rote translation at all.
Of course, once you accept that it's _not_ "why don't you just translate it," you _can_ iteratively use an LLM to implement the ThingNVidiaGaveYouInAKernel, but it probably isn't well-trained, yet, on low-level AMD optimization tricks, so the kernel you end up with will likely be slower than the CUDA one.
galaxyLogic 3 hours ago [-]
> translating "doBigDNNThingNVidiaGaveMeInAKernel()" to "doBigDNNThingByHandBecauseAMDDoesntSupportIt()" isn't a rote translation at all.
I wonder if this points to a deeper limitation of AI, it can not do coding tasks it has not seen in its training material. Or could it possibly "generalize" to accompllish something like this anyway?
mdp2021 13 hours ago [-]
> AI is supposed have solved the
Which AI? LLMs are coding facilitators and code producers.
A problem is solved when the solution is reliable. Non-deterministic Neural Networks are not reliable. In fact,
> more mechanical[] task
that suggests an expectation of process and procedure, which is still not a capability of current architectures.
Sure, you can ask a brains-deficient operator to perform a huge task, but then you'll have to check the whole product, and that remains not cheap.
larnon 8 hours ago [-]
The tool itself (AI) may not be reliable, but that is also very true for every other tool (e.g. Human). Also, you are right about when a problem is solved, but this doesn't need the tool to be reliable as you said, just the solution part. Hence, as long as the produced code works as intended, it doesn't matter what you used to produce the output.
galaxyLogic 3 hours ago [-]
Saw this on the web:
"AMD and Anthropic also formed a multiyear engineering partnership to optimize ROCm using Claude"
Undoubtedly, the frontier labs are already doing something akin to this as part of their new chip design endeavors.
ravenstine 1 days ago [-]
That's really interesting. I have no experience writing anything that involves GPUs/TPUs, but over the years I've consistently read that CUDA is the "real moat" of Nvidia, which I never totally believed, but the way you describe makes it seem like it's not actually a moat in the slightest. It just happens to be an ecosystem associated with hardware that is not only considered the gold standard but happens to be more open than potential competition. Could it be that Nvidia has been on top because none of the competition has actually tried kicking them where it hurts?
kevstev 22 hours ago [-]
I think OP is overstating it a bit tbh. So Nvidia has the market for the hardware, which helps, but there is literally no alternative to CUDA. Nvidia keeps it a scalpel for skilled users, its not super easy to use, but unlocks orders of more magnitudes of power for the use cases it excels at vs CPUs. I don't have access to anything like it in the Apple ecosystem.
AMD has had years to try and counter it, but just has not. Google is kinda trying to do an end run around it with TPUs but they are still niche high end stuff with limited availability.
Its really just CUDA, and CUDA can be seen as somewhat akin to C for assembly used by Nvidia's gpus- In many ways a wrapper around the low level hardware that often has those details bleed through.
YuechenLi 21 hours ago [-]
I think there is a misconception here: CUDA is not even close to being C for assembly used by Nvidia's GPUs, PTX is, and it uses JIT to compile to Nvidia's GPU assembly, SASS. It honestly easier just to have LLMs write PTX directly than to go through CUDA C/C++ at all.
Again, "CUDA" isn't a programming language, it stands for Compute Unified Device Architecture; "C/C++ for CUDA" are the high-level languages that compiles to PTX and then SASS as well CPU orchestration code via NVCC.
And to be honest, pretty much everything you can do in CUDA C/C++, you can also do in HLSL/GLSL compiled to SPIR-V, as long as the Vulkan hardware extension is available.
kevstev 19 hours ago [-]
My point with the c comparison is that it often feels like a very thin layer that still requires you to know a lot about what's going on under the hood. It was more to give a view in what it's like to work with the lib/api.
Maybe I wasn't being precise enough with my language for this forum, and also my last hands on experience with it was roughly 6 years ago, maybe it's gotten better. But it was much less (and forgive the imprecision!) python/pytorch-like where you say hey take this big blob of data and just slice and dice it on your many cores, and more like ok, here is the data, let's cudamemcopy it in these size chunks over to the gpu itself, to be used by this block of threads and run these commands (kernel in cudaspeak) on it. Much more painstaking and micromanagey of the resources.
Pytorch IMHO feels like a proper abstracted API that hides the details and lets you just unleash the fury at the cost of some efficiency, while the cuda api itself, similar to working with C, forces you to really think about the low level details. I have a heavy backend and systems development background, and while it wasn't really intimidating to me, it was like wow you really have to have a deep working knowledge of how these things work and it felt like a step back in time IMHO.
I doubt that's going to satisfy you but I think it gives a clearer picture of what using cuda is like if you typically use higher level languages and haven't touched C since college.
YuechenLi 17 hours ago [-]
I think you are pretty precise and your comment is appreciated, so I think I need to clarify my wording as well. I think a lot of CUDA C/C++'s difficulties are self-inflicted because they conflate two things into what they call "kernels": runtime and shaders. Fundamentally, GPU programming for compute is easier than either graphics programming as well as CPU programming because you are limited on what you can do on the GPU, memory allocation is static because the matrix size doesn't really change dynamically during runtime and branching behaviors are generally to be avoided for GPU compute.
It's pretty heretical for me to say this, but a lot of GPU compute complexity that Nvidia is doing in CUDA is unnecessary and is by the simple fact that to do anything meaningful you have to either use their library or handle allocation/scheduling yourself. Imagine if JavaScript required you to handroll part of the V8/Node's JIT compiler, allocator and scheduler yourself every time you just want to make a webpage, that is essentially what CUDA is doing.
The actual "program" that runs on the GPU, the compute shaders in PTX/SPIR-V, are very low level but pretty straight forward once you get down to it.
compiler-guy 1 days ago [-]
To paraphrase the apocryphal Winston Churchill quote about democracy:
“CUDA is the worst development ecosystem in existence. Except for all the others.”
lostlogin 23 hours ago [-]
‘Yes, I Am Drunk, But CUDA is Ugly. Tomorrow I Will Be Sober, And CUDA Will Still Be Ugly.’
elictronic 24 hours ago [-]
Retraining a large high paid user base is often a non-starter. To put this in perspective, Boeing’s eventual retraining costs for all the pilots for the 737Max was around 5 billion dollars.
Looking at software more specifically the Linux foundation reported based on software dev salaries in 2008 it would be 1.4 billion to only write the Linux kernel.
Up until about 2023 there wasn’t enough money involved to have any reason to make a real CUDA killer even if you could get it adopted.
0xDEAFBEAD 16 hours ago [-]
As agentic coding continues to improve, won't it get easier for devs to retrain for new languages/frameworks/etc.?
galaxyLogic 14 hours ago [-]
And couldn't we just ask AI to translate our program in one language/framework into another?
I was under the impression that AI was supposed to remove software-moats, let us all ask it to write our custom MS Word for us for instance?
mandeepj 23 hours ago [-]
Yet, Microsoft pulled it off with a new .NET Framework, and Apple with its new iOS SDK. There are many more examples besides those two.
eterm 22 hours ago [-]
Microsoft pulled it off with dotnet, sort of, because they approached it like a completely new language, sold people on the benefits of it.
And the people they were selling that to, ( It's free and open souce now! ), were a very different group to the market they left behind on .NET Framework, who are often still struggling to make the transition now.
Had they actually killed off .NET Framework, it would have been a different story, much more like the VB6/VBA to VB.NET transition, which so few people bothered with that VB.NET died out, because if you had to retrain that much, you figured you might as well go to C# or a instead, or indeed a completely different language entirely.
I briefly worked professionally on a VB.NET project, but outside that job I've never met anyone else who can say the same. I've met a few who went straight from VB6 to C# though.
radicalbyte 13 hours ago [-]
The whole push behind .Net Core was to get .Net running on Linux well and natively and was being lead largely by those of us using .Net for web. We were being murdered by other languages at the time; mono was an option but Microsoft shops usually needed to target something blessed/backed by MS.
Largely the same market (Enterprise) but not the different segment (web as opposed to Windows/WinForms).
I ported about 15 years of projects from various versions of .Net to .Net Core whilst they were developing (and sent feedback to the team - they were asking us to do that) and the process was pretty reasonable. You were only really stuck if you were using something very very Windows specific (certain image processing libraries iirc) and even then it was largely manageable.
The old full-fat framework is, AFAIK, still supported, as there's a whole lot of legacy code which is Windows specific which is still expensive / hard to port over.
eterm 13 hours ago [-]
Unfortunately "something windows specific" was pretty broad.
Between MSMQ, WCF over named pipes, MSDTC, and MSI installers, there's a lot to replace that is hard to provide the same guarantees or performance with straight replacements, if they even exist.
The end goal, being on modern dotnet, is better, but it's difficult to get there with a phased approach without accepting a temporary worsening, which is often hard to sell.
Especially while Framework is still supported.
compiler-guy 23 hours ago [-]
Both of those were a vendor X electing to stop updating framework A in favor of framework B. That’s a high cost for vendor X’s users, but if their business depends on vendor X, they have no choice. Maybe they can switch to vendor Y at that point, but now you are switching both vendors and frameworks.
Replacing CUDA with another framework has much lower motivation. That advantages of the new framework must cover the switching costs and the risk of such a switch. All while CUDA continues to evolve and allow access to additional features.
Apple and Microsoft had something of a captive userbase. New vendor on the block trying to replace CUDA does not.
dragonwriter 22 hours ago [-]
Neither of those are a competitor replacing another, they are a same-vendor replacement. This is easier, because the company whose established product you are trying to displace is cooperating (because its you!) not actively resisting.
galaxyLogic 14 hours ago [-]
Anthropic rewrote Bun in Rust, with much help from AI of course
cepp 19 hours ago [-]
One of the rarely-mentioned value adds Nvidia provides is nccl[1] which makes multi-node networking and topology essentially plug and play. The other players have since caught on [2][3] and are working hard to catch up but I'd say networking is a real moat.
It's a totally reasonable question, and one that everyone asks when they're learning about CUDA. The frustrating answer to your last question is that lots of companies have shipped GPU dev environments that can theoretically be used instead of CUDA. AMD has ROCm, Apple has had a couple projects (OpenCL, Metal), Intel has some stuff, and there are newer efforts like TinyGrad + a generation of slightly higher level frameworks from AI companies, like Triton from OpenAI.
The basic problem is that CUDA has become something of a Schelling point. If you want to train a model right now, the highest performance you can get is almost certainly on CUDA. From the basic general matrix multiply operation, to specific NN architectures, CUDA is going to have incredibly optimized implementations out of the box. And it's going to make multi-GPU training so much easier. And all the dependencies you build on (those layers you import from PyTorch or Transformers or whatever) are going to work optimally right away on CUDA. And that weird random repo that you found with a unique optimizer--it runs on CUDA too. And now the cool new implementation that you're about to release is also going to be built for CUDA.
It's so tempting to think "Just write replacement software", but you also need to transition the entire ecosystem in large part to match CUDA's effectiveness, and you need to get comparable performance out of your chip/library combo as NVIDIA can get out of its cards with CUDA.
There's a whole story here to how effective NVIDIA has been at navigating this. Very early on, they heavily prioritized PyTorch and TensorFlow, getting involved in the projects as much as they could and making sure they always ran best on CUDA. But the TLDR is that yes, you're right, another company could write a CUDA competitor. But actually replacing CUDA is a much larger task.
I'm personally hopeful that with the rise of coding agents, we see more movement on this front with other projects moving into view. It will take some time for any ecosystem to start to emerge that can dislodge CUDA for researchers who don't want to dive that deep into the stack, but hopefully we start to see some momentum build.
dannyw 17 hours ago [-]
CUDA has many problems, but I would say less problems than ROCm, etc; even before considering the ecosystem and that more people (or open source projects) have already solved CUDA's problems for you.
Ironically, there was an open source project that was making great progress on CUDA compatibility on AMD hardware. AMD hired the lead developer, and then he shut down the project.
bri3d 16 hours ago [-]
ZLUDA is still alive. AMD sponsored it and the project was briefly halted during a dispute with them, but it’s been making steady progress.
It doesn’t really make sense for AMD themselves or most use cases, though; any compatibility shim just adds problems on top of problems, and for AMD, entrenching a competitors technology even more never really seemed like a great idea.
csomar 1 days ago [-]
Software has always been the moat but for some reason it's always hamstrung by upper management. The latest of the frenzies being replacing sane (or whatever we have) of development practices with AI-slop.
Management likes it because it removes software developers from the loop.
szundi 1 days ago [-]
[dead]
musebox35 1 days ago [-]
The biggest advantage of tpus is the high bandwidth fiber optic interconnect between them that allows distributed computing on pods with thousands of tpus and the co-design of cooling systems that go with their racks. I do not think that we will see personal tpus any time soon.
mycall 20 hours ago [-]
My Pixel 9 has a personal tpu
szundi 1 days ago [-]
[dead]
whatever1 1 days ago [-]
Now with LLMs why a programming framework is a moat?
Someone 13 hours ago [-]
> Google's limitation is that they still don't offer TPUs in a PCI-E card/dev board that people can plug in to their PC for local development
... with performance greater or comparable to even the lowest performance Nvidia card?
tomaskafka 1 days ago [-]
I had a hard time understanding why didn’t AMD make a better developer experience for this two years ago, and am now even more baffled that even with all the LLMs they still don’t seem to have moved a single inch, despite this probably being a tens of billions dollars worth feature.
kllrnohj 24 hours ago [-]
LLMs make the dev environment almost irrelevant. llama.cpp supports AMD with both ROCm and Vulkan and that's nearly all that matters now. TBD how much AMD's AI Halo play will change things if at all, but they got a lot of positive press in launch reviews for having an actually robust software story for once.
CorrectHorseBat 23 hours ago [-]
Hardware companies are notoriously bad at software. The software they use sucks, the languages they use suck, the internal tooling software they write sucks.
They don't know what good developer experience is, how do you expect them to deliver it to other people?
schopra909 23 hours ago [-]
I’m not entirely sure if local development will lead to Nvidia’s supremacy being challenged.
I think a simple reason why it’s been hard to unseat in Nvidia is first mover advantage. A lot more water has flown through Nvidia pipes than TPUs or AMDs chips for that matter.
TPUs and AMD chips aren’t priced cheaper than NVIDIA (at least for my purposes training models). So there hasn’t been an impetus for me to venture there and use those chips.
Anecdotally, folks I know who have tried using TPUs and AMD chips have hit more issues with the underlying drivers than with NVIDIA chips. That costs time and money to fix.
Eventually the other chips will go through enough iterations and stability will be reached
npunt 24 hours ago [-]
Are the switching costs of CUDA ecosystem potentially threatened because LLMs are now quite good at transcoding into other languages? In other words, is Nvidia's greatest strength (AI) also potentially its undoing?
robocat 20 hours ago [-]
> Google's limitation is that they still don't offer TPUs in a PCI-E card/dev board
Nvidia's sells hardware yet their market cap is about the same as Google's.
How much value could Google get by selling hardware too? Google'd be selling to competitors, so difficult to capture much of the value and would decrease Google's value as an AI company. Maybe a child company?
bdangubic 20 hours ago [-]
Google sells TPUs already, to competitors :)
ijidak 22 hours ago [-]
Genuine question. Given that LLMs are supposed to allow us to rewrite anything, and I am an LLM believer, what I don't understand is: how does CUDA continue to be a moat in a world where LLMs can rewrite entire software development stacks? If NVIDIA is right about AI, isn't this same technology going to erode the software side of this same software moat?
polanyer 8 hours ago [-]
It seems like path dependent lock in to me and a risk/reward calculation.
What do you gain by not using CUDA vs what do you risk?
gr_norm 17 hours ago [-]
Indeed. Now, given that CUDA is apparently not being usurped, update your priors.
ceehex 20 hours ago [-]
well done you realised no one knows what they are talking about
akoboldfrying 18 hours ago [-]
Interesting take on Google's TPUs. What I've previously heard (and still believe) is that Google's decision to only rent out, never sell, their TPUs is a deliberate and savvy strategy for bolstering GCP, which will work provided that TPUs are able to actually compete with other hardware (in practice meaning Nvidia). A few months ago there was some discussion on HN comparing them, and I think the verdict at the time was that their latest-gen TPUs win on compute-per-Joule for LLM-type workloads by quite a margin, which I think is huge for those who want to run LLMs at scale.
HeWhoLurksLate 1 days ago [-]
I mean they had/have the Coral but that's in an entirely different market segment
bigyabai 1 days ago [-]
Coral and the TPUs are ASICs, and therefore are barely reprogrammable. It doesn't really compare to the complexity and flexibility of CUDA ALUs.
jcfrei 1 days ago [-]
In many investment theses - like Nvidia's bet that demand for compute will keep growing - the first order assumption is usually correct. Yes, demand for more compute, chips, infrastructure is huge and each year some additional data centers will be built. Where such investment bets usually fail is in the second-order assumptions: Ie. the expectation of the growth of demand. This is where there's a high chance that the current expectations are likely exaggerated. So: demand is likely to persist for the foreseeable future but not increase every year. And that can upend the whole investment story. That can be enough to make these bonds a huge burden for Nvidia in the end. Not because people stopped buying more compute but because they stopped buying more every year.
onlyrealcuzzo 1 days ago [-]
What makes this insanely hard to predict is that the compute needed for the same quality output has roughly gone down 90% every 18 months for ~5 years.
1) We don't know how long that trend will continue, but you do know where to look for when it may end (if smaller sized models continue to compress the knowledge effectively of larger models).
2) We don't know when the appetite for higher cost models might go down and by how much if smaller models get "good enough" and price becomes far more important.
It is entirely possible that 5 years from now, there's >100x LLM inference going on - but demand for AI chips (including memory) is only 2x or less.
It is also entirely possible that at some size - LLMs pick up some emergent capability that doesn't scale well to smaller sizes - and that there's an incredible boost to demand to get that capability.
It's just very hard to predict.
mattnewton 1 days ago [-]
I think efficiency is unlikely to result in lower demand for compute, instead more useful compute per watt increases the value of that compute; and we are not going to run out of economically useful things to do with it anytime soon on the demand side.
The harder thing to forecast for me is if we hit a wall on increasing efficiency, either on the model weights side or silicon side, with current approaches. If we have to switch to something like burning the model weights into silicon to continue to make gains, then the current math on general purpose accelerators might be upside down.
ekunazanu 1 days ago [-]
I agree; I don't think there's any reason to assume Jevons paradox won't apply.
> If we have to switch to something like burning the model weights into silicon to continue to make gains
I think that's already being considered semi-seriously [0][1]
What's really interesting is that if you scale it to higher densities (eg 3nm and stacked die) with ComputeInMemory for fp8 you can reasonably start to fit 30B-70B models. With MoE and multiple stacked die, just like HBM, you could fit an open weight near frontier 1T model (like GLM5.2) at similarly much lower power <10kW and high token rates >2ktps. For running a bunch of agents where fill rate and speed/latency are important it may not matter that you're 6-18mo behind on weights. The process for the chip design could be largely automated, and new silicon pumped out as new weights are available (with a 3-6mo delay).
ryukoposting 23 hours ago [-]
> efficiency is unlikely to result in lower demand for compute, instead more useful compute per watt increases the value of that compute
I buy that. Jevon's Paradox, sure.
> and we are not going to run out of economically useful things to do with it anytime soon on the demand side
This I don't buy. Not fully, at least. Whether or not there's demand for LLMs in some particular field is one thing, whether or not there is a sustainable business model to be built out of that demand is another thing entirely.
There is a staggering amount of money pouring into startups looking for novel use cases for LLM-based agents. As usual, 99% of them will fail, but those other 1% are going to have to look harder and harder to find a novel use case that can actually be served profitably.
First of all, there's only so many places where a chatbot is going to sell. But, that also seems to be the only interface anyone can come up with that allows a user to steer an agent through a long-running task reliably. I'd love to be proven wrong here.
Also, if current trends plateau and large datacenters are still needed for complex tasks, that would stimy growth of LLM usage across entire industries.
But, if present trends continue, then local inference will become feasible for most tasks. That would lower the barrier to entry across tons of heavily-regulated and/or cost-sensitive industries. But, widespread local inference will almost certainly come with a painful market correction centered around hyperscalers, which would itself dry up the pool for ventures into new markets.
the_sleaze_ 16 hours ago [-]
Replace "chat-bot" with "Human Being" because the models I've been using are significantly better than 90% of the human-chat-bots that I must talk to on the phone while scheduling and coordinating my internet installation for example.
Now for every human replacement, that is 1 unit less of communication and bureaucratic burden (HR, middle management etc) that the org requires.
ryukoposting 16 hours ago [-]
> the models I've been using are significantly better than 90% of the human-chat-bots that I must talk to on the phone while scheduling and coordinating my internet installation for example.
I guess I don't know what to say except that my experience is the polar opposite of yours.
I moved to a new state at the beginning of the year. Needed a new doctor, needed to schedule apartment tours, needed to talk to my employer about insurance and relocation stuff, etc etc. Lots of chatbots, a handful of humans. Humans consistently did what I needed them to do, the chatbots just didn't. I could list examples but I'd be typing all night.
And yknow what, my one call with Comcast to get my internet set up was downright pleasant. The rep was knowledgeable and a good conversationalist.
adrianN 1 days ago [-]
When efficiency reaches the point where local models on consumer hardware are good enough, demand for cloud tokens could rapidly shrink.
bryanlarsen 1 days ago [-]
Very few consumers are going to spend multiple thousands of dollars to save $10 per month. Companies absolutely will to save hundreds per month per employee, but that's not consumer hardware.
adrianN 14 hours ago [-]
Well if the trend that the comment further up in this thread claimed continues and compute requirements keep dropping exponentially then perhaps in a few years you can have today’s frontier performance on the normal laptop you already have on your desk anyway.
HDBaseT 21 hours ago [-]
Many gamers already spend $1000+ on a GPU.
If you can integrate AI accelerators into consumer cards (you can), you can have local AI for "reasonably" cheap. This is Nvidia's long term goal if you listen to what Jensen has to say.
The limitation is entirely on memory right now. Just a few years ago we could of been strapping 80-100GB to cards for under $200 (BoM).
pseudosavant 1 days ago [-]
It could be quite a while before we reach that point though. 5+ years easily.
I've been keenly interested in the ability to run local models, but the hardware is just not there. Consumer RAM speeds and capacity will have to significantly increase before local models will be able to perform as well as even the lowest end GPT-5.6 Luna model.
This is on the backdrop of RAM becoming prohibitively expensive. And without the speed and quantity of RAM, it becomes impossible to generate tokens at interactive speeds, regardless of model. There is a fundamental dependency between calculating all of the active params with the given RAM speed.
Even with a model that has been quantized all the way down to Q4, the DGX/RTX Spark chip with 128GB of RAM can only generate ~18 tokens/sec for a MoE model with only 30B active parameters. There haven't been any broadly useful models below 30B active parameters. And that is for a $5000+ piece of hardware that will be one of the best for running on-device models.
I really want to buy instead of rent my AI, but the economics are truly terrible.
m463 1 days ago [-]
> think efficiency is unlikely to result in lower demand for compute
You can't save yourself rich.
hylaride 1 days ago [-]
It's also hard to predict how much money will be burned going down wrong avenues. The internet was the future, but it took a lot of failed companies to eventually land on a sustainable model that brought us the giants we have today.
Railways were also the future, but that didn't stop a rush to build out (often subsidized) lines that were ultimately uneconomical (either because they were corrupt or the planned settlements never arrived).
If AI is similar, then there's going to be a long slowdown on compute spend until the surplus is worked through. A good historical analogy could be the fiber optic buildouts of the late 1990s. The demand for data never really went down much, but the industry eventually commodified and took down some large companies (Nortel, especially)
galaxyLogic 13 hours ago [-]
I rhink there's a difference with AI because -- it brings true value because you pay for the tokens, you only pay for what you use. That is true value.
Compare to just paying for an internet connection, you have bandwidth but not sure what you can do with it that is valuable.
Let's say you use AI to produce software. There;s no limit as to how high the quality you want your software to have. And how fast you want your project to be complete. There's plenty of room for higher quality, and more performant AI. As AI becomes chepaer people will use more of it, they're not going to say "We have enough AI".
Compare to railroads. Yes you pay for the distance travelled but there's a limit to how much people wwill want to travel, how it will benefit them.
aurareturn 15 hours ago [-]
It is entirely possible that 5 years from now, there's >100x LLM inference going on - but demand for AI chips (including memory) is only 2x or less.
I doubt it. If LLM inference efficiency is 50x better than today, then there could be 1000x increase in inference volume due and we'll end up needing even more chips.
Jevons paradox should win out for a long time for AI.
When internet connections got faster than 56k modems, we didn't use the same amount of bandwidth but faster. We used more bandwidth doing things like 4k streaming. I see the same in AI inference. If AI inference is that much more efficient, it will just enable more use cases for AI.
Even after so many years, internet traffic continues to grow at an increasing rate.
amelius 1 days ago [-]
Nvidia's great superpower is flexibility. You can easily run models of very different types on the same card; and their hardware is great for R&D.
However, at some point AI may be good enough for most people and then it makes sense to make an ASIC for the model (or group of models); and at that point you don't need Nvidia.
I suppose this scenario will happen in various moments at different levels.
whatever1 1 days ago [-]
Each of the hyperscalers has put like 250B each in the last year for infra. That means that they need to be writing AI profits to the tune of 20B per year just to keep up with the cost of the cash they burned.
We are not there. But they better figure it out soon. The cash flows dried up, and everyone is taking debt to support the capex. Google for the first time in its public history is cash flow negative. Amazon too.
maerF0x0 1 days ago [-]
Plus on top of that 1st and 2nd order can be correct, but then the price is too high, meaning people lose money even if correct about the future, but over pay for it.
FuriouslyAdrift 1 days ago [-]
They also have to be feeling the heat of the ASIC vendors. AMD just acquired Taalas and they work with Cerebras all the time on special projects. ASICs outgun nVidia's chips by an order of magnitude.
wmf 1 days ago [-]
Not really. The GPU+LPU combination is going to be pretty good once it comes out.
tolugenius 1 days ago [-]
More interesting take on Nvidia's position than I've come across before. One thing to be noted is 1) Nvidia is already making moves in robotics so even if their position in AI (moreso llms) diminished, they certainly have another big avenue arguably harder to just get into (although I'm not sure what efforts Google is doing for the tpu in robotics). Another point is Nvidia is still the main player in the west, that is, China certainly can and will create their own full stack without reliance on US companies. That puts Europe and other countries in an interesting, do you buy Nvidia because it's the only option or for security. That's to say I believe Nvidia's position relied on many different things being true at the same time, and we're moving towards an environment where those things are certainly being contested at (roughly) the same time.
godshatter 1 days ago [-]
I'm just hoping that some day they can get back into the relatively small market of gaming gpus, even if for just nostalgia sake.
wongarsu 1 days ago [-]
Even in the west, Nvidia's dominance is bound to weaken. There is a notable uptick of articles on HN about people running large models on AMD hardware. And while I don't know official sales figures, I know we have trouble getting our AMD system delivered
AMD's software story is still a lot worse than Nvidia's. But patching up vllm to run one or two models you care about on AMD hardware is a much easier proposition than using them in most other fields of AI.
ekianjo 1 days ago [-]
amd is not putting nearly enough effort to improve their software its almost suspicious
officeplant 1 days ago [-]
They've struggled with software even in the ATi days. It's just normal AMD behavior. Drivers will always be half baked.
moralestapia 1 days ago [-]
nVidia will fail right after reaching an 8 trillion valuation and supplying 80% of the world's hardware!
Trust me guys, it's over!
doctorwho42 1 days ago [-]
I think this is a great example of the disconnect people have in these types of conversations.
You can both become a company that supplies 80% of the world with your type of product, and then still have your stock go down in value.
All it takes is over evaluation by the stock market. Then a course correction from unsustained growth on growth (second order). So even if you continually replace YoY 80% of the world's hardware on a rotating business, but you don't increase market share or increase demand (aka growth)... Your business looks stagnant to the stock market, and there isn't really anything you can do about it. The best you can do is track inflation +/- 2%.
And that's why a lot of older established companies were dividend stocks. You don't expect to growth anymore, but that's not where the value is anymore... The value is in the reliable sales that will happen after infinitum because your company controls a majority share of the business... And that's ok! Unfortunately, silicon valley has created a philosophy of 'you gotta expand into new fields or your on the decline' - aka neo-monopolization
acdha 1 days ago [-]
Nobody is saying they’re outright failing, but that they’re not going to be printing money the way they have been recently. Think about Intel circa 2010: most of their competitors like POWER or MIPS were marginalized, they owned the desktop and server markets with a bit of competition from AMD well contained, and their biggest desktop competitor (Apple) had just switched. A lot of analyst predictions … did not match what happened next. The same was true of Cisco a decade earlier. Both companies are still there but they don’t set the terms in their market segments.
I’m not predicting Nvidia will MBA themselves to death in the near future but I think there’s a tendency to overstate how profitable companies will stay. The more money Nvidia makes, the more motivated their competitors will be to get a piece of that market and the more customers will be looking for alternatives like the push into TPUs which the article discussed.
The current administration is definitely corrupt enough that you could imagine an anti-competitive deal of some sort but I don’t think there’s a way for even that to change matters because key competitors are well-connected American companies willing to play that game, too.
coredog64 1 days ago [-]
> The more money Nvidia makes, the more motivated their competitors will be to get a piece of that market and the more customers will be looking for alternatives like the push into TPUs which the article discussed.
Your margin is my opportunity
- Jeff Bezos
bigyabai 1 days ago [-]
But people have been saying this about CUDA for twenty years, and we are not any closer to a replacement GPGPU paradigm today.
The root comment in this thread was about Nvidia hedging their bet on lost AI market share. They recognize that a reduced pace in training and inference competition will undercut their business, but CUDA isn't a one-trick pony for LLMs alone. TPUs are - you can't even reuse the same architecture for training and inference, they're separate ASICs unlike CUDA cores/ALUs. Veterans of crypto mining will tell you that the ASICs lost in the end, as Nvidia was evolving their hardware faster than the ASIC manufacturers could iterate. When the crypto acceleration landscape diversified away from ETH/BTC into altcoins, Nvidia was still there making money hand-over-fist from mining hardware.
I guess you could argue that robotics, world models or computer vision won't be a trillion-dollar market. But Nvidia is positioned to be the first mover in all of these markets, and none of their competitors are even coming close to the integrated stack that they sell consumers.
acdha 22 hours ago [-]
> But people have been saying this about CUDA for twenty years, and we are not any closer to a replacement GPGPU paradigm today.
How much money was in it for the first decade or so? I think AMD was asleep at the switch but e.g. Apple just did their own thing for the parts which they prioritized.
My understanding is also that Anthropic and OpenAI have also worked to decouple themselves so I think it’s likely that the CUDA moat is going to be less of a barrier than it used to be from the perspective of guaranteeing Nvidia profits.
bigyabai 19 hours ago [-]
Money wasn't really the problem. Apple pulled OpenCL together pro-bono, and worked with Khronos to find willing industry stakeholders that would oppose Nvidia. OpenCL needed hardware standardization though, and nobody wanted to design or implement on a scalable GPGPU architecture like CUDA had. AMD and Apple both bet big on raster efficiency, which turned out to be a terrible play when Nvidia was already putting dedicated ray tracing and tensor hardware into their GPUs. They both bet the farm against each other, and only Nvidia won.
Once Apple fully left Khronos, AMD played the smartest card they had; they architecturally split RDNA and CDNA into separate product lines, so they could optimize them independently. This staunched the bleeding, and gave AMD a datacenter presence that Apple Silicon could only dream of. Still not a scalable architecture, but better than nothing.
coredog64 24 hours ago [-]
> you can't even reuse the same architecture for training and inference
AWS begs to differ. They originally split between `Trainium` and `Inferentia` but now support both with `Trainium`
bigyabai 23 hours ago [-]
I stand corrected, only Google's post-Ironwood TPUs have the split as well.
Nonetheless, TPU architectures are still a systolic array, and have their own limitations for scalability and flexibility. CUDA is no silver bullet, but it satisfies the demands of the edge and research customers very well.
Google is mostly the party behind the whole VLA principle.
rcr-anti 1 days ago [-]
For awhile I've found two things hard to square, that the hardware and software making up current gen AI will bring us to a socioeconomic singularity, and the reality the thing they're mostly trying to emulate is a few pounds of meat and fat running on tens of watts equivalent. On one hand the current AIs are obviously super human in some tasks, get completely dunked on in others by far simpler organisms. My cat can catch a bug out of the air, Fable 5 in Cowork can lack the dexterity to make a slideshow because I had LibreOffice instead of Microsoft Office. Not even close to analogous, but point being they appear to have pretty fundamental differences in how they can interface with the world that the economic thesis seems to gloss over.
atom_arranger 1 days ago [-]
Another interesting discrepancy is that people think current GPUs are maybe capable of running AGI but they still can barely manage photorealistic rendering of a single room in realtime, or simulate something like a shirt thrown into a pile of laundry. They can generate a video of it based on millions of existing videos, but not do a real simulation of light and physics in realtime.
TheBicPen 21 hours ago [-]
An AGI doesn't need that level of detail to do most tasks effectively. To use the OP's example, a cat catching a bug out of the air does not need to run a fluid dynamics simulation of airflow over the bug's wings to be able to catch it. A cheap approximation of the flight path is sufficient. Perhaps some physical tasks will need that level of detail but many will not.
TonyStr 12 hours ago [-]
Is the human brain capable of doing real simulations of light and physics in realtime? Or does it hallucinate the details and represent some low-resolution mental image? You may be overestimating the capabilities of flesh-based neural networks and underestimating the capabilities of silicon-based neural networks.
atom_arranger 5 hours ago [-]
That's a good point. It's just kind of surprising, there's been a lot of incremental progress towards rendering and simulation for a long time. If you'd asked 10 years ago if we'd have AGI or realistic realtime rendering and physics of a single room first I think most people would have answered the former.
Melatonic 22 hours ago [-]
You make a good point. Maybe the AI apocalypse will actually be when someone hooks up a cat brain to a super advanced AI.
rglover 20 hours ago [-]
No need. The 'apocalypse' will come purely out of the hubris and greed chasing untold fortunes as opposed to a healthy, considered integration.
The decision to scare the shit out of normal people early on was a short-term decision that will cost everyone dearly (it led non-technical people to believe in things about LLMs that just aren't true—even worse, they've now made difficult business decisions on bad information with seemingly limited consideration of secondary effects).
The amount of short-sightedness alone should give pause, but at least in America, we seem to have collectively lost our minds in worship of the almighty dollar.
Judgment day approaches.
hammock 16 hours ago [-]
> a few pounds of … running on tens of watts equivalent.
Aren’t you already describing a laptop computer? Add a local LLM and you’re fully there
dzonga 1 days ago [-]
Nvidia has been playing a dangerous but profitable game since the Crypto boom.
but now I think they probably have bitten more than they can chew.
Apple already proved with their unified memory - that as long you have the capacity you can run capable models locally - thereby goes demand for inference if everyone is running some model locally.
For training - Chinese models have proved that you don't need the latest & greatest in Nvidia hardware. Same as TPUs.
only time will tell.
hermitShell 1 days ago [-]
It seems they had a head start but are now facing stiff competition on all fronts. Software moat, GPU's for gaming, and its distant cousin datacenter compute. They rightfully invested their insane profits into many ventures, and how many of those have turned around into profit?
They are also a robotics AI company with Omniverse. They are also an AI company with Nemotron. They are also a bleeding edge network equipment company after the Mellanox aquisition.
They stand to make a lot of money if they succeed in every venture. Good for Jensen taking risks and driving innovation, I hope they succeed in chewing even 50% of what they bit off.
mrandish 21 hours ago [-]
> They rightfully invested their insane profits into many ventures
This is the inherent downside of such a rapid rise to being the world's most valuable company AND still being considered a growth stock. At their massive scale, the number of new adjacent businesses that have both sufficient size and potential growth is limited.
> I hope they succeed in chewing even 50% of what they bit off.
Anything approaching that is vanishingly unlikely. They're being forced to play the game more like a VC. The question is if a few unicorn winners can offset dozens of losers. The challenge is that, unlike a VC, their bets are much more correlated around AI.
pletnes 1 days ago [-]
Nvidia sell iot boards with unified architecture. Would not be shocked if they launch pc/laptop/server boards at some point.
buildbot 1 days ago [-]
Unified memory DGX Spark and RTX Spark laptops are already a thing :)
leoc 1 days ago [-]
Right, it's clear that nVidia is taking care and trying to position itself so that it can continue making sales if and when inference goes local. And it's in a much better intrinsic position to do that than the LLM SaaS vendors are: nVidia sells shovels to the army, but it also knows how to sell shovels to Walmart. Whether the financial relationships that Huang's got his company into will cause it problems if the market shifts is a different question, though.
ABS 1 days ago [-]
AFAIK NVIDIA unified memory is not as... "unified" as Apple's
FuriouslyAdrift 1 days ago [-]
As is AMD Strix Point
wmf 1 days ago [-]
RTX Spark already launched.
synergy20 1 days ago [-]
that undercuts their core business, so it will be a defensive play at most to fend off mac and amd's local inference offerings
bigyabai 1 days ago [-]
I don't know how people can say this with a straight face. Nvidia was selling desktop-grade ARM SOCs before Apple Silicon was ever announced, specifically for edge robotics, computer vision and ML.
The absolute fastest desktop Mac GPUs cannot beat an Nvidia laptop GPU in prefill or inference speeds. Apple Silicon is a non-entity for professional datacenter deployment and arguably unusable for frontier models at agentic context sizes. AMD is Nvidia's primary worry, and they're not doing much better in terms of GPGPU SOC compute.
officeplant 1 days ago [-]
>Nvidia was selling desktop-grade ARM SOCs before Apple Silicon was ever announced
You can believe all you want that the dinky little jetson boards were desktop grade when historically the ARM SoC portion of a jetson board couldn't even keep up with broadcom/rockchip SoCs. It's taken until recently for the actual arm compute portion of Nvidia SoC's to be worth a damn at all, and they still fall far behind Apple let alone the rest of the pack like Qualcomm/Samsung.
bigyabai 1 days ago [-]
I don't have to believe. I've run KDE and GNOME on the Tegra boards, you get full-fat CUDA support without sacrificing Vulkan drivers. It's incredible.
You can believe all you want that good single-core performance will corner the edge compute market. It hasn't, Graviton has more buy-in than any Apple Silicon chip ever got.
HDThoreaun 23 hours ago [-]
Every time nvidia takes its fab time and uses it to build anything other than datacenter chips it is losing money due to the massive markups the datacenter products have. Expanding their consumer offering means the datacenter backlog is going down which is very bad for their margins. Consumers will never pay 10-100x what it costs to fab something like datacenter users will.
bigyabai 22 hours ago [-]
In many cases it isn't Nvidia paying for the fab time. For instance, the Nintendo Switch 2 is basically a pure-play design and support product for Nvidia, while Nintendo negotiates with Samsung for the actual SOC prices. Their IP philosophy is closer to AMD's than Apple's, Nvidia has long roped in 3rd party manufacturers to mark up, integrate and sell their hardware.
ragall 20 hours ago [-]
I'd be happy enough if Nvidia decided to launch an update to the now dated (2019) Shield streaming box, which is still the best on the market as to codec support and image quality.
wslh 1 days ago [-]
Yes. The more capable SoC PCs become, the weaker the "everything has to run in the datacenter" argument gets. As more powerful SoC PCs from multiple vendors appear over the next few years, that advantage may shrink for many workloads.
2OEH8eoCRo0 1 days ago [-]
What's the danger? They slide back down to being just a gaming graphics card company with a $10 share price?
georgemcbay 1 days ago [-]
> What's the danger? They slide back down to being just a gaming graphics card company with a $10 share price?
Nvidia dropping from being a $5 trillion company to a $242 billion company would be 1929 levels of bad.
Global economy end of days stuff, especially since Nvidia can't crash that hard without a lot of other stuff crashing with it.
Ekaros 1 days ago [-]
How much in value could Nvidia safely drop and over what period of time for it be fine for greater economy? What level of correction would be manageable?
And I am pretty sure that their value will drop in 5 to 10 years.
grey-area 1 days ago [-]
Just for context, they make up 5% (yes 5%) of holdings in global index funds like VWRA.
That is an astounding figure and means markets are highly imbalanced, unless you think Nvidia represents 5% of the global economy.
twoodfin 1 days ago [-]
4.7% of the discounted future earnings pie of 3761 of the largest current publicly traded global companies.
That’s still a lot, but “the global economy” over the horizon where those earnings are not discounted to 0 contains a lot more than what’s listed in VWRA today.
HDBaseT 21 hours ago [-]
There is a number of index funds holding 7.5% of NVIDIA.
This consolidation is scary. I've started putting a little bit in emerging markets and China. I cannot trust the US long term, hopefully I'm wrong!
grey-area 12 hours ago [-]
I see China as very risky due to Taiwan.
2OEH8eoCRo0 5 hours ago [-]
I see Nvidia as very risky due to their chips all being fabbed in Taiwan and Taiwan is risky due to China.
KaseyKim 1 days ago [-]
Nvida's action may affect the semi- marketing
22 hours ago [-]
alexpotato 4 hours ago [-]
Back in during the dotcom boom, there were multiple examples of companies being bought for ridiculous amounts. Often as the result of a bidding war.
Even back then, some economists used the "hidden wallet auction" as an example of how this could happen.
To summarize:
- there is a wallet
- you don't know how much is in the wallet
- you bid on amount to buy the wallet
- if you get the highest bid you win
- crucially, if you lose then you still have to pay
This is often cited as a game that you do not want to play b/c it's a. hard to predict the upside, b. the downside is huge.
That being said, people still got into these auctions and because of sunk cost fallacy, decided to keep bidding even if they might lose.
The hyperscaler race feels a bit like the above but no one seems to ant to admit it.
KaiMagnus 1 days ago [-]
IMO focusing on the hyperscalers is kind of misleading.
Yes, for programmers and tech companies AI is kinda boring now, but AI integration in general is still kind of uncharted territory.
There are so many small companies and individuals just getting started with AI today and I believe a large the customer base (and revenue) is still untapped. Hell, I’m discovering new use cases regularly still and the average mismanaged 30 people whatever SaaS vendor probably didn’t even get started yet.
yaportmax 1 days ago [-]
This is what so many people on HN and the market are constantly missing.
Jason Kottke almost didn't found his blog in 1998, famously quoted as saying: "I thought I was too late, that no one would be interested." Needless to say, the internet was a tiny joke in 1998 compared to what it is now.
We are just barely scratching the surface of what's possible with AI, both in terms of the leading edge and in the 'torso' of the economy (the portion you're describing).
Folks from Silicon Valley working in AI-forward companies have a skewed perception of how many people have adopted this technology so far. Codex recently celebrated hitting 10 million users. This is a great milestone and all, but to put it in context, Microsoft office has a billion users. Sure, many people use Claude Code and or some other harness and the growth is staggering, but the overall scale is tiny compared to software as a whole. Costs of serving and usage are still very high, prohibitively so for many, so we aren't even close to market saturation.
And even at the leading edge, people who do work in those AI-forward companies; models are still slow, require hand holding, and produce suboptimal outcomes sometimes. Imagine the value when instead of needing to prompt it once per 30 mins, you prompt it once per day. Then once per week. Then once per month. Imagine all this running not on 3 trillion parameter models, not on 10 trillion, but 100 trillion. What kind of computer infra will be needed then? Certainly more than we have today.
RyanOD 1 days ago [-]
I've always marveled at how one can pick any year since the internet went mainstream and in that year people thought, "Oh my goodness, the internet is amazing!"
Then, move five years forward from that year and look back. In every case, people think, "Hah! The internet was so simple then!"
In 2031, I suspect we'll say the same about 2026.
w10-1 23 hours ago [-]
Yes and no...
So much of the dotcom bust was essentially: "anything internet will work", but even after it was pretty easy to find different ways to publish web pages or migrate to new forms of social media, video, short-video, etc.
It's orders of magnitude harder to automate work, which is the business value proposition of AI (whether you are displacing worker or adding breakthrough capacity), and it's not clear the LLM hyperscalers will get the value-add from that.
On the consumer side, it might be a race to the bottom: same ad profits, but now you need AI to produce it.
thelastgallon 18 hours ago [-]
I wonder why Google doesn't create an open-source CUDA alternative. Google released Kubernetes to stay relevant/competitive in the cloud wars, they were a distant third. They now have an opportunity to create an open source industry standard.
Or the companies spending trillions of dollars can do a Manhattan Project (Or X-Prize) and let a thousand startups work on it. One will succeed. Between Google, Amazon, FB, Microsoft, Apple, AMD, Qualcomm, Intel (and dozens of other companies) there is enough economic incentive to do it. Also, isn't this what AI is supposed to be extremely good at, CUDA experts can continue to write CUDA (without having to learn anything new), a translation layer will rewrite it. If software can be one-shot from markdown files, this can't be impossible.
Erikun 1 days ago [-]
I see we have reached the stock market phase of Universal Paperclips.
doctorwho42 1 days ago [-]
Honestly, that game with a few changes would be very on the nose today :D
Theodores 1 days ago [-]
Phew. I thought we were just about to leave it, albeit not quite getting to space, just dissolving the financial system so our AI overlords can make some more harvester drones.
Dardalus 1 days ago [-]
Tend to agree with Ben's thesis RE Demis and DeepMind not really being focused on the agentic coding race. That being said, it remains to be seen whether Sergey and Koray can inspire the foot soldiers in the same way that Sama and Dario do. I'm not too optimistic, and that's to say nothing of the fact that Google cannot possibly hope to compete with these other companies on potential employee upside.
mrandish 21 hours ago [-]
> Google cannot possibly hope to compete with these other companies...
It's possible Google has intentionally decided to take a more conservative blended approach than purely competing at the bleeding edge of the frontier. If so, they obviously have no incentive to state it publicly but the recent departures and financials are consistent with the idea. It also makes sense that a company so much bigger, longer-term and (somewhat) more diversified than pure-play frontier labs would play the game to align with their strengths (capital, balance sheet, breadth, etc).
In their position, why not take an 'arms supplier' strategy in the near-term while drafting behind the frontier labs as a fast follower in AI, essentially betting the AI race is more akin to the Indy 500 than a quarter-mile drag race. If they're wrong and it IS more like a drag race, Google is in a better position to absorb and adjust than a frontier lab, for whom current valuations and capex spending requirements nearly require this to be a relatively short, winner-take-all race.
HDThoreaun 23 hours ago [-]
I think the linked semi-analysis piece is about right. Google is more focused on protecting its 4.2 trillion dollar golden goose than pushing cutting edge which leads to a bureaucratic nightmare that researchers simply dont need to engage with when openAI/anthropic are offering even more money. Why waste time dealing with bureaucracy at google when you can be top dog at openAI or anthropic?
gizajob 1 days ago [-]
Laughable to cite and reiterate the idea that Google is cooked where it comes to SoTA and AI in general when they operate, reliably and successfully for decades, one of the largest computing infrastructures on Earth and will likely continue to usefully serve the 90% of AI requests that don’t involve managing large codebases. Also seems strange to suggest that Google would need to Aquihire a company like Thinking Machines when it could spin up their AI model in a couple of weeks on its own TPUs if it felt like it. Demis likely wants to focus on his specific interest at the junction of biochemistry, neurobiology and computation which is more specific and unique to Demis than building a general purpose Q&A search model.
epolanski 2 hours ago [-]
I actually agree with the article stating that not aiming for SOTA is actually a benefit for Google.
Gemini is already good for enterprise users, most companies I know are using Gemini 3.1 to interact with their email and sheets and creating presentations on the fly.
clarkmoody 1 days ago [-]
> To translate such figures into comparable 2026 magnitudes, multiply by a factor of 1,200.
Perhaps this has something to do with the economic dislocations and world wars between the 1870s and today?
znnajdla 1 days ago [-]
There's another factor which Ben failed to consider. Which is that NVIDIA doesn't need to rely on demand for their proprietary CUDA stack or their GPUs growing -- they are already selling directly to the consumer, and likely capturing much higher margins. They are moving up stack, not down, where demand for raw compute matters less. With the DGX Spark and Jensen’s statement about “open models”, their next product is likely a strong hint: consumer devices to fulfill the Mac Mini demand craze. They are probably going to start burning LLMs durectly onto sillicon and then selling DeepSeek-in-your-home to individual developers. I bet that would sell even better than Anthropic Max coding plans and is not dependent on hyperscaler funded boom-bust cycles. So Ben’s analysis highlights the risk of their existing business not growing but they are likely planning new businesses.
cmiles8 1 days ago [-]
Nothing goes up and to the right forever. Nothing.
Building a business model on the belief that “this time is different” always finds storms on the horizon.
CodesInChaos 1 days ago [-]
A lot of things go up forever, as long as you denominate them in an inflationary currency ;-)
cmiles8 1 days ago [-]
There’s a massive difference between “going up forever” and “point B is higher than point A.”
The current setup can’t sustain a downturn, even if yes 20 years from now point B is likely to be higher than present.
That’s the danger. Those that are going to get wiped out by the AI bubble burst aren’t wrong about AI being huge long term, they just put themselves in a position to not survive the storms that happen between points A and B.
rglover 20 hours ago [-]
This is the best articulation of this problem/paradox I've read yet.
pelotron 1 days ago [-]
What if we build our whole economy on that belief?
rglover 1 days ago [-]
I guess we'll find out soon enough.
odiroot 1 days ago [-]
Or at least our public pension systems.
barbecue_sauce 22 hours ago [-]
Well, they certainly can't go left.
davedx 23 hours ago [-]
People and pundits have been dooming and bearing on Nvidia for as long as it's been around. It increased in intensity when gpus were used for large scale crypto mining and it became material to their operations, and continued as AI ("the bubble") started to really take off.
Over those years, my NVDA stock has been by far my biggest winner. I'm now up more than 1500% on it.
Let the dooming continue
petesergeant 1 days ago [-]
> The subsequent bankruptcy of Jay Cooke & Company triggered the Panic of 1873, culminating in endless railroad bankruptcies across the country, a multi-year depression, multi-decade deflation, and, one could argue, the financial conditions that made Europe, four decades later, into a tinder box.
American history education needs some dire reform.
petesergeant 1 days ago [-]
> After the departure of DeepMind CEO Demis Hassabis (technically promoted to chairman, but no longer in charge of day-to-day operations) and Gemini co-lead and former Chief Scientist Jeff Dean, along with a host of other prominent researchers, SemiAnalysis declared that Gemini is Cooked: "For all intents and purposes, we believe DeepMind is no longer a frontier lab"
Counterpoint: xAI pooped out a frontier model based on nothing but capital and one man's desire to push a right-wing political narrative. Google has the talent, and the money, and the experience, they just need some leadership.
gigatexal 21 hours ago [-]
This is fine. everything is on fire it’s not a bubble. ;-)
echelon_musk 1 days ago [-]
Is this just an ad for a new book about trains?
Disappointed by the lack of Tom Cruise.
RustaIsBest 1 days ago [-]
[flagged]
cmpxchg8b 1 days ago [-]
That has literally nothing to do with the article.
I did not mean to be snarky. But still, the proposition that a railroad bankruptcy “led to world war” (direct quote from the article) 40 years later is ridiculous.
tomhow 14 hours ago [-]
We just don’t want denunciatory rhetoric like “is where I stopped reading” or “this guy should re-read what he puts out” or “ridiculous”. If someone is wrong just point out where they’re wrong. Educate us, don’t fulminate.
zaphar 1 days ago [-]
He was referencing a book that made that case if you "squint". If you read that as actually a serious "this caused world war 1" statement rather than. This looks to have gotten some dominoes rolling that may have contributed to WW1 then that says more about you than it does about the article itself.
dh2022 1 days ago [-]
Re: "This looks to have gotten some dominoes rolling that may have contributed to WW1 " - people who believe this should read some history about beginning of WW1. I recommend starting with "The Guns of August".
simonw 1 days ago [-]
> Blaming some railroad bankruptcy for starting WW1 is where I stopped reading.
What a weird reason to stop reading an article.
dh2022 1 days ago [-]
I am assuming the rest of article is as well researched as the, rather long and incredibly incorrect, introduction.
Bayesian inference and all....
Altaba 1 days ago [-]
Ben is wrong; demand for compute, aka revenue backlogs, is mythical and will collapse, simply because of two reasons :
1. Circular investment/spending.
2. Too much capital in the system, so returns cannot be hit regardless because the barrier is too high. (Evidence being every capital cycle in history)
motoxpro 1 days ago [-]
I think more interesting take here would be WHEN this will happen. I don't think Ben, or anyone else, thinks we wont have some sort of correction or stabilization in supply/demand (he has said as much) But when will that occur? 2 months? 2 years? 20 years?
Altaba 24 hours ago [-]
well its typically when it becomes clear that the private equity firms taking the risk decide they can not get the returns they need, forcing the backstoppers such as Nvidia to take that burden, and the whole ecosystem collapses.
motoxpro 20 hours ago [-]
I guess my point is that until that collapse, returns are very very very good. This can (and probably will) go on for a decent amount of time more, regardless of the inevitable things you point out.
My guess is at least another 2 years, as most people don't use AI yet, or maybe more precisely, AI is not used in the underlying workflows (which are invisible to the consumer) that make up most people's jobs.
Who knows if I am right. My original post was just pointing out that what you say is about timing, not whether it's true or not, because of course its true.
Altaba 18 hours ago [-]
Yes, agreed, it is timing. I think a sign we are getting closer is the new equity issuances, which kind of are leveraging the current environment and the retail excitement.
u1hcw9nx 1 days ago [-]
Ever free newsletter and talking head spouts narratives like this free. If you want something that quantifies and gives actionable information, you must do it yourself or pay for it. What are your below $2000/month sources for good analysis?
BigTTYGothGF 1 days ago [-]
If you have to ask you're not going to benefit from it.
1 days ago [-]
tguedes 1 days ago [-]
The website from this article. It's not free. Ben Thompson releases 1 free article a week but the other 3 articles published each week requires a $15/month subscription. Ben Thompson is also very influential in Silicon Valley and the overall tech/media industry.
u1hcw9nx 1 days ago [-]
I know him, I subscribed for a while. Even his paid content is lacking. I'm looking more Valens Research kind of analysis. SemiAnalysis is also good in the higher tier.
ps. Being influential in Silicon Valley just means you are influential, it does not mean substantial. Leopold is still influential and gets money thrown at him at $100s of million despite having no substance.
kaonwarb 1 days ago [-]
Criticism with no justification behind it is cheap.
ey5e5uer5ur 1 days ago [-]
irony
1 days ago [-]
claytonjy 1 days ago [-]
you might be looking for SemiAnalysis? I only read the free portions of articles but they have various paid options, mostly targeting investors with information and tools.
To me they come across as overzealous peddlers and hacks on some market pumping mission.
bwfan123 1 days ago [-]
> overzealous peddlers and hacks
They are the marketing wing of the AI ecosystem. Their recent article on how SpaceX would drive 500B in data-center revenue was ludicrous-mode. Lets revisit this in a few years.
dnnehgf 1 days ago [-]
so short them. if you think that the demand for skilled-labor-substitutive capital is saturable in the medium term or that improvements at the model level eat those at the hardware/cuda level or that nvidia just has the timing wrong, short them.
Google's limitation is that they still don't offer TPUs in a PCI-E card/dev board that people can plug in to their PC for local development and sane low level API to develop against, instead you have to go through their cloud and their full software stack which greatly limits ecosystem growth. The minute that Google figures that out, that's when Nvidia's dominance would be challenged.
I do agree that it’s really not great, and I also have never been a strong believer in the CUDA moat overall; as the need for GPUs moves from research to production (inference), companies are plenty willing to build software from scratch anyway (and we see this with AMD GPUs being in plenty high demand in the datacenter and enthusiast market now).
AI is supposed have solved the "coding problem". But shouldn't translating a program from one platform to another be an even easier, more mechanical, task for the AI?
This is actually a corollary to the point I was making about "CUDA" usually also including a ton of the included kernels and not just referring to a crappy programming environment; translating mid-level C that does math between two runtimes wouldn't be hard for an LLM, but translating "doBigDNNThingNVidiaGaveMeInAKernel()" to "doBigDNNThingByHandBecauseAMDDoesntSupportIt()" isn't a rote translation at all.
Of course, once you accept that it's _not_ "why don't you just translate it," you _can_ iteratively use an LLM to implement the ThingNVidiaGaveYouInAKernel, but it probably isn't well-trained, yet, on low-level AMD optimization tricks, so the kernel you end up with will likely be slower than the CUDA one.
I wonder if this points to a deeper limitation of AI, it can not do coding tasks it has not seen in its training material. Or could it possibly "generalize" to accompllish something like this anyway?
Which AI? LLMs are coding facilitators and code producers.
A problem is solved when the solution is reliable. Non-deterministic Neural Networks are not reliable. In fact,
> more mechanical[] task
that suggests an expectation of process and procedure, which is still not a capability of current architectures.
Sure, you can ask a brains-deficient operator to perform a huge task, but then you'll have to check the whole product, and that remains not cheap.
"AMD and Anthropic also formed a multiyear engineering partnership to optimize ROCm using Claude"
FROM: https://finance.yahoo.com/markets/stocks/articles/ex-amd-exe...
AMD has had years to try and counter it, but just has not. Google is kinda trying to do an end run around it with TPUs but they are still niche high end stuff with limited availability.
Its really just CUDA, and CUDA can be seen as somewhat akin to C for assembly used by Nvidia's gpus- In many ways a wrapper around the low level hardware that often has those details bleed through.
Again, "CUDA" isn't a programming language, it stands for Compute Unified Device Architecture; "C/C++ for CUDA" are the high-level languages that compiles to PTX and then SASS as well CPU orchestration code via NVCC.
And to be honest, pretty much everything you can do in CUDA C/C++, you can also do in HLSL/GLSL compiled to SPIR-V, as long as the Vulkan hardware extension is available.
Maybe I wasn't being precise enough with my language for this forum, and also my last hands on experience with it was roughly 6 years ago, maybe it's gotten better. But it was much less (and forgive the imprecision!) python/pytorch-like where you say hey take this big blob of data and just slice and dice it on your many cores, and more like ok, here is the data, let's cudamemcopy it in these size chunks over to the gpu itself, to be used by this block of threads and run these commands (kernel in cudaspeak) on it. Much more painstaking and micromanagey of the resources.
Pytorch IMHO feels like a proper abstracted API that hides the details and lets you just unleash the fury at the cost of some efficiency, while the cuda api itself, similar to working with C, forces you to really think about the low level details. I have a heavy backend and systems development background, and while it wasn't really intimidating to me, it was like wow you really have to have a deep working knowledge of how these things work and it felt like a step back in time IMHO.
I doubt that's going to satisfy you but I think it gives a clearer picture of what using cuda is like if you typically use higher level languages and haven't touched C since college.
It's pretty heretical for me to say this, but a lot of GPU compute complexity that Nvidia is doing in CUDA is unnecessary and is by the simple fact that to do anything meaningful you have to either use their library or handle allocation/scheduling yourself. Imagine if JavaScript required you to handroll part of the V8/Node's JIT compiler, allocator and scheduler yourself every time you just want to make a webpage, that is essentially what CUDA is doing.
The actual "program" that runs on the GPU, the compute shaders in PTX/SPIR-V, are very low level but pretty straight forward once you get down to it.
“CUDA is the worst development ecosystem in existence. Except for all the others.”
Looking at software more specifically the Linux foundation reported based on software dev salaries in 2008 it would be 1.4 billion to only write the Linux kernel.
Up until about 2023 there wasn’t enough money involved to have any reason to make a real CUDA killer even if you could get it adopted.
I was under the impression that AI was supposed to remove software-moats, let us all ask it to write our custom MS Word for us for instance?
And the people they were selling that to, ( It's free and open souce now! ), were a very different group to the market they left behind on .NET Framework, who are often still struggling to make the transition now.
Had they actually killed off .NET Framework, it would have been a different story, much more like the VB6/VBA to VB.NET transition, which so few people bothered with that VB.NET died out, because if you had to retrain that much, you figured you might as well go to C# or a instead, or indeed a completely different language entirely.
I briefly worked professionally on a VB.NET project, but outside that job I've never met anyone else who can say the same. I've met a few who went straight from VB6 to C# though.
Largely the same market (Enterprise) but not the different segment (web as opposed to Windows/WinForms).
I ported about 15 years of projects from various versions of .Net to .Net Core whilst they were developing (and sent feedback to the team - they were asking us to do that) and the process was pretty reasonable. You were only really stuck if you were using something very very Windows specific (certain image processing libraries iirc) and even then it was largely manageable.
The old full-fat framework is, AFAIK, still supported, as there's a whole lot of legacy code which is Windows specific which is still expensive / hard to port over.
Between MSMQ, WCF over named pipes, MSDTC, and MSI installers, there's a lot to replace that is hard to provide the same guarantees or performance with straight replacements, if they even exist.
The end goal, being on modern dotnet, is better, but it's difficult to get there with a phased approach without accepting a temporary worsening, which is often hard to sell.
Especially while Framework is still supported.
Replacing CUDA with another framework has much lower motivation. That advantages of the new framework must cover the switching costs and the risk of such a switch. All while CUDA continues to evolve and allow access to additional features.
Apple and Microsoft had something of a captive userbase. New vendor on the block trying to replace CUDA does not.
[1] https://developer.nvidia.com/nccl [2] https://pytorch.org/blog/torchcomms/ [3] https://rocm.docs.amd.com/projects/rccl/en/latest/
The basic problem is that CUDA has become something of a Schelling point. If you want to train a model right now, the highest performance you can get is almost certainly on CUDA. From the basic general matrix multiply operation, to specific NN architectures, CUDA is going to have incredibly optimized implementations out of the box. And it's going to make multi-GPU training so much easier. And all the dependencies you build on (those layers you import from PyTorch or Transformers or whatever) are going to work optimally right away on CUDA. And that weird random repo that you found with a unique optimizer--it runs on CUDA too. And now the cool new implementation that you're about to release is also going to be built for CUDA.
It's so tempting to think "Just write replacement software", but you also need to transition the entire ecosystem in large part to match CUDA's effectiveness, and you need to get comparable performance out of your chip/library combo as NVIDIA can get out of its cards with CUDA.
There's a whole story here to how effective NVIDIA has been at navigating this. Very early on, they heavily prioritized PyTorch and TensorFlow, getting involved in the projects as much as they could and making sure they always ran best on CUDA. But the TLDR is that yes, you're right, another company could write a CUDA competitor. But actually replacing CUDA is a much larger task.
I'm personally hopeful that with the rise of coding agents, we see more movement on this front with other projects moving into view. It will take some time for any ecosystem to start to emerge that can dislodge CUDA for researchers who don't want to dive that deep into the stack, but hopefully we start to see some momentum build.
Ironically, there was an open source project that was making great progress on CUDA compatibility on AMD hardware. AMD hired the lead developer, and then he shut down the project.
It doesn’t really make sense for AMD themselves or most use cases, though; any compatibility shim just adds problems on top of problems, and for AMD, entrenching a competitors technology even more never really seemed like a great idea.
Management likes it because it removes software developers from the loop.
I’m not familiar with this field, but to my brain, https://www.amazon.com/s?k=Google+Coral seem to show me several such options.
They don't know what good developer experience is, how do you expect them to deliver it to other people?
I think a simple reason why it’s been hard to unseat in Nvidia is first mover advantage. A lot more water has flown through Nvidia pipes than TPUs or AMDs chips for that matter.
TPUs and AMD chips aren’t priced cheaper than NVIDIA (at least for my purposes training models). So there hasn’t been an impetus for me to venture there and use those chips.
Anecdotally, folks I know who have tried using TPUs and AMD chips have hit more issues with the underlying drivers than with NVIDIA chips. That costs time and money to fix.
Eventually the other chips will go through enough iterations and stability will be reached
Nvidia's sells hardware yet their market cap is about the same as Google's.
How much value could Google get by selling hardware too? Google'd be selling to competitors, so difficult to capture much of the value and would decrease Google's value as an AI company. Maybe a child company?
What do you gain by not using CUDA vs what do you risk?
1) We don't know how long that trend will continue, but you do know where to look for when it may end (if smaller sized models continue to compress the knowledge effectively of larger models).
2) We don't know when the appetite for higher cost models might go down and by how much if smaller models get "good enough" and price becomes far more important.
It is entirely possible that 5 years from now, there's >100x LLM inference going on - but demand for AI chips (including memory) is only 2x or less.
It is also entirely possible that at some size - LLMs pick up some emergent capability that doesn't scale well to smaller sizes - and that there's an incredible boost to demand to get that capability.
It's just very hard to predict.
The harder thing to forecast for me is if we hit a wall on increasing efficiency, either on the model weights side or silicon side, with current approaches. If we have to switch to something like burning the model weights into silicon to continue to make gains, then the current math on general purpose accelerators might be upside down.
> If we have to switch to something like burning the model weights into silicon to continue to make gains
I think that's already being considered semi-seriously [0][1]
[0] https://taalas.com/products/
[1] https://ir.amd.com/news-events/press-releases/detail/1296/am...
I buy that. Jevon's Paradox, sure.
> and we are not going to run out of economically useful things to do with it anytime soon on the demand side
This I don't buy. Not fully, at least. Whether or not there's demand for LLMs in some particular field is one thing, whether or not there is a sustainable business model to be built out of that demand is another thing entirely.
There is a staggering amount of money pouring into startups looking for novel use cases for LLM-based agents. As usual, 99% of them will fail, but those other 1% are going to have to look harder and harder to find a novel use case that can actually be served profitably.
First of all, there's only so many places where a chatbot is going to sell. But, that also seems to be the only interface anyone can come up with that allows a user to steer an agent through a long-running task reliably. I'd love to be proven wrong here.
Also, if current trends plateau and large datacenters are still needed for complex tasks, that would stimy growth of LLM usage across entire industries.
But, if present trends continue, then local inference will become feasible for most tasks. That would lower the barrier to entry across tons of heavily-regulated and/or cost-sensitive industries. But, widespread local inference will almost certainly come with a painful market correction centered around hyperscalers, which would itself dry up the pool for ventures into new markets.
Now for every human replacement, that is 1 unit less of communication and bureaucratic burden (HR, middle management etc) that the org requires.
I guess I don't know what to say except that my experience is the polar opposite of yours.
I moved to a new state at the beginning of the year. Needed a new doctor, needed to schedule apartment tours, needed to talk to my employer about insurance and relocation stuff, etc etc. Lots of chatbots, a handful of humans. Humans consistently did what I needed them to do, the chatbots just didn't. I could list examples but I'd be typing all night.
And yknow what, my one call with Comcast to get my internet set up was downright pleasant. The rep was knowledgeable and a good conversationalist.
If you can integrate AI accelerators into consumer cards (you can), you can have local AI for "reasonably" cheap. This is Nvidia's long term goal if you listen to what Jensen has to say.
The limitation is entirely on memory right now. Just a few years ago we could of been strapping 80-100GB to cards for under $200 (BoM).
I've been keenly interested in the ability to run local models, but the hardware is just not there. Consumer RAM speeds and capacity will have to significantly increase before local models will be able to perform as well as even the lowest end GPT-5.6 Luna model.
This is on the backdrop of RAM becoming prohibitively expensive. And without the speed and quantity of RAM, it becomes impossible to generate tokens at interactive speeds, regardless of model. There is a fundamental dependency between calculating all of the active params with the given RAM speed.
Even with a model that has been quantized all the way down to Q4, the DGX/RTX Spark chip with 128GB of RAM can only generate ~18 tokens/sec for a MoE model with only 30B active parameters. There haven't been any broadly useful models below 30B active parameters. And that is for a $5000+ piece of hardware that will be one of the best for running on-device models.
I really want to buy instead of rent my AI, but the economics are truly terrible.
You can't save yourself rich.
Railways were also the future, but that didn't stop a rush to build out (often subsidized) lines that were ultimately uneconomical (either because they were corrupt or the planned settlements never arrived).
If AI is similar, then there's going to be a long slowdown on compute spend until the surplus is worked through. A good historical analogy could be the fiber optic buildouts of the late 1990s. The demand for data never really went down much, but the industry eventually commodified and took down some large companies (Nortel, especially)
Compare to just paying for an internet connection, you have bandwidth but not sure what you can do with it that is valuable.
Let's say you use AI to produce software. There;s no limit as to how high the quality you want your software to have. And how fast you want your project to be complete. There's plenty of room for higher quality, and more performant AI. As AI becomes chepaer people will use more of it, they're not going to say "We have enough AI".
Compare to railroads. Yes you pay for the distance travelled but there's a limit to how much people wwill want to travel, how it will benefit them.
Jevons paradox should win out for a long time for AI.
When internet connections got faster than 56k modems, we didn't use the same amount of bandwidth but faster. We used more bandwidth doing things like 4k streaming. I see the same in AI inference. If AI inference is that much more efficient, it will just enable more use cases for AI.
See for example, internet traffic over time: https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcS779VS...
Even after so many years, internet traffic continues to grow at an increasing rate.
However, at some point AI may be good enough for most people and then it makes sense to make an ASIC for the model (or group of models); and at that point you don't need Nvidia.
I suppose this scenario will happen in various moments at different levels.
We are not there. But they better figure it out soon. The cash flows dried up, and everyone is taking debt to support the capex. Google for the first time in its public history is cash flow negative. Amazon too.
AMD's software story is still a lot worse than Nvidia's. But patching up vllm to run one or two models you care about on AMD hardware is a much easier proposition than using them in most other fields of AI.
Trust me guys, it's over!
You can both become a company that supplies 80% of the world with your type of product, and then still have your stock go down in value.
All it takes is over evaluation by the stock market. Then a course correction from unsustained growth on growth (second order). So even if you continually replace YoY 80% of the world's hardware on a rotating business, but you don't increase market share or increase demand (aka growth)... Your business looks stagnant to the stock market, and there isn't really anything you can do about it. The best you can do is track inflation +/- 2%.
And that's why a lot of older established companies were dividend stocks. You don't expect to growth anymore, but that's not where the value is anymore... The value is in the reliable sales that will happen after infinitum because your company controls a majority share of the business... And that's ok! Unfortunately, silicon valley has created a philosophy of 'you gotta expand into new fields or your on the decline' - aka neo-monopolization
I’m not predicting Nvidia will MBA themselves to death in the near future but I think there’s a tendency to overstate how profitable companies will stay. The more money Nvidia makes, the more motivated their competitors will be to get a piece of that market and the more customers will be looking for alternatives like the push into TPUs which the article discussed.
The current administration is definitely corrupt enough that you could imagine an anti-competitive deal of some sort but I don’t think there’s a way for even that to change matters because key competitors are well-connected American companies willing to play that game, too.
Your margin is my opportunity - Jeff Bezos
The root comment in this thread was about Nvidia hedging their bet on lost AI market share. They recognize that a reduced pace in training and inference competition will undercut their business, but CUDA isn't a one-trick pony for LLMs alone. TPUs are - you can't even reuse the same architecture for training and inference, they're separate ASICs unlike CUDA cores/ALUs. Veterans of crypto mining will tell you that the ASICs lost in the end, as Nvidia was evolving their hardware faster than the ASIC manufacturers could iterate. When the crypto acceleration landscape diversified away from ETH/BTC into altcoins, Nvidia was still there making money hand-over-fist from mining hardware.
I guess you could argue that robotics, world models or computer vision won't be a trillion-dollar market. But Nvidia is positioned to be the first mover in all of these markets, and none of their competitors are even coming close to the integrated stack that they sell consumers.
How much money was in it for the first decade or so? I think AMD was asleep at the switch but e.g. Apple just did their own thing for the parts which they prioritized.
My understanding is also that Anthropic and OpenAI have also worked to decouple themselves so I think it’s likely that the CUDA moat is going to be less of a barrier than it used to be from the perspective of guaranteeing Nvidia profits.
Once Apple fully left Khronos, AMD played the smartest card they had; they architecturally split RDNA and CDNA into separate product lines, so they could optimize them independently. This staunched the bleeding, and gave AMD a datacenter presence that Apple Silicon could only dream of. Still not a scalable architecture, but better than nothing.
AWS begs to differ. They originally split between `Trainium` and `Inferentia` but now support both with `Trainium`
Nonetheless, TPU architectures are still a systolic array, and have their own limitations for scalability and flexibility. CUDA is no silver bullet, but it satisfies the demands of the edge and research customers very well.
https://deepmind.google/models/gemini-robotics/
Google is mostly the party behind the whole VLA principle.
The decision to scare the shit out of normal people early on was a short-term decision that will cost everyone dearly (it led non-technical people to believe in things about LLMs that just aren't true—even worse, they've now made difficult business decisions on bad information with seemingly limited consideration of secondary effects).
The amount of short-sightedness alone should give pause, but at least in America, we seem to have collectively lost our minds in worship of the almighty dollar.
Judgment day approaches.
Aren’t you already describing a laptop computer? Add a local LLM and you’re fully there
but now I think they probably have bitten more than they can chew.
Apple already proved with their unified memory - that as long you have the capacity you can run capable models locally - thereby goes demand for inference if everyone is running some model locally.
For training - Chinese models have proved that you don't need the latest & greatest in Nvidia hardware. Same as TPUs.
only time will tell.
They are also a robotics AI company with Omniverse. They are also an AI company with Nemotron. They are also a bleeding edge network equipment company after the Mellanox aquisition.
They stand to make a lot of money if they succeed in every venture. Good for Jensen taking risks and driving innovation, I hope they succeed in chewing even 50% of what they bit off.
This is the inherent downside of such a rapid rise to being the world's most valuable company AND still being considered a growth stock. At their massive scale, the number of new adjacent businesses that have both sufficient size and potential growth is limited.
> I hope they succeed in chewing even 50% of what they bit off.
Anything approaching that is vanishingly unlikely. They're being forced to play the game more like a VC. The question is if a few unicorn winners can offset dozens of losers. The challenge is that, unlike a VC, their bets are much more correlated around AI.
The absolute fastest desktop Mac GPUs cannot beat an Nvidia laptop GPU in prefill or inference speeds. Apple Silicon is a non-entity for professional datacenter deployment and arguably unusable for frontier models at agentic context sizes. AMD is Nvidia's primary worry, and they're not doing much better in terms of GPGPU SOC compute.
You can believe all you want that the dinky little jetson boards were desktop grade when historically the ARM SoC portion of a jetson board couldn't even keep up with broadcom/rockchip SoCs. It's taken until recently for the actual arm compute portion of Nvidia SoC's to be worth a damn at all, and they still fall far behind Apple let alone the rest of the pack like Qualcomm/Samsung.
You can believe all you want that good single-core performance will corner the edge compute market. It hasn't, Graviton has more buy-in than any Apple Silicon chip ever got.
Nvidia dropping from being a $5 trillion company to a $242 billion company would be 1929 levels of bad.
Global economy end of days stuff, especially since Nvidia can't crash that hard without a lot of other stuff crashing with it.
And I am pretty sure that their value will drop in 5 to 10 years.
https://stockanalysis.com/quote/lon/VWRA/holdings/
That is an astounding figure and means markets are highly imbalanced, unless you think Nvidia represents 5% of the global economy.
That’s still a lot, but “the global economy” over the horizon where those earnings are not discounted to 0 contains a lot more than what’s listed in VWRA today.
This consolidation is scary. I've started putting a little bit in emerging markets and China. I cannot trust the US long term, hopefully I'm wrong!
Even back then, some economists used the "hidden wallet auction" as an example of how this could happen.
To summarize:
- there is a wallet
- you don't know how much is in the wallet
- you bid on amount to buy the wallet
- if you get the highest bid you win
- crucially, if you lose then you still have to pay
This is often cited as a game that you do not want to play b/c it's a. hard to predict the upside, b. the downside is huge.
That being said, people still got into these auctions and because of sunk cost fallacy, decided to keep bidding even if they might lose.
The hyperscaler race feels a bit like the above but no one seems to ant to admit it.
Yes, for programmers and tech companies AI is kinda boring now, but AI integration in general is still kind of uncharted territory.
There are so many small companies and individuals just getting started with AI today and I believe a large the customer base (and revenue) is still untapped. Hell, I’m discovering new use cases regularly still and the average mismanaged 30 people whatever SaaS vendor probably didn’t even get started yet.
Jason Kottke almost didn't found his blog in 1998, famously quoted as saying: "I thought I was too late, that no one would be interested." Needless to say, the internet was a tiny joke in 1998 compared to what it is now.
We are just barely scratching the surface of what's possible with AI, both in terms of the leading edge and in the 'torso' of the economy (the portion you're describing).
Folks from Silicon Valley working in AI-forward companies have a skewed perception of how many people have adopted this technology so far. Codex recently celebrated hitting 10 million users. This is a great milestone and all, but to put it in context, Microsoft office has a billion users. Sure, many people use Claude Code and or some other harness and the growth is staggering, but the overall scale is tiny compared to software as a whole. Costs of serving and usage are still very high, prohibitively so for many, so we aren't even close to market saturation.
And even at the leading edge, people who do work in those AI-forward companies; models are still slow, require hand holding, and produce suboptimal outcomes sometimes. Imagine the value when instead of needing to prompt it once per 30 mins, you prompt it once per day. Then once per week. Then once per month. Imagine all this running not on 3 trillion parameter models, not on 10 trillion, but 100 trillion. What kind of computer infra will be needed then? Certainly more than we have today.
Then, move five years forward from that year and look back. In every case, people think, "Hah! The internet was so simple then!"
In 2031, I suspect we'll say the same about 2026.
So much of the dotcom bust was essentially: "anything internet will work", but even after it was pretty easy to find different ways to publish web pages or migrate to new forms of social media, video, short-video, etc.
It's orders of magnitude harder to automate work, which is the business value proposition of AI (whether you are displacing worker or adding breakthrough capacity), and it's not clear the LLM hyperscalers will get the value-add from that.
On the consumer side, it might be a race to the bottom: same ad profits, but now you need AI to produce it.
Or the companies spending trillions of dollars can do a Manhattan Project (Or X-Prize) and let a thousand startups work on it. One will succeed. Between Google, Amazon, FB, Microsoft, Apple, AMD, Qualcomm, Intel (and dozens of other companies) there is enough economic incentive to do it. Also, isn't this what AI is supposed to be extremely good at, CUDA experts can continue to write CUDA (without having to learn anything new), a translation layer will rewrite it. If software can be one-shot from markdown files, this can't be impossible.
It's possible Google has intentionally decided to take a more conservative blended approach than purely competing at the bleeding edge of the frontier. If so, they obviously have no incentive to state it publicly but the recent departures and financials are consistent with the idea. It also makes sense that a company so much bigger, longer-term and (somewhat) more diversified than pure-play frontier labs would play the game to align with their strengths (capital, balance sheet, breadth, etc).
In their position, why not take an 'arms supplier' strategy in the near-term while drafting behind the frontier labs as a fast follower in AI, essentially betting the AI race is more akin to the Indy 500 than a quarter-mile drag race. If they're wrong and it IS more like a drag race, Google is in a better position to absorb and adjust than a frontier lab, for whom current valuations and capex spending requirements nearly require this to be a relatively short, winner-take-all race.
Gemini is already good for enterprise users, most companies I know are using Gemini 3.1 to interact with their email and sheets and creating presentations on the fly.
Perhaps this has something to do with the economic dislocations and world wars between the 1870s and today?
Building a business model on the belief that “this time is different” always finds storms on the horizon.
The current setup can’t sustain a downturn, even if yes 20 years from now point B is likely to be higher than present.
That’s the danger. Those that are going to get wiped out by the AI bubble burst aren’t wrong about AI being huge long term, they just put themselves in a position to not survive the storms that happen between points A and B.
Over those years, my NVDA stock has been by far my biggest winner. I'm now up more than 1500% on it.
Let the dooming continue
American history education needs some dire reform.
Counterpoint: xAI pooped out a frontier model based on nothing but capital and one man's desire to push a right-wing political narrative. Google has the talent, and the money, and the experience, they just need some leadership.
Disappointed by the lack of Tom Cruise.
What a weird reason to stop reading an article.
Bayesian inference and all....
1. Circular investment/spending.
2. Too much capital in the system, so returns cannot be hit regardless because the barrier is too high. (Evidence being every capital cycle in history)
My guess is at least another 2 years, as most people don't use AI yet, or maybe more precisely, AI is not used in the underlying workflows (which are invisible to the consumer) that make up most people's jobs.
Who knows if I am right. My original post was just pointing out that what you say is about timing, not whether it's true or not, because of course its true.
ps. Being influential in Silicon Valley just means you are influential, it does not mean substantial. Leopold is still influential and gets money thrown at him at $100s of million despite having no substance.
https://semianalysis.com/
They are the marketing wing of the AI ecosystem. Their recent article on how SpaceX would drive 500B in data-center revenue was ludicrous-mode. Lets revisit this in a few years.