AI, memory, and a simple bet

Markets have fascinated me for a long time, and with the recent AI bubble going on and NVDA stock seeming to be burning through limitless propellant to push the company out of this galaxy and beyond (an obvious metaphoric way to emphasise it's strong growth), I have been looking at the AI narrative as an outsider. I don't work at a financial institution with access to research that costs more than my yearly salary, nor I work for a frontier tech company to know exactly where it's headed, but there is something about AI that seems pretty simple to me, and I want to write it down.

Some of the smartest people in the world have all shifted to working on AI systems, not those that only answer to a query or that turn a prompt to a video, but AI systems that actually do things, and do things better than us. Take writing software, operates programs, working through problems, and eventually doing these things at a cost that makes it useful to almost everyone. Since autonomous driving is largely solved and keeps improving exponentially and a fraction of the cost, these AI systems I'm referring to might be an AI autopilot that flies the planes better and faster that a human pilot can do.

Nowdays, I see a lot of the focus seems to be shifting towards coding. I think it's is still pretty terrible at doing larger tasks reliably. It's either that, or me not having access to the right tools, or me not paying enough for the latest and greatest tech, or me not using/navigating it correctly. Either way, it can produce something that looks convincing, then get lost in its own changes. But I do think that it will get pretty damn good, and probably sooner than I would expect. I don't ever bet AGAINST something that can just get better.

My question recently has been around what does an AI that CAN DO actually need?

Obviously, lots of matrix multiplication power, faster GPUs and CPUs, and high bandwidth bridges. But it also needs somewhere to keep everything it is working with. Just like us humans (but it will obviously be much better than us), it needs to store what it consumes, and it needs to store what it produces, and it needs to access information that it knows and doesn't know, fast, locally, and at a decent cost. When this happens, I don't really think a human can surpass it. My reasoning here is biased based on my limited intellectual capacity, but I still believe it would turn out to be true.

In general, I think it's all about input and output, and in the Ai systems world, it's digital, so it's gotta stay in some memory.

Think about a regular computer program that does a task. It needs a place to store instructions, input, a really fast place (and algorithms) to keep its working state and outputs, and a fast enough (although fast is never enough) place to keep it's long term output. Now imagine an AI system working on an entire software project. It needs access to the code, dependencies, requirements, results from previous attempts, and whatever else is relevant to the thing it is trying to accomplish. This last part is important - so many things on the internet can be relevant to the task (ex. research, OSS code, certain libraries etc.) so that it's output can be extremely good.

Simply giving it more and more information will not necessarily make it better. It needs the RIGHT information, and it needs to access it quickly and efficiently. This inherently means that disk and network I/O, just like with regular programs should be a bottleneck.

And the AI is all about producing. Code, documents, images, intermediate results, logs. Some will be useful, some will be garbage, and some will be needed for the next task. All of that creates more digital state to store, access and move around. Note that this only applies to coding, which I'm mostly familiar with. But if I were to think about other industries like transport, logistics, medicine, food, construction, metals etc. etc., well, damn, that should logically imply an explosion in demand in all these things, and memory.

There is a distinction here. The fast memory used while a model is running is different from the storage where generated files end up. Keeping a million documents on an SSD does not mean they all need to sit in expensive memory next to a GPU. But an AI system doing useful work will need both, in different amounts.

My reasoning is pretty simple: when (not if) these systems become useful enough and will keep improving faster and cheaper, more people will use them, for more tasks, and more of those tasks will run at the same time. I think that, just like with anything else that gets cheaper to access over time, cheaper tasks will encourage enough additional usage to keep demand for memory growing.

I do not know much about memory fabs, manufacturing yields, or how the entire industry works in much detail. I know about ASML and its monopoly, they sell the machines, but usually most money is made by folks that use the machines to bring the product to the consumer. I understand why programs need memory, and nowdays, why AI systems need memory. And I find it difficult to imagine millions (or billions) of AI systems doing increasingly complex work without a substantial amount of it.

Well, who produces (and/or designs the Verilog schematics for) the memory?

I know one and I have been digging into it's SEC filings - Micron. There's probably others, but I know that there should only be a handful of players, and it's some sort of monopolistic industry. Just like ASML that should keep printing cash in the next years, although that one is a big monopole with zero competition.

But as a starting point, the idea seems fairly straightforward to me. When (not if) AI becomes capable enough to do useful work cheaply, I expect a lot more of that work to happen and that work wouldn't just need a place to be stored, but a place to be accessed, quickly and efficiently, and I want to understand the companies supplying it.

I just hope that Micron doesn't become the AMD of the AI cycle and they deliver the right vertically integrated technology ON TIME.

AMD is pretty good and interesting, although with their own software issues, and a decent part of the portfolio, but for whatever reason, it doesn't go up like NVDA. Maybe it will test my patience, and maybe Micron will do so, too.

Thanks for reading!

PS: Oh and did you know that there's a single company that has the right ships (and just a few of them) to lay out network fiber optics across the ocean? Maybe that would be end up being the unwanted bottleneck in the end, although, I think Elon and co. will make orbital network connectivity cheap and abundant for everyone (even the aliens).