Google says its new algorithm reduces AI memory overhead by 6x which could be good news for the RAMpocalypse but bad news for Micron and co

Other than the AI bubble bursting or hype dying down, the other thing that could allow the RAMpocalypse to ease off is a technological change that leads to a dramatic reduction in how much memory AI needs. To that end, Google has cooked up TurboQuant, a new compression algorithm that promises to reduce memory demand by about 6x. And memory maker stock prices have already dropped, likely as a result of this.

Although we should resist being reductive and assuming Google's new algorithm is responsible for these market changes—lest we forget the effects on crucial material availability thanks to the war in Iran—a 6x claimed reduction in memory demand must surely account for at least some of it.

According to Google, TurboQuant "optimally addresses the challenge of memory overhead in vector quantization" and "achieves a high reduction in model size with zero accuracy loss."

In other words, it makes vector compression—which is critical for AI models understanding and processing information, as they do so using vectors—require less memory than it has until now and crucially without the normally associated loss of accuracy from compressing things down.

The basic idea, to exclude lots of details and simplify greatly, seems to be a shift from calculating things in terms of standard vectors and instead using a more absolute reference system. Which, to my non-mathematical ears at least, sounds a bit like moving away from vectors:

"Instead of looking at a memory vector using standard coordinates (i.e., X, Y, Z) that indicate the distance along each axis, PolarQuant converts the vector into polar coordinates using a Cartesian coordinate system. This is comparable to replacing 'Go 3 blocks East, 4 blocks North' with 'Go 5 blocks total at a 37-degree angle'."

This, ultimately, means no need for data normalisation, which should "eliminate the memory overhead that traditional methods must carry." Google has put the new algorithm through its paces in a bunch of benchmarks, and the results, according to the Big G, at least, show that "TurboQuant achieves perfect downstream results across all benchmarks while reducing the key value memory size by a factor of at least 6x."

Again according to Google, the results also "demonstrate a transformative shift in high-dimensional search... [allowing] for building and querying large vector indices with minimal memory, near-zero preprocessing time, and state-of-the-art accuracy."

Naturally, such a big improvement, if true, could drastically change the AI server market. Which means it could change the amount of memory that AI companies are wanting to buy from Micron, SK Hynix, and Samsung.

As identified by investor boffins, that is indeed what the market is predicting we will see, because stock prices for the big three memory makers have already dropped. Samsung's, for instance, has dropped by about 8% since a couple of days ago, SK Hynix's by about 11%, and Micron's by about 10%. And these are all actually after factoring in a slight rebound today.

If AI companies need to buy less memory, that will of course raise the amount of supply open to the general consumer market, including for gaming PCs, laptops, handhelds, and other goodies. Which should in theory mean memory gets cheaper.

This highlights the stark difference between what's good for memory makers and what's good for we end users. It's something I've noticed a lot over the past few months: the less stock and the more demand there is, the happier memory investors and analysts seem to be, and the unhappier consumers are. Which is obvious, of course, but it's interesting to see it to go the other way for a change.

We shouldn't consider anything a done deal, though. After all, Micron has already said there is "demand significantly in excess of our available supply for the foreseeable future." So for all we know, much of the 'freed up' memory production could just go straight back into AI server racks rather than our PCs. There's also the fact that compute hungry AI folk will just end up running larger models if the memory requirement drop. But there's a chance, at least, and I'll keep clutching at it.