Current research points to memristors, which can work both as memory cells, and as weights in a n×m grid representing a fully connected n->m layer that executes in 1 clock. I forgot which company was showing prototypes since pre-covid… and now Google is so full of wannabes that I can’t seem to find it, oh well.
Cerebras is at the limit of SRAM, that’s true.
Spintronics could be the next step, but seems to be way less ready for production.
Higher dimensionality would be nice, but even at 2D, being able to push multiple processes at once, through multiple n×m layers, would already give those 5 orders of magnitude, at least for inference. Since training also involves an inference step, it would speed that too, just not as much.
Self-training would be the next step after that… I don’t think I’ve seen research in that regard, but maybe I’ve just missed it.