@j_bertolotti Okay but hear me out here. We know that in the animal kingdom there is a correlation between brain cells per body mass and intelligence. I think what the AI people are talking about is getting up to the level of a human, dolphin, elephant but what you're talking about is a kind of plateau when we reach that point. In other words, brain cells do matter but with diminishing returns and where humans are already at the point of marginal (or worse) gains. You know what I mean?
@apodoxus We didn't get LLMs or generative models by using the same networks as in the '60s but a million times bigger. We got them by rethinking the basic way these networks are structured. Any qualitative step forward is also likely to be due to a smart idea and not more brute power.
@j_bertolotti@apodoxus Dunno - to within some range, surely there's some sort of trade between the number of neurons used and the cleverness of their arrangement. I highly doubt that LLMs are anywhere near as well-organized as the human brain, so they may well be in the realm where "more neurons" still gives huge advantages. 🤷♂️ At some point there's a diminishing return, and then we'll need to talk about "better structure" (which I, personally, would guess means "more complex structure" - I'm staggered by how 𝑝𝑟𝑖𝑚𝑖𝑡𝑖𝑣𝑒 the current crop of AI really are) but at any one time we might well be at a point where sheer quantitative growth can still beat out real advances...
Add comment