@ErikJonker this may be true, but unfortunately the inference cost of the models at this scale is completely unsustainable for that sort of use, or anything equivalent.
It takes 128 GPUs per instance to run GPT-4.
The cost per conversation on GPT-3 was 38c.
GPT-4 is roughly 3 times as expensive to run: or $1.14 PER CONVERSATION.
At those rates, routine casual usage is just not viable...even assuming we could give every child 128 dedicated GPUs, which we definitely cannot.
@mattlav1250 ...if Google is good in 1 thing it's infrastructure and scalability, i guess they have thought this true, also they use their own hardware
@ErikJonker Yes, but AZURE, which runs GPT, is ALSO a huge scaling machine, at least as large as Google Cloud.
It's true Google's dedicated Tensor processors / VPUs are more efficient for this task than generic GPUs, which probably does give them a slight edge relative to GPT.
But that still doesnt change the fundamental maths of this price point: The FLOPS cost is the FLOPS cost, no matter how efficiently you allocate it around. At same point you just have to pay it.
@ErikJonker Also, I strongly dispute your "its Google, so I'm sure they must have worked out the economics / know what they're doing" take.
The history of Google Stadia, Duplex, Duo, Allo and the whole rest of the GOOGLE GRAVEYARD would strongly suggest that that is not how their organisation does business, and is not a reasonable assumption to make.
@mattlav1250 Google Stadia was great by the way and worked perfectly, I was a happy customer 😃 , it did not fail because of technology or infrastructure ,also one of the products they killed and paid all the costs back to the customer (software, hardware etc)
@ErikJonker Yes EXACTLY...the TECHNOLOGY worked fine...just like Bard etc no doubt will. At no point have I indicated that I don't think they know how to build technology. They are highly capable technologists.
It's the ECONOMICS /marketplace that they never worked out, just like they doubtless have not worked out the economics of Bard before launching.
Google's organisational culture is to experiment and launch things on spec and see what happens, then retrench or discontinue when it doesnt.
@mattlav1250 ..great points, ofcourse we can question the economics of a product google is launching, they often fail which is part of their strategy, i only have the impression that their infrastructure is more capable for worldwide/broad services in this field then Microsoft, but i can be totally wrong, time will tell.
Add comment