It's the GPT-3.5 version, which is prone to all sorts of mistakes and hallucinations - further strengthening the pattern where most people form their impressions of what this stuff can and can't be useful for through access to the weaker models
"When I speak in front of groups and ask them to raise their hands if they used the free version of ChatGPT, almost every hand goes up. When I ask the same group how many use GPT-4, almost no one raises their hand. I increasingly think the decision of OpenAI to make the “bad” AI free is causing people to miss why AI seems like such a huge deal to a minority of people that use advanced systems and elicits a shrug from everyone else."
@simon I seen charts that say gpt 4 isn't much more accurate than gpt3.5 . fits my impression.
Blaming GPT3.5 on why people are not convinced by LLM's they call AI is not convincing me.
The most recent I've seen is this one from the Claude 3 release, which has GPT 4 scoring significantly higher than 3.5 on every benchmark listed https://www.anthropic.com/news/claude-3-family
My own impression has been that GPT 4 is far more useful and less likely to make mistakes than 3.5 for the kinds of prompts I use
@simon oh vendor performance information...
These benchmarks and their data are also in the training data. LLM generally perform worse with alternative formulations of the questions in the benchmarks. https://arxiv.org/pdf/2402.19450.pdf
GPT4 is the best, but size does not justify the cost/size. GPT3.5 now The "vanilla LLM".
It's the defined normal and a standard you can talk about.
@Zeugs yeah I'm pretty skeptical of the numeric benchmarks - I trust the ELO ratings leaderboard more https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard but mostly I trust my own experience using the models - at least in terms of which models work better for the specific tasks and prompts I use them for
@Zeugs To answer your original question then: I believe GPT-4 is significantly more accurate, more useful and less likely to make mistakes/hallucinate than GPT 3.5 based on a combination of:
My own extensive experience using those models
Various benchmark scores, despite their questionable reliability
The Chatbot Arena scores, which reflect large numbers of votes from real humans trying out real prompts
@simon
GPT 4 is just not that much better than GPT3.5, also in a hallucination benchmark, which I don't have at hand.
It just not justifies the cost for anyone but MS. It's not worth it for OpenAI, the customers and companies.
Maybe your experience with GPT3.5 is extraordinarily bad, but works for anyone else. 🤷
I played it through. I started with GPT 2. It's better but has still the same flaws.
I have to look at this for work but it's pointless with the existing expectations.
@Zeugs For me, GPT-4 just tipped the balance past a point where the hallucinations and mistakes were no longer so common that I felt like I was wasting my time trying to get LLMs to do useful things
So even if it's not an order of magnitude better, the improvements were material for me
@aarbrk my guess in terms of theory is that size matters - they're called Large Language Models for a reason
It's still all about probabilities. The larger the model, and the more training data that's been poured into it, the more likely a prompt is going to result in a response that captures something accurate about the real world
Add comment