there’s a big need for something stronger than #RAG, but more flexible and cheaper than a giant all-knowing #LLM.
A great part about RAG is that it’s just a database. You just issue INSERT/UPDATE and yeah, that’s how you maintain knowledge. No million dollar training process
@kellogh the interesting idea that I keep coming back to #RAG is that it can direct any model to give you a more precise answer. The idea of fine-tuning or additional training doesn’t reduce the hallucination effect.
@mako the example i gave — GPT4 got less confused than perplexity — RAG isn’t everything, and the main places it falls apart is in its text interface. it got confused because the context was partitioned across messages, so the embedding used for RAG didn’t have enough information.
the reason we do smaller chunks for embeddings is because, as the chunk size increases, it continues to add information and it gets harder for the embedding to represent the main themes
@mako i guess we just need an innovation in embedding models — prompt-able embedding models. give it the full context, but only generate an embedding for the last n-words. I’m not entirely sure that hasn’t been done already…
it’s one of those things that OpenAI probably won’t do because it doesn’t get us closer to AGI (RAG doesn’t help symbolic reasoning), even though it’s extremely useful and would make for overwhelmingly practical use cases
@kellogh The more I dig into "2024 AI" the more I feel like it is no more likely to solve for symbolic reasoning than "1959 AI" or "1985 AI" because we still don't have the scale to compete with organic cellular density and that makes not optimizing for more achievable goals, like machine-built expert systems (what I think your middle ground turns out to be) all the more aggregious, since we really do have breakthroughs in 2024 that can propel us forward compared to 1959 and 1985.
@kellogh how did you get to the conclusion that AGI is circumscribed by symbolic reasoning? I guess I don't have a satisfying definition of AGI. I expect AGI will be assessed by moving goalposts to a feature set likely to depend on RAG in practice.
Add comment