It seems to be a LLM that he has trained by using other LLM’s (Falcon and MPT). Training is one of the largest costs and biggest difficulties in producing a viable LLM, and using another LLM to train yours is a neat shortcut ( which is why ChatGPT and Llama expressly forbid it in their licences) .
It’s an interesting development, having a narrow-scope, dedicated to a single task LLM, instead of General Purpose one, but using a general purpose model to train it. It would presumably narrow the scope for hallucinations, albeit for a narrow scope of answers.
Add comment