xek, 5 months ago

@simon FWIW, I tried following the instructions there in a fresh venv and got Error: 'gguf' is not a known model. llm --version shows 0.12, not sure if I'm missing a plugin or something that adds this.

simon, 5 months ago

@xek Did you install the latest llm-llama-cppplugin?

xek, 5 months ago

@simon Ah, sorry, forgot to include that. llm plugins shows it at 0.3.

jeremybmerrill, 5 months ago

@xek @simon Same error, actually. (Likewise llm 0.12, llm-llama-cpp 0.3) If it's helpful, llm models also doesn't show any gguf-related output.

simon, 5 months ago

@jeremybmerrill @xek what does "llm plugins" output?

jeremybmerrill, 5 months ago

@simon @xek from $ vllm plugins I get llm-llama-cpp and llm-gpt4all (installed separately)

xek's suggestion of $ llm llama-cpp models returns {} but fixes the problem and I can now run llm -m gguf ... (well, I run out of RAM, but close enough and I should've anticipated that)

simon, 5 months ago

@jeremybmerrill @xek OK I'll look into that, needing to run "models" like that is weird and shouldn't be necessary

osma, 5 months ago

@simon
Excellent as always! Thanks!

Minor nitpick: You say that Mistral Small beats GPT-3.5 on every metric. But in the table it has slightly lower scores for WinoGrande and MT Bench.

simon, 5 months ago

@osma Oops good catch, thanks, I'll update the copy

simon, 5 months ago

Noteworthy that Mistral 7B was released on September 26 and there are already seven LLM plugins that can execute it, either locally or via a hosted API:

llm-mistral llm-llama-cpp llm-gpt4all llm-mlc llm-replicate llm-anyscale-endpoints llm-openrouter

Mistral appears to be establishing itself as the default LLM alternative to OpenAI's models

simon, 5 months ago

Added another option: you can run Mixtral as a llamafile and then configure my LLM tool to talk to it via its OpenAI-compatible localhost API endpoint https://simonwillison.net/2023/Dec/18/mistral/#llamafile-openai