penguin42, 5 months ago

@stsquad AI stuff is interesting; I've played with the free ChatGPT's & Bard; they seem to be pretty good at explaining or searching for stuff;and yes I've seen others suggest use for review. That arm version conversion you mention is impressive; you can't blame it getting unstuck on an MMU! I'm tempted to try a local run; I'm being told that the llama.cpp runs OK on a 32G RAM host even without GPU. I find the way the parameter types shrink from f32->f16->f8->i4 fascinating architecturally.

stsquad, 5 months ago

@penguin42 the quantised 7B models run pretty well in my experience if you have enough RAM and cores. Certainly enough to experiment with.