seinecle, 4 months ago

cc @TedUnderwood ?

TedUnderwood, 4 months ago

@seinecle I don’t actually have first hand experience on this yet though I want to soon.

My understanding is that memory (VRAM or Apples unified memory) matters a lot. Also people sacrifice some quality by quantizing models to make them smaller.

seinecle, 4 months ago

@TedUnderwood very useful, thx. Also, looking at hosted servers with GPU, starting prices are around 150$ / month (Hetzner). More expensive than my usual projects :/

TedUnderwood, 4 months ago

@seinecle the advice I’ve been getting over on bsky is that you can do a lot locally with Apple silicon