lain, 4 months ago

@Moon or get the mac studio

Moon, 4 months ago

@lain the what

lain, 4 months ago

@Moon https://www.apple.com/mac-studio/

Moon, 4 months ago

@lain woah I never saw such a thing.

piggo, 4 months ago

@Moon @lain lmao over half of the thing is just a massive cooler

lain, 4 months ago

@piggo @Moon not everything can be as elegant and tasteful as a pc ML setup.

lain, 4 months ago

@Moon it seems to be pretty popular with the LLM crowd because it can run even the largest models

why, 4 months ago

@Moon or rackmount mac pro with 192gb 🤤

Moon, 4 months ago

@why holy shit that is expensive. also why does the rack mount only do 192 when you can get a macbook pro with 128.

coolboymew, 4 months ago

@Moon @why because Apple

Moon, 4 months ago

@coolboymew @why Surprised there's not a Saudi prince option for 1TB of RAM.

Suiseiseki, 4 months ago

@Moon For your freedoms sake, why not trade in your mac for a GNUbooted thinkpad?

Moon, 4 months ago

@Suiseiseki I have a thinkpad, I need a system to do AI with

Suiseiseki, 4 months ago

@Moon Have you considered not running proprietary malware bullshit generators?

Moon, 4 months ago

@Suiseiseki I only run libre AI software and models.

Suiseiseki, 4 months ago

@Moon I've looked for such, but everything I've looked at as been proprietary.

Is the software really free if it can't be run at all on freedom respecting computers?

lain, 4 months ago

@Suiseiseki @Moon there are like 5 different apache 2.0 LLMs and tons of engines to run them

Moon, 4 months ago

@Suiseiseki There are various model files in different sizes. They need to fit inside your VRAM. So, you can get normal-sized models and run. them on Linux. I do in fact have a Linux box with 24GB of VRAM but that is still not enough for the largest models. A Mac, if you spend the money, can run a model in over 190GB.

Suiseiseki, 4 months ago

@Moon The software isn't free if it can only be compiled against proprietary libraries and only run on certain models of proprietary GPUs sorry.

I don't see how Linux is relevant in this context, the proprietary GPU libraries manage the VRAM really.

Moon, 4 months ago

@Suiseiseki Oh that is true, on my Linux machine it has to use the proprietary nvidia driver. no way around that.

mischievoustomato, 4 months ago

no intel oneapi support yet? :(

allison, 4 months ago

@mischievoustomato @Suiseiseki @Moon the highest end alchemist card currently available is still pretty weak for ai stuff afaik, whether battlemage changes the situation remains to be seen

mischievoustomato, 4 months ago

i guess even meteor lake npu/igpus are too weak for that too :(

Moon, 4 months ago

@mischievoustomato @Suiseiseki @allison yeah I think you gotta go proprietary.

and nvidia drivers are such garbage. they break every time I upgrade and I have to recompile everything.

allison, 4 months ago

@Moon @Suiseiseki @mischievoustomato it absolutely can not be understated how much apple hardware just works for this usecase in comparison.

vic, 4 months ago

@Moon @Suiseiseki > if you spend the money

wowzers

Moon, 4 months ago

@vic @Suiseiseki Yeah, I cannot justify that price for only 192GB of RAM.

vic, 4 months ago

@Moon @Suiseiseki So I really can't just go get a MEGA ULTRA GAMER ELITE PC with a 4090 (24 GB VRAM) and run the biggest AI stuff on it by having it share/offload some of the model to system RAM? I've never really delved into the big AI/GPU stuff.

feld, 4 months ago

deleted_by_author

Moon, 4 months ago

@feld @Suiseiseki @vic I haven't kept up with the state of the art with games. It used to not really be a problem that VRAM wasn't shared because you just pushed never pulled. I assume shared is better because you don't even have to copy. But for AI stuff you're pushing AND pulling.