Mr beast: I bought one of the world’s fastest computer and we are going to run the biggest Minecraft server in the world for a competition with 1 million players! the winner will get 10000 dollars and also I’m donating 10000 diapers to Angola
The array’s services were used by scientists across the state of Wyoming and the rest of the country when needed. The 5.34 petaflops system was mainly used for weather and climate studies, helping the National Science Foundation better study climate change and other Earth-related sciences.
Does the hardware being all so arranged as it is in this manner to create a supercomputer make any difference to that evaluation? Like does the work of putting all the outdated hardware together in the complex way needed to make it functional for supercomputing make it potentially cheaper than buying more modern hardware but having to build it all yourself?
Does the hardware being all so arranged as it is in this manner to create a supercomputer make any difference to that evaluation?
The storage drives for all of this have been stripped. You can’t just run commands on the hardware… you have to figure out how to cluster things with software, buy drives for it all, have it all installed in a datacenter somewhere which is going to cost way more than the purchase price.
The labor costs for the technical people required to do this are way more than half a million a year.
Didn’t know they’d taken out the storage drives but I was aware despite my general ignorance that it’s not turnkey ready to go. I guess what I’m wondering is, is there any part of the of the process involved in designing and building such a supercomputing cluster that is already taken care by buying it in the manner that it has been sold and could that in any way offset the increased costs of trying to bring such a cluster online rather than starting from scratch? I’m not saying it is the case, so much as wondering aloud for anyone with expertise to chime in, to see if that’s a way it could make sense.
I understand there’s a mountain to climb to bring this thing in to a usable state for anyone, but could it maybe get you to base camp more quickly?
I’m guessing that due to the nature of this server (scientific calculations by scientists and students who sent jobs remotely, if I understood right) everything was done on RAM and then stored on SAN (separate servers) afterwards
Otherwise taking out all the storage from each single blade AND putting it back nicely in the racks would have been a massive job, if they did that I would expect to have it half dismantled on the pictured
Some of those nodes are broken, it’s got coolant leaks and the processors are 8 years old. I read in another article they meant to replace it a couple years ago but couldn’t due to COVID supply chain problems (makes sense tbh).
What I’m saying is, it might be “assembled” already but due to the power draw, you definitely don’t want to run something this out of date. It’s huge. Might be a logistical problem to even get enough juice for this thing. Another poster pointed out it might need its own electrical substation. IMHO it’s only worth parting out. Far too expensive to run.
Those cabinets look nice though, with the Cheyanne graphic on the front.
Selling for parts. Techtechpotato did a rough estimate and RAM/CPUs alone are currently worth and 1M (using eBay prices). If you add all the support components (network cards, mainboards, PSUs) it might be worth it.
The piecemeal nature of selling thousands of parts means the wages for a group necessary to coordinate it all would probably make the whole thing not feasible.
Ebay prices are higher than market prices imo. 15% ebay cut + 3% paypal fees + sales tax + shipping is brutal.
Yes and no. You can sell easily 2-3 units at the current price, but 8000 pieces…
Technology is improving fast, the E5-2697 v4 has a similar benchmark to a Ryzen 5600, which uses half the power, and making a computer with desktop components is cheaper (except for used registered ECC RAM which is being dumped for cheap)
And that Ryzen is the old generation, the newer are even faster
It’s likely not worth using, maybe splitting the nodes out individually for colleges to use for research projects for a few years. The cabinets are probably worth something because of the graphics on them. The power consumption, lack of the high speed storage, and large scale industrial cooling system make it impractical to use. You would probably need an entire power substation in order to run it. You could probably install it at an industrial site, but you would still need to come up with networking and installation, which could cost more than what you paid for the thing in the first place.
I mean, even if some company has a practical need for it, or even just wanted to sell compute time on it, it wouldn’t be worth it due to the operating and installation costs. Although, it wouldn’t surprise me to see individual nodes from this poping up on /r/homelab, those guys are nuts.
It looks like each blade has 4 modules with 2 processors each with up to 9 blades plus management and networking in each blade cabinet and 4 of those in each rack. Liquid cooling is only an option, so it could be possible to run it on air only. I couldn’t find much on the cooling system other than it’s self contained if you have one if the separate cooling cabinets. It does look like is an air to water radiator. You could pay run it off of a pool pump or something.
Ian Cutress recently did a video on the topic here (I think he changed the title to reflect the end price of the auction), which does a bit of a breakdown. You for example also have to add shipping costs (from a certified company) to the price.
Pretty crazy to think that it is actually not sure whether spending less than 500k on a supercomputer is worth it. Goes to show how far technology has come.
I guess if everything sells you might make profit, but then it also comes with a lot of hassle and risk. And for actually using it, I imagine that electricity cost would be a huge factor.
I mean there definitely are some valuable metals in there, but I can’t imagine that this is a competitive price to pay for them, especially since extraction wouldn’t be easy. And some parts do have value, even if it ends up being the case that running the full cluster isn’t economic anymore.
I do wonder who at this point could use all those processors (and Mainboards), but the ram might still be reasonable to use, maybe the cables, the cabinets themselves too. And I think the video also mentions that there are two managing servers. Those might be most likely to actually be useful for their original purpose.
Pretty crazy to think that it is actually not sure whether spending less than 500k on a supercomputer is worth it.
Has more to do with the market for supercomputers. They are monsters to keep fed so it’s not a question of if you can buy it but rather if you can run it. But customers for supercomputers are in the market because they need the most raw power that the technology is capable of supplying, so buying and installing a decade old supercomputer (which is going to have the same operating costs at a lower capability than a new one) doesn’t make sense.
You also have to consider that the downtime’s going to be a lot higher on this equipment as you’re going to start having components hit the end of their useful life.
Have fun selling 8000 individual CPUs on eBay, so handling, storage, shipping… and even if that is for free it would only be 1/4 of what they paid. Not too mention that flooding the marked will mean the pieces goes down.
13% eBay fees, around $5 to pack and ship it, not including your time. Around $8 net per chip, so, $60k less returns/losses, then taxes on that. It might be worth it for one of those large liquidation companies, but they usually charge companies to recycle their equipment or pay very little at auctions.
For the E5-2697 v4 that are in this server I see much higher prices even on Chinese e-waste resellers, about $45
But I don’t think that there’s enough demand on the market for 8000 of them
Maybe if you pair with those unstable "x79” Chinese desktop motherboards with server sockets and then sell them as package like “ATX motherboard with 18 cores CPU and 256 gb ECC RAM” maybe can find more customers
For the ram they’re going to make much more, 5000 sticks of 64gb ECC DDR4 are still expensive
Edit: no, I accidentally saw the prices of unregistered ECC RAM. Registered ECC RAM is very cheap and super abundant as only servers use that. Corps aren’t stupid and buy used memory for their expensive servers, so when a server is decommissioned, registered ECC RAM is dumped on ebay for pennies
Ah I thought I saw v3 when I looked it up, so about double the price then but with fees (and probably low demand) my original estimate was probably correct enough
I haven’t run anything older than Skylake since 2020. I imagine anyone planning to run these either hasn’t done the math on energy costs or lives somewhere where electricity is dirt cheap.
I currently run 2 of these xeon CPUs in my server. When I specced it out I was looking at about $300/year of energy usage and that was running at like 50% capacity daily. The thing has half the cores disabled because I currently don’t need that much compute and the only thing running on it daily is TrueNAS. It has ran multiple game servers and labs in the past few months and still doesn’t use that much compute. Really just depends what you’re doing with it.
Add comment