this post was submitted on 25 Mar 2025
39 points (100.0% liked)

Technology

1087 readers
35 users here now

A tech news sub for communists

founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
[–] pinguinu@lemmygrad.ml 5 points 6 days ago (3 children)

The Mac Studio they talk about running the 685b model on costs 12 000 dollaridoos (after 20% taxes here). I get that it's a consumer device and that it draws less power, but at that point you'd just get a server for less. The power consumption is an outlier since it has everything on chip

[–] yogthos@lemmygrad.ml 8 points 6 days ago (1 children)

I think the key part is that you can run these large scale models cheaply in terms of energy cost. The price of hardware will inevitably come down going forward, but now we know that there is no fundamental blocker for running models efficiently.

[–] pinguinu@lemmygrad.ml 2 points 6 days ago (1 children)

I generally agree, but given how niche a powerful SoC like this is, I doubt it matters right now (<5 years). I understand it proves a point, but I wager there's still a long-ish way to see power-efficient hardware like this available for cheaper (which will most likely come from China natively)

[–] yogthos@lemmygrad.ml 6 points 5 days ago

Yeah, a 5 year or so timeline before we see SoC design becomes dominant is a good guess. There are other interesting ideas like analog chips that have potential to drastically cut power usage for neural networks as well. Next few years will be interesting to watch.

[–] CriticalResist8@lemmygrad.ml 4 points 5 days ago (1 children)

But imagine what you'll be able to run it on in four more months. But yeah, it's stretching the definition of consumer hardware a bit.

[–] pinguinu@lemmygrad.ml 1 points 5 days ago (1 children)

You can use the smaller models on (beefy) consumer hardware already. That's something, right? 😅

[–] CriticalResist8@lemmygrad.ml 3 points 5 days ago (1 children)

I want the full 1TB model running on my 10 year old linux laptop

[–] pinguinu@lemmygrad.ml 2 points 5 days ago

Just put your persistent memory as swap. Easy