All in all pretty decent sorry I attached a 35 min video but didn't wanna link to twitter and wanted to comment on this...pretty cool tho not a huge fan of mark but I prefer this over what the rest are doing...
The open source AI model that you can fine-tune, distill and deploy anywhere. It is available in 8B, 70B and 405B versions.
Benchmarks
What do 8B, 70B, and 405B refer to?
Parameter count. 8 billion ... Colloquially the model size, and hence how smart it is. 405 billion parameters is big. We didn't have anything even close to that size and with current technology to download and tinker around, until just now.
I mean, from what I can tell we still don't, at least as home users. The full size model won't fit on any commercial hardware. Even with a top of the line 4090 GPU you're limited to the 8B model if you want to run it offline, and that still charts lower than the last-gen 70B model.
Still cool to have it be available, though.
The full size model barely runs on 160 GB VRAM and something like 200 GB CPU buffer. I’m trying to scale it across many GPUs but haven’t had much luck yet.