this post was submitted on 27 Jan 2024
24 points (96.2% liked)

LocalLLaMA

3392 readers
2 users here now

Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.

Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.

As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.

Rules:

Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.

Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.

Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.

Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.

founded 2 years ago
MODERATORS
 

On my machine I'm running opensuse tumbleweed and has the amdgpu driver installed. I use it for gaming and recently I've become interested in running LLMs. So I would like to keep a balance of both without compromising too much on performance.

I know that there are proprietary drivers for AMD cards but I'm hesitant to install it as I've heard that it performs less efficiently in games when compared to the open source driver.

I'm mainly confused about this ROCM thing. Is it not included with the opensource amdgpu drivers ? Or is it available as a separate package?

So what driver to use ?

Or perhaps, is it possible to run oogabooga or stable diffusion within a distrobox container (with the proprietary drivers) and still keep using the open source gpu drivers for the Host operating system.

top 16 comments
sorted by: hot top controversial new old
[–] taladar@sh.itjust.works 10 points 2 years ago (3 children)

To do general purpose GPU calculations on AMD hardware you need a GPU that is supported by ROCm (AMD's equivalent to CUDA). Most of the gaming GPUs are not.

There is a list here but be aware that that is for the latest rocm version, some tools might still use older versions with different supported devices.

https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html#supported-gpus

[–] madnificent@lemmy.world 5 points 2 years ago (2 children)

Has that changed recently? I've ran ROCm successfully on an RX6800. I seem to recall that was supported, the host OS (Arch) was not.

[–] turbodrooler@lemmy.world 2 points 2 years ago

No, GFX1030 is still supported.

[–] taladar@sh.itjust.works 2 points 2 years ago

When I tried it maybe a year or so ago there were four supported chipset in that version (5.4.2 I think) of rocm but I don't remember which card models those were since they were only specified in that internal chip name. Mine wasn't supported at the time (5700XT)

[–] turbodrooler@lemmy.world 4 points 2 years ago

This link is misleading. For example, the Radeon RX6800 IS supported because it is the same chip as one of the Radeon Pros. GFX1030. Many others are too…though support does not go very far back.

[–] exu@feditown.com 2 points 2 years ago

Llama.cpp supports OpenCL as well and performs better than rocm in my limited experience. That should work on basically any GPU.

[–] madnificent@lemmy.world 4 points 2 years ago

Latest ollama has support for AMD GPUs. I had to compile from source to make it pick up the GPU on my system.

[–] turbodrooler@lemmy.world 2 points 2 years ago

Look into llamafile. It makes things so easy.

[–] Falcon@lemmy.world 1 points 1 year ago

Basically, RoCM and CUDA allows one to do math on the GPU. Most Linear Algebra operations (i.e. LLM or NNs and ML generally) can be parallelized over a GPU which is much more performant than CPU.

To perform calculations on GPU, one needs some sort of interface to to their programming language of choice, NVIDIA has CUDA which is in CPP with bindings to python: (pytorch, Tensorflow etc. ), Julia: Flux etc.

RoCM is AMDs solution, there bindings are young and not widely implemented.

My advice, play around with Flux RoCM and PyTorch RoCM just to get an idea. Suffice it to say, when I started doing RL and LLMs more seriously I gave up my colab and sold my AMDs to fund a 3060.

[–] turkishdelight@lemmy.ml 0 points 1 year ago

llama.cpp (and ollama) has AMD support through ROCm and also now Vulkan.