Technology

Ok, I walked over to my PC to give you a working command line for llama.cpp. You need to make sure it is compiled with support for hipBLAS / ROCm which is the equivalent AMD framework to CUDA, if you want it to run on your GPU.

./main -ngl 24 -m models/tinyllama-1.1b-chat-v1.0.Q5_K_M.gguf --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -i -ins

This will put it into interactive mode so you can try to chat with it. Running on my GPU it cranks out almost 160 tokens per second, which is way faster than anyone can type. On CPU (-ngl 0) it will make 90 which is still fast. TinyLlama is not a great chatter and should be treated more as a prediction or answer engine. i.e:

>Write a paragraph about borscht.
Borscht is a traditional Russian soup made with beetroot, potatoes, and a variety of spices. It is often served during the winter months in Russia, Ukraine, Belarus, and other Soviet-style countries. Borscht is similar to borscht in Poland, but has different ingredients and a slightly different preparation method. In Poland, beets are boiled until they become tender, then blended with potatoes and vegetable broth to create the soup. In Russia, beetroot is removed from the pot before cooking and replaced with other vegetables such as carrots, celery, and onions. The resulting mixture is then simmered until it is thickened, creating a hearty and flavorful soup. Borscht is usually served cold or at room temperature, and can be accompanied by sour cream, slices of crusty bread, or grilled meats such as kebabs.

It does know a surprising amount, considering it would fit on a CDROM