SuspiciousCarrot78@aussie.zone to Selfhosted@lemmy.worldEnglish · 19 days agoDo you host your own AI?message-squaremessage-square81linkfedilinkarrow-up1109arrow-down129file-text
arrow-up180arrow-down1message-squareDo you host your own AI?SuspiciousCarrot78@aussie.zone to Selfhosted@lemmy.worldEnglish · 19 days agomessage-square81linkfedilinkfile-text
minus-squarebrucethemoose@lemmy.worldlinkfedilinkEnglisharrow-up55arrow-down1·edit-219 days agoAn aside for anyone reading this: https://sleepingrobots.com/dreams/stop-using-ollama/ And that barely scratches the surface. Please. Use anything but Ollama. Even APIs.
minus-squareSuspiciousCarrot78@aussie.zoneOPlinkfedilinkEnglisharrow-up10·19 days agoLlama.cpp or death!
minus-squaretristynalxander@mander.xyzlinkfedilinkEnglisharrow-up4·19 days agoIt’s not that hard to use llama.cpp directly anyway. Why would I use a wrapper when I can just run a python script?
minus-squareBlackLaZoR@lemmy.worldlinkfedilinkEnglisharrow-up2·18 days agoI use LMStudio, because it has quality of life improvements like nice GUI and huggingface search engine. Also they have Vulkan backend that at least on 7900XTX is ~10% faster than rocm (on LLama 3 8b Q4_0 it gets 115Tokens/s vs 105 on rocm)
minus-squarebrucethemoose@lemmy.worldlinkfedilinkEnglisharrow-up3·19 days agoOr exllama! Vllm, sglang, Lorax. Koboldcpp, Aphrodite, text-generation-webui, LM Studio, powerinfer, ktransformers, mlc-LLM, really whatever floats your boat. Just not ollama, specifically.
minus-squarevagabond@lemmy.dbzer0.comlinkfedilinkEnglisharrow-up4·18 days agoDidn’t know this. Going to switch this weekend, thanks for sharing this!
An aside for anyone reading this:
https://sleepingrobots.com/dreams/stop-using-ollama/
And that barely scratches the surface. Please.
Use anything but Ollama. Even APIs.
Llama.cpp or death!
It’s not that hard to use
llama.cppdirectly anyway. Why would I use a wrapper when I can just run a python script?I use LMStudio, because it has quality of life improvements like nice GUI and huggingface search engine. Also they have Vulkan backend that at least on 7900XTX is ~10% faster than rocm (on LLama 3 8b Q4_0 it gets 115Tokens/s vs 105 on rocm)
Or exllama! Vllm, sglang, Lorax. Koboldcpp, Aphrodite, text-generation-webui, LM Studio, powerinfer, ktransformers, mlc-LLM, really whatever floats your boat. Just not ollama, specifically.
Didn’t know this. Going to switch this weekend, thanks for sharing this!
thank you