Do you host your own ML / AI / LLM? What do you use, and what do you use it for?

  • algernon@lemmy.mlEnglish
    7012·
    17 days ago

    Yes. My Actual Intelligence lives in my head, and runs mostly on coffee.

  • D_Air1@lemmy.mlEnglish
    19·
    17 days ago

    Yeah, I’m using qwen 31b a3b on an amd 9070xt requires a bit of cpu offloading, but still plenty fast. Using it wall llama.cpp. Combine that with some mcp’s such as ddg-search to make it truly useful by actually being able to search online.

    I mostly use it for small tedious tasks with well defined inputs and outputs. For example when hyprland recently changed from their own configuration language to lua. At first I started going line by line translating my config to the new lua language until I realized oh wait this is exactly the type of thing that ML is useful for. Going from the well defined hyprland configuration language to their also well defined lua syntax. It banged it out in less than a minute with only a single mistake which I easily fixed. The mistake it made was that it forgot to translate the comments to lua. It did it in less than a minute and worked first try. Where as I had made several typos and gotten a few lines wrong when I was doing it by hand.

    Not to say that I couldn’t do it. I would have gotten it done in about half an hour, but less than a minute is a lot faster.

    I also used it to transform a bunch of unstructured data into json data, so that I could then use purpose built tools like jq to parse that. If I’m having trouble finding certain information. I’ll ask it to find me some resources to look at.

    Basically small well defined tasks and parsing data is what I use it for and it seems to be pretty good at that.

    What I don’t like is the way companies try to market it to people. I don’t believe people should be trying to summarize emails or messages from loved ones, writing essays or any other creative tasks for the most part. Translating is okay. I don’t expect a machine to be able to decide things for me or to be some filter between me and others.

  • fluxx@mander.xyzEnglish
    10·
    15 days ago

    I do, but I am becoming increasingly more disappointed as time goes on. Not just self hosted, llms in general. They sometimes help, but they mislead so many times and waste time that you don’t even notice. I think that’s the trap. When you succeed at a task, you become impressed but don’t notice how many times it failed doing a simple task. And as soon as you scratch the surface, you see how you would have done it differently and perhaps in a better way. Even just googling is bad. It does research for you, but it has no critical thinking and can’t decide what is better from the results it gets (other than google ranking) so it often leads you to think it did as good as you would, when it’s nowhere near as good. Every time I did the googling myself after it did, I did it much better. And I mean MUCH better. Ask it to find the app, it misses the most important ones, hallucinates a bunch, for ex. I found this to be the case with frontier models as well.

    Self hosting has its benefits, but seeing how the ecosystem looks right now, concluding this is a huge bubble is inevitable. It reminds me of crypto so much. It looks rich and plentiful, but as soon as you dig a mm under the surface - nobody has tested it, it’s got a critical bug, it is overblown and there are issues with no response. No docs, no info, no nothing. For the biggest thing in technology in history, it is awfully hollow. I don’t mean it in a condescending way, in fact community is enthusiastic and very helpful, it’s just that it doesn’t live up to what most would expect.

    A caveat I need to mention is I have not used it for coding - I have an irrational fear and resistance towards it, being a programmer. I just won’t touch it, even if it means the end of my career. I’m trying to be grown-up about it, but so far, I dont want to use it, for good and bad reasons.

  • Nednarb44@lemmy.worldEnglish
    9·
    17 days ago

    I do, I use ollama. I mostly just tinker, but I use with with home assistant for a quasi Alexa like experience with the voice assistant, I use it for summarizing some YouTube transcripts in too lazy to read/watch, and I’ve tried to see how capable it is with coding.

  • Strider@lemmy.worldEnglish
    101·
    16 days ago

    No. I still have no use for it and everything I use is automated without at a far lower footprint.

  • PetteriPano@lemmy.worldEnglish
    112·
    16 days ago

    Running qwen3.6 27b through llama.cpp.

    It’s about as capable as sonnet 3.5.

    I use it for light scripting, but real coding is done by cloud models.

    I’m also using it as the brain for my Hermes agent. It sends me digests of news, subreddits, chats that I’d like to read but don’t have time for. It does a great job researching things on the web for me, too.

    • SuspiciousCarrot78@aussie.zoneOPEnglish
      11·
      16 days ago

      Do you mean Sonnet 4.5?

      I don’t have the rig to run it at real speeds but I’ve played with it over API. Seems pretty good.

      • PetteriPano@lemmy.worldEnglish
        2·
        15 days ago

        No, it needs a lot more babysitting than 4.5 does. 3.5 was on the same level of mistakes, at least on the quants I have to use.

  • Alexander@sopuli.xyzEnglish
    5·
    17 days ago

    Technically, TTS/STT are mostly MLs; I’m pretty sure many people run these. I have a setup but I’m better with buttons that with spoken words, and I listen to ambient sounds or music. I think some day I’ll make voice assistant for talking to while driving, but that’s not a trivial task hardware-wise, even if I used cloud LLM layer, which I won’t. Putting AI on baremetal sounds like an interesting project.

    I have a homemade “local agent” that can actually “code” somewhat, I use it just to figure out how this thing works on the inside practically. Mostly useless otherwise (also I have GPU that’s older than AI, so it’s kind of fun technical task to run this stuff on pure RAM+swap). Feels like the whole hype is greatly overrated, but I appreciate a chance to learn something new anyway.

        • brucethemoose@lemmy.worldEnglish
          6·
          17 days ago

          Not anymore. Not with hybrid offloading, where the GPU handles dense tensors and the CPU only runs the sparse MoEs. I’m running a 300B model on a single 3090, and its faster than I can read.

          You just need to use the right framework, and the right model.

          I’d suggest trying ik_llama.cpp and a MoE like one of these: https://huggingface.co/models?other=ik_llama.cpp&sort=modified&search=35B

          And speculative decoding like DFlash or MTP (which you can also get specific models for).

          EDIT: Wrong link.

          • atzanteol@sh.itjust.worksEnglish
            31·
            17 days ago

            I’ll check that out - speed isn’t my biggest issue so much as coding performance… The qwen 3.5 model I was using can write code, but it’s… Meh? Like sometimes it doesn’t even compile.

            I did try tweaking llama.cpp to do some cpu offloading and it does seem to allow for much larger contexts at a modest performance loss. I’ll check out larger models.

            • brucethemoose@lemmy.worldEnglish
              3·
              17 days ago

              CPU offloading is too slow unless you use a hybrid MoE model, with the --n-cpu-moe parameter, specifically.

              This only offloads “sparse” parts of the model to the CPU, which take up a lot of RAM but are very compute-lite to run. In practice, thats most of the size of modern MoE LLMs.

              • robber@lemmy.mlEnglish
                1·
                16 days ago

                Since implementation of the --fit parameter and its relatives, and --fit on becoming the default, llama.cpp intelligently decides what to offload. For me, it made --n-cpu-moe obsolete.

                • brucethemoose@lemmy.worldEnglish
                  2·
                  16 days ago

                  Mostly, yeah.

                  Sometimes it’s better to “cut it close,” with (for instance) a 27B model that’s nearly OOMing your VRAM fully offloaded, but you know will be fine in regular use without too many programs open.

                  In my case, with MiMo 2.5, it fills both my CPU and GPU RAM rather completely, so it’s best to set a static value so I don’t swap CPU RAM, and don’t OOM on the GPU either.

  • iceberg314@slrpnk.netEnglish
    5·
    17 days ago

    Ollama with gemma 4 for LLM stuff, coding brainstorming, etc.

    Comfy ui with z-image or stable diffusion for images.

  • superglue@lemmy.dbzer0.comEnglish
    4·
    17 days ago

    I use my gaming rig to serve up qwen3.6-coder to Open Web UI and that’s been very successful in helping me refactor my home lab to be more effecient and easier to support. Over the years of building my server I got everything working, but lets just say it’s a bot of a mess and a lot of shortcuts were taken.

    I plan to look into ComfyUI soon but I do that have much of a use case for it at the moment.

  • ccunning@lemmy.worldEnglish
    4·
    17 days ago

    I’ve got ollama setup with whisper and piper and a HA voice PE, but I honestly haven’t gotten around to configuring much yet. Most notable thing was being able to use the wake word to start a timer, but it was pickier than old Siri about the precise wording.

  • Jakeroxs@sh.itjust.worksEnglish
    4·
    16 days ago

    Yes, llama-swap and I use it for home assistant text-gen notifications, basic coding tasks, etc

    If anyone here self-hosts definitely check out llama-swap as it has some nifty features for hotswapping LLMs, image generation models and voice models.