I don’t really want companies or anyone else deciding what I’m allowed to see or learn. Are there any AI assistants out there that won’t say “sorry, I can’t talk to you about that” if I mention something modern companies don’t want us to see?

  • Cease@mander.xyz
    13·
    8 months ago

    There’s plenty of open source models that don’t really have any restrictions, you just have to host them yourself (which you can do on your own computer if you have a decent gpu)

    for example: mixtral 8x7b

    just use koboldcpp or something similar to run the GGUF files and you’re good

      • Cease@mander.xyz
        1·
        8 months ago

        I’ve never personally had issues with 8x7b refusing requests, but I guess I haven’t really plumbed the depths on what it might agree or disagree to, I have run it through the ordinary gambit (pretend to be public figure x, make dangerous item y, say untrue thing about company z) and it hasn’t given me any problems but sure whatever works for you

  • SpicyTaint@lemmy.worlddeleted by creatorEnglish
    5·
    8 months ago

    If you have a good enough NVIDIA card, probably a 1080ti or better, download KoboldCPP and a .gguf model from huggingface and run it locally.

    The quality is directly tied to your GPU’s vram size and how big of a model you can load into it, so don’t expect the same results as an LLM running on a data center. For example, I can load a 20gb gguf model into a 3090 with 24gb of vram.

    • Cease@mander.xyz
      2·
      8 months ago

      Actually not 100% true, you can offload a portion of the model into ram to save VRAM to save money on a crazy gpu and still run a decent model, it just takes a bit longer. I personally can wait a minute for a detailed answer instead of needing it in 5 seconds but of course YMMV

      • SpicyTaint@lemmy.worlddeleted by creator
        1·
        8 months ago

        Is there a general term for the setting that offloads the model into RAM? I’d love to be able to load larger models.

        I thought CUDA was supposed to just supposed to treat VRAM and regular RAM as one resource, but that doesn’t seem to be correct.

  • Eyedust@lemmy.dbzer0.comEnglish
    3·
    8 months ago

    You’re probably looking for an abliterated model. Be sure you can run it, first, as localhosting models needs high VRAM. You’ll want RAM, too, in the case of GGUF models.

    I’d have to write a whole half a book here to explain how to use them, but that information is freely available online. If you don’t have a beefy GPU, look into how to host GGUF models.