Adding an optional anubis config by dessalines · Pull Request #285 · LemmyNet/lemmy-ansible
github.com
This adds an optional anubis, which stands in front of lemmy-ui only. It doesn't block or impede federation, or apps, only the lemmy-ui front end. I tested this via a fresh server, and it worked.

I’ve recently added anubis to lemmy.ml, and it seems to be working well.

I have a PR to add anubis to lemmy-ansible (our main installation method), and I could use some help tweaking / optimizing its botPolicy.yaml config, for federated services.

Anyone with experience running anubis, this would be much appreciated.

      • Otter@lemmy.caEnglish
        4·
        2 months ago

        We are not running Anubis, although we do block a large number of AI/LLM companies through IP addresses. Each time we block a new one, it makes a noticeable difference in the performance graphs.

    • poVoq@slrpnk.netEnglish
      6·
      2 months ago

      This is the botPolicy.yaml that we use on slrpnk.net :

      bots:
  - name: known-crawler
    action: CHALLENGE
    expression:
      # https://anubis.techaro.lol/docs/admin/configuration/expressions
      all:
        # Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36
        - userAgent.contains("Macintosh; Intel Mac") && userAgent.contains("Chrome/125.0.0.0") # very old chrome?
        - missingHeader(headers, "Sec-Ch-Ua") # a valid chrome has this header
    challenge:
      difficulty: 6
      algorithm: slow

    # Assert behaviour that only genuine browsers display.
    # This ensures that Chrome or Firefox versions
  - name: realistic-browser-catchall
    expression:
      all:
        - '"User-Agent" in headers'
        - '( userAgent.contains("Firefox") ) || ( userAgent.contains("Chrome") ) || ( userAgent.contains("Safari") )'
        - '"Accept" in headers'
        - '"Sec-Fetch-Dest" in headers'
        - '"Sec-Fetch-Mode" in headers'
        - '"Sec-Fetch-Site" in headers'
        - '"Accept-Encoding" in headers'
        - '( headers["Accept-Encoding"].contains("zstd") || headers["Accept-Encoding"].contains("br") )'
        - '"Accept-Language" in headers'
    action: CHALLENGE
    challenge:
      difficulty: 2
      algorithm: fast

  - name: generic-browser
    user_agent_regex: (?i:mozilla|opera)
    action: CHALLENGE
    challenge:
      difficulty: 4
      algorithm: fast

status_codes:
  CHALLENGE: 202
  DENY: 406

dnsbl: false

#store:
#  backend: valkey
#  parameters:
#    url: redis://valkey-primary:6379/0

      I think I just took it over from Codeberg.org back from when they still used Anubis. Nothing really relevant to Lemmy specifically and it is only in front of the frontends, not the s2s federation API.

      It seems though like there are some crawlers that use 3rd party hosted alternative frontends to crawl (unintentionally?) through the federation API, so something in front of that would be useful I guess.

  • olof@lemmy.mlEnglish
    2·
    2 months ago

    Not Lemmy specific, but I wanted to set up Anubis in a setup where I have one reverse proxy (nginx) handling many different domains. Last time I looked, it seemed to need one Anubis instance per domain. Is that still the case? Goal was to have a single Anubis instance and route all through it

    • Dessalines@lemmy.mlOPEnglish
      2·
      2 months ago

      I’m not an expert, but I think the fact that you need to set a TARGET in anubis, IE, where does anubis send you after passing it, means that you do need separate anubis’s for each site.

    • poVoq@slrpnk.netEnglish
      2·
      2 months ago

      You could probably put Anubis in front of your reverse-proxy, but then you need something else in front of it that handles TLS certificates. So maybe something like this: HAProxy->Anubis->Nginx.