Experiencing intermittent 503 errors from lemmy world using from photon frontend.
These urls return 503: https://lemmy.world/api/v3/site? https://lemmy.world/api/v3/user/mention?limit=20&page=1&unread_only=false&sort=New https://lemmy.world/api/v3/private_message/list?limit=20&page=1&unread_only=false https://lemmy.world/api/v3/user/replies?limit=20&page=1&unread_only=false&sort=New
error body:
{“error”:“Offline”}
Hi,
sorry for the late response to this. The immediate issue is resolved, and we’re looking to extend our monitoring in the coming days to have better insights on the underlying issue, as well as alerts before it would get to the point of throwing errors.
For some reason Lemmy has been having significantly more open file handles (including e.g. connections) than usual, leading to a denial of service. At this point it’s unknown whether this is caused by a bug in Lemmy, an attack, or something else.
Hey thank you for looking into it. Maybe some kernel tuning needed?
Unlikely, we haven’t changed these limits recently and we haven’t exceeded the limits in the past. While it’s possible to change the limits, it’s not clear yet why they are being exceeded currently.
