Anubis vor SearXNG: Bots mit Proof of Work ausbremsen
Meine Entdeckung des Jahres um
ungewollten Traffic von Bots und Crawlern von einer Webseite herauszufiltern. Anubis
#Anubis #Self_Hosting #Reverse_Proxy #Linux
anubis
Follow from the Fediverse
Follow #anubis from Mastodon or any other Fediverse app and its public posts arrive in your timeline. No vutuv account needed.
@anubis@tags.vutuv.de
Posts with this tag
2 posts
Filters
We tweaked our firewalls and our Anubis rules. The attack persisted, but it was contained.
We also scaled our systems to avoid more accidental downtimes caused by these scrapping operations… Then the attackers scaled up their scrapping campaign, and found new ways to bypass our #Anubis config.
This past night I managed to come up with a stricter configuration that blocks virtually all of them (I don’t know for how long this will last, though).
As of now, our Anubis instance is stopping around 40 requests per second.
I know that this is “nothing” for anyone working in very big systems, but bear in mind that this is after I already blocked around 3.5% of all the IPv4 addresses existing, and it accounts for 95% of our traffic. For every legitimate request we have, we have 19 that come from bots.
P.S.: In an ideal world, I would openly share the rules I found to work well… Unfortunately I don’t think that would be a good idea. I don’t want to make the scrappers’ job any easier.
