Programming

There's hardly any cost to a bot operator, malicious , opportunistic or legitimate, to hit your end-point, so once they found a reason to hit it, hitting it a million more times costs cents.

Operators like Meta seem to make it a sport, trying to hit you with multiple parallel requests from multiple sources, across both IPv4 and IPv6 simultaneously, resulting in an effective DDoS for small and medium end point owners and increasing costs significantly for anyone trying fruitlessly to stay ahead of their onslaught.

The malicious traffic by contrast, attempts to sneak in a request with dynamic rate throttling as part of their attempts to stay hidden.

Between these two extremes are the opportunistic operators who hit the same 404 endpoint day after day, hour after hour, minute by minute, for weeks with specific blocks the only remedy.

There are plenty of legitimate bots that quietly go about their business, hitting you every couple of seconds, leaving you alone for long stretches, incrementally crawling, honouring the robots.txt file and generally acting the way a considerate adult might. They've been getting lower and lower in numbers over the years.

Source: I have logs.