Programming

Info for anyone reading, while the read was quite interesting, the whole article turned out to be an ad.

It’s really not surprising that it’s so fast, since you can easily get newly created repos and repos made public from a github API (the “list public events” one at /events). Makes sense that people are polling this and feeding it to TruffleHog.

There's hardly any cost to a bot operator, malicious , opportunistic or legitimate, to hit your end-point, so once they found a reason to hit it, hitting it a million more times costs cents.

Operators like Meta seem to make it a sport, trying to hit you with multiple parallel requests from multiple sources, across both IPv4 and IPv6 simultaneously, resulting in an effective DDoS for small and medium end point owners and increasing costs significantly for anyone trying fruitlessly to stay ahead of their onslaught.

The malicious traffic by contrast, attempts to sneak in a request with dynamic rate throttling as part of their attempts to stay hidden.

Between these two extremes are the opportunistic operators who hit the same 404 endpoint day after day, hour after hour, minute by minute, for weeks with specific blocks the only remedy.

There are plenty of legitimate bots that quietly go about their business, hitting you every couple of seconds, leaving you alone for long stretches, incrementally crawling, honouring the robots.txt file and generally acting the way a considerate adult might. They've been getting lower and lower in numbers over the years.

Source: I have logs.

There are criminal but professsional groups with million-$ budgets out there.