cross-posted from: https://discuss.online/post/32165111
I realize my options are limited, but what about any robots.txt style steps? Thanks for any suggestions.
cross-posted from: https://discuss.online/post/32165111
I realize my options are limited, but what about any robots.txt style steps? Thanks for any suggestions.
A big issue is that this works for bots that announce themselves as such, but there’s lots that pretend to be regular users, with fake user agents and ips selected from a random pool with each ip only sending like 1-3 request/day, but overall many thousands of requests. In my experience a lot of them are from huawei and tencent cloud/ASN
Yes, if that is true (and I am not that suprised about it) it is nearly impossible to block them this way.