hylobates@jlai.lu to Selfhosted@lemmy.worldEnglish · 20 hours agoBased on this graph, and this graph alone, guess at what time I completely blocked OpenAI crawlersjlai.luexternal-linkmessage-square52fedilinkarrow-up1426arrow-down15file-text
arrow-up1421arrow-down1external-linkBased on this graph, and this graph alone, guess at what time I completely blocked OpenAI crawlersjlai.luhylobates@jlai.lu to Selfhosted@lemmy.worldEnglish · 20 hours agomessage-square52fedilinkfile-text
minus-squareAHemlocksLie@lemmy.ziplinkfedilinkEnglisharrow-up5·3 hours agoPretty sure I’ve repeatedly heard about the crawlers completely ignoring robots.txt, so does Cloudflare really do that much?
minus-squareSv443@sh.itjust.workslinkfedilinkEnglisharrow-up1·10 minutes agoLike a lock on a door, it stops the vast majority but can’t do shit about the actual professional bad guys
Pretty sure I’ve repeatedly heard about the crawlers completely ignoring robots.txt, so does Cloudflare really do that much?
Like a lock on a door, it stops the vast majority but can’t do shit about the actual professional bad guys