hylobates@jlai.lu to Selfhosted@lemmy.worldEnglish · 7 days agoBased on this graph, and this graph alone, guess at what time I completely blocked OpenAI crawlersimagemessage-square79linkfedilinkarrow-up1587arrow-down17file-text
arrow-up1580arrow-down1imageBased on this graph, and this graph alone, guess at what time I completely blocked OpenAI crawlershylobates@jlai.lu to Selfhosted@lemmy.worldEnglish · 7 days agomessage-square79linkfedilinkfile-text
minus-squarequick_snail@feddit.nllinkfedilinkEnglisharrow-up4·7 days agoY’all need to learn to cache things, shiit
minus-squarepoVoq@slrpnk.netlinkfedilinkEnglisharrow-up7arrow-down2·6 days agoThis is not how things work on the modern web. Did you just wake up from a 20 year coma?
minus-squareFreedomAdvocate@lemmy.net.aulinkfedilinkEnglisharrow-up1·edit-28 hours agoWhenever you’re browsing even a semi popular website these days there’s probably a 98% chance you’re hitting a cloudflare cached version of it. Have you been asleep the last 10 years?
minus-squarepoVoq@slrpnk.netlinkfedilinkEnglisharrow-up1·8 hours agoFor static sites, yes. To actually protect dynamic sites against AI crawlers, Cloudflare has to do much more than just caching. And besides that, Cloudflare is a huge single point of failure and highly privacy invasive.
minus-squarehylobates@jlai.luOPlinkfedilinkEnglisharrow-up1·6 days agoThere is cache but for some reason this does not affect every page I serve currently
Y’all need to learn to cache things, shiit
This is not how things work on the modern web. Did you just wake up from a 20 year coma?
Whenever you’re browsing even a semi popular website these days there’s probably a 98% chance you’re hitting a cloudflare cached version of it. Have you been asleep the last 10 years?
For static sites, yes. To actually protect dynamic sites against AI crawlers, Cloudflare has to do much more than just caching.
And besides that, Cloudflare is a huge single point of failure and highly privacy invasive.
There is cache but for some reason this does not affect every page I serve currently
Fix the cache.