Andy Reid to Technology@lemmy.worldEnglish • 1 year agoAI companies are violating a basic social contract of the web and and ignoring robots.txtwww.theverge.comexternal-linkmessage-square101fedilinkarrow-up1860arrow-down114cross-posted to: technology@beehaw.org
arrow-up1846arrow-down1external-linkAI companies are violating a basic social contract of the web and and ignoring robots.txtwww.theverge.comAndy Reid to Technology@lemmy.worldEnglish • 1 year agomessage-square101fedilinkcross-posted to: technology@beehaw.org
minus-squareKillingTimeItselflinkfedilinkEnglish0•1 year agohmm, i though websites just blocked crawler traffic directly? I know one site in particular has rules about it, and will even go so far as to ban you permanently if you continually ignore them.
minus-square@Bogasse@lemmy.mllinkfedilinkEnglish1•1 year agoDetecting crawlers can be easier said than done 🙁
minus-squareKillingTimeItselflinkfedilinkEnglish-1•1 year agoi mean yeah, but at a certain point you just have to accept that it’s going to be crawled. The obviously negligent ones are easy to block.
hmm, i though websites just blocked crawler traffic directly? I know one site in particular has rules about it, and will even go so far as to ban you permanently if you continually ignore them.
Detecting crawlers can be easier said than done 🙁
i mean yeah, but at a certain point you just have to accept that it’s going to be crawled. The obviously negligent ones are easy to block.