@Hubi@feddit.de to Reddit@lemmy.worldEnglish • 1 year agoReddit shares plunge almost 25% in two days, finish the week below first day closewww.cnbc.comexternal-linkmessage-square127fedilinkarrow-up1726arrow-down115cross-posted to: reddit@lemmy.ml
arrow-up1711arrow-down1external-linkReddit shares plunge almost 25% in two days, finish the week below first day closewww.cnbc.com@Hubi@feddit.de to Reddit@lemmy.worldEnglish • 1 year agomessage-square127fedilinkcross-posted to: reddit@lemmy.ml
minus-square@SacrificedBeans@lemmy.worldlinkfedilink11•1 year agoRobots.txt is a compliance standard afaik, more like an advisory guide. It cannot enforce bans on scrapers, it sets some polite boundaries for the website.
Wasn’t there robots.txt at least?
Robots.txt is a compliance standard afaik, more like an advisory guide. It cannot enforce bans on scrapers, it sets some polite boundaries for the website.
There was, and still is