technology
technology neo 5mo ago 0%

Should Hexbear have a more robust robots.txt?

Consider https://arstechnica.com/robots.txt or https://www.nytimes.com/robots.txt and how they block all the stupid AI models from being able to scrape for free.

0
2
Comments 2