When AI Crawls Your Website: robots.txt and Terms of Use as Legal Tools

Your website is a business asset, and AI companies are hungry for the content on it. Whether you want your pages used to train models, surfaced in AI search, or left alone, the tools that express and enforce that choice are partly technical and partly legal, and the law around them is unsettled and actively litigated.

Start with robots.txt. It is a plain text file at the root of your domain that tells automated crawlers which parts of the site they may access. It is the standard signal, and reputable operators honor it, but it rests on voluntary compliance and is not, by itself, a clearly binding contract. Its legal significance has historically been evidentiary: in Field v. Google, Inc., 412 F. Supp. 2d 1106 (D. Nev. 2006), a court found an implied license where a site owner who could have blocked automated copying chose not to. The lesson from that line of authority is that silence can be read as permission, which is why an intentional, well-configured robots.txt matters.

The new wave of AI-scraping litigation is testing whether robots.txt can do more. Recent suits by publishers and platforms against AI developers press the argument that ignoring robots.txt, especially when paired with a site’s terms of use, defeats any claim of an implied license and can support DMCA-circumvention and breach-of-contract claims. Courts have not settled these questions, so treat robots.txt as an important signal and evidence-builder, not an ironclad barrier.

That is why terms of use carry the contractual weight robots.txt lacks. A well-drafted terms of use can prohibit automated scraping and AI training on your content, condition access on acceptance, and give you a breach-of-contract claim that does not depend on copyright. Enforceability turns on presentation: in Nguyen v. Barnes & Noble, Inc., 763 F.3d 1171 (9th Cir. 2014), the court refused to enforce passive “browsewrap” terms offered by hyperlink alone, without reasonable notice or affirmative assent. Placement and notice matter.

Copyright remains the backstop for original content, though its application to AI training is exactly what the courts are now fighting over.

Practical steps for a site owner. Configure robots.txt deliberately, deciding agent by agent who may crawl, and remember that the default of silence tends to read as permission. Put real restrictions in your terms of use, state expressly that scraping and AI training are prohibited, and present the terms so acceptance is clear rather than buried. Layer the two, because the technical signal and the contractual prohibition reinforce each other, which is exactly the combination the current lawsuits are built on. And preserve evidence: if a crawler ignores your robots.txt and terms, log it.

None of this guarantees an outcome in a fast-moving area. But site owners who set clear signals and clear terms are in a far stronger position than those who did nothing.

Legal note: This article provides general information and is not legal advice.