Cloudflare has just expanded what robots.txt can do. With its new Content Signals Policy, publishers can explicitly signal how AI bots should (or shouldn’t) use their content and not just whether they can crawl it.
Cloudflare’s Content Signals Policy accomplishes that by letting site owners express usage preferences via three new signals:
search: permission for content to appear in search indexes and support standard search features
ai-input: whether AI models can treat your content as input for generating responses or summaries
ai-train: whether your content can be used to train or fine-tune AI models
Signals are declared like: