IETF setting standards for AI preferences

The use of Internet content for training large language models ("AI") has become a contentious topic. Training a model requires voluminous information, and the Internet is readily available as a source of diverse content. However, many publishers and authors object to these new uses of their work without permission or compensation.

While the IETF doesn't take a position on the legal questions in this space, we do track developments, and last September the IAB AI-CONTROL Workshop highlighted a need for clearer communication between content publishers and AI model trainers. 

Right now, AI vendors use a confusing array of non-standard signals in the robots.txt file (defined by RFC 9309) and elsewhere to guide their crawling and training decisions. As a result, authors and publishers lose confidence that their preferences will be adhered to, and resort to measures like blocking their IP addresses.

To address this need, the AIPREF working group has been chartered to define:

1. a common vocabulary to express authors' and publishers' preferences regarding use of their content for AI training and related tasks, and
   
   
2. means of attaching that vocabulary to content on the Internet, either by embedding it in the content or by formats similar to robots.txt, and a standard mechanism to reconcile multiple expressions of preferences. 

The Working Group will hold its first meeting during IETF 122 Bangkok, where administrative and initial scoping discussions will take place. Shortly thereafter, it will hold an interim meeting from 8-10 April in Brussels, Belgium. This is an open meeting, but registration is required.

If you are interested in monitoring or participating in this work, subscribe to the AIPREF working group mailing list.