seolounge
There is concern about the lack of an easy way to opt out of having one’s content used to train large language models (LLMs) like ChatGPT. There is a way to do it, but it’s neither straightforward nor guaranteed to work.
How AIs Learn From Your Content
Large Language Models (LLMs) are trained on data that originates from multiple sources. Many of these datasets are open source and are freely used for training AIs.
Some of the sources used are:
- Wikipedia
- Government court records
- Books
- Emails
- Crawled websites
There are actually portals and websites offering datasets that are giving away vast amounts of information.
One of the portals is hosted by Amazon, offering thousands of datasets at the Registry of Open Data on AWS.
Screenshot from Amazon, January 2023The Amazon portal with thousands of datasets is just one portal out of many others that contain more datasets.
Wikipedia lists 28 portals for downloading datasets, including the Google Dataset and the Hugging Face portals for finding thousands of datasets.
Datasets of Web Content
OpenWebText
A popular dataset of web content is called OpenWebText. OpenWebText consists of URLs found on Reddit posts that had at least three upvotes.
The idea is that these URLs are trustworthy and will contain quality content. I couldn’t find information about a user agent for their crawler, maybe it’s just identified as Python, I’m not sure.
Nevertheless, we do know that if your site is linked from Reddit with at least three upvotes then there’s a good chance that your site is in the OpenWebText dataset.
More information about OpenWebText is here.
Common Crawl
One of the most commonly used datasets for Internet content is offered by a non-profit organization called Common Crawl.
Common Crawl data comes from a bot that crawls the entire Internet.
The data is downloaded by organizations wishing to use the data and then cleaned of spammy sites, etc.
The name of the Common Crawl bot is, CCBot.
CCBot obeys the robots.txt protocol so it is possible to block Common Crawl with Robots.txt and prevent your website data from making it into another dataset.
However, if your site has already been crawled then it’s likely already included in multiple datasets.
Nevertheless, by blocking Common Crawl it’s possible to opt out your website content from being included in new datasets sourced from newer Common Crawl data.
The CCBot User-Agent string is:
CCBot/2.0
Add the following to your robots.txt file to block the Common Crawl bot:
User-agent: CCBot Disallow: /
An additional way to confirm if a CCBot user agent is legit is that it crawls from Amazon AWS IP addresses.
CCBot also obeys the nofollow robots meta tag directives.
Use this in your robots meta tag:
<meta name="robots" content="nofollow">
Blocking AI From Using Your Content
Search engines allow websites to opt out of being crawled. Common Crawl also allows opting out. But there is currently no way to remove one’s website content from existing datasets.
Furthermore, research scientists don’t seem to offer website publishers a way to opt out of being crawled.
The article, Is ChatGPT Use Of Web Content Fair? explores the topic of whether it’s even ethical to use website data without permission or a way to opt out.
Many publishers may appreciate it if in the near future, they are given more say on how their content is used, especially by AI products like ChatGPT.
Whether that will happen is unknown at this time.
More resources:
Featured image by Shutterstock/ViDI Studio