Step-by-Step Guide: Add llms.txt to WordPress or cPanel

As AI models like GPT, Gemini, and Claude increasingly crawl websites for training data, site owners now have a way to control how their content interacts with these systems—through a file called llms.txt. Just like robots.txt helps you manage search engine crawlers, llms.txt serves as your site’s policy file for Large Language Models (LLMs).

Adding an llms.txt file to your website gives you control over whether AI systems can read, use, or ignore your content. This step has become essential for privacy, data management, and brand safety in the age of generative AI.

What is llms.txt?

llms.txt (short for Large Language Model System text) is a plain text file placed in your site’s root directory (e.g., https://yourdomain.com/llms.txt). It communicates your preferences to AI crawlers, indicating whether they can use your site’s data for training or indexing.

For example:

User-agent: GPTBot
Disallow: /

This means OpenAI’s GPT crawler is not allowed to scrape your site.

The Purpose of llms.txt for AI Web Crawlers and Search Engines

This file ensures:

• Data control: Decide what AI bots can access.
• Transparency: Clarify your data usage policies.
• Compliance: Stay aligned with emerging AI ethics and legal standards.
• SEO balance: Maintain visibility while protecting intellectual content.

Understanding How llms.txt Works

llms.txt vs robots.txt: What’s the Difference?

Feature	robots.txt	llms.txt
Purpose	Controls traditional web crawlers (e.g., Googlebot)	Controls AI/LLM crawlers (e.g., GPTBot)
Placement	Root directory (/robots.txt)	Root directory (/llms.txt)
Impact	Affects indexing & SEO	Affects AI data training & content use

Structure and Syntax of a Proper llms.txt File

A standard llms.txt follows a similar format to robots.txt:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Allow: /

Each section specifies an AI crawler and your permission for it.

Preparing to Add llms.txt to WordPress or cPanel

Before proceeding, ensure you have:

• Access to your WordPress admin dashboard or cPanel.
• A basic text editor (e.g., Notepad, VS Code).
• File management permissions.

Recommended Tools

• File Manager (in cPanel)
• WP File Manager Plugin
• FTP Client (like FileZilla)
• Online llms.txt validators

How to Add llms.txt to WordPress (Step-by-Step)

Step 1: Create the llms.txt File

Open a text editor and create a simple file named llms.txt. Add your preferred crawler rules, such as:

User-agent: GPTBot
Disallow: /

Step 2: Upload the File Using WordPress Plugins

• Install and activate WP File Manager.
• Navigate to /public_html/ or your site’s root folder.
• Upload llms.txt directly into that directory.

Step 3: Verify File Location

Go to https://yourdomain.com/llms.txt. If it loads properly, the file has been added successfully.

Step 4: Test Using an Online Validator

Use tools like https://llmstxt.org/tester to verify that your rules are correctly formatted and accessible.

How to Add llms.txt Using cPanel (Step-by-Step)

Step 1: Access File Manager in cPanel

Login to your hosting account and open File Manager.

Step 2: Upload llms.txt to the Root Directory

• Locate your public_html folder.
• Upload llms.txt here.
  (This is the same directory as your index.html or wp-config.php.)

Step 3: Check File Permissions

Ensure file permissions are set to 644 (readable by everyone).

Step 4: Validate Configuration

Access https://yourdomain.com/llms.txt in your browser.
If you can see the file, your configuration is correct.

Best Practices for Configuring llms.txt

• Keep syntax clean (no typos or formatting errors).
• Always use User-agent and Allow/Disallow properly.
• Don’t block crawlers unnecessarily—only those you distrust.
• Update your llms.txt regularly as AI bots evolve.

How to Test and Monitor llms.txt Functionality

You can use tools like:

• llmstxt.org/tester
• Google Search Console’s URL Inspection
• Curl commands (curl https://yourdomain.com/llms.txt)

Regularly reviewing your file ensures you maintain the right data-sharing balance.

Troubleshooting Common llms.txt Issues

llms.txt Not Found Error

Ensure the file is in the root directory, not inside a subfolder like /wp-content/.

Permissions or File Path Issues

File permissions should be 644. Anything stricter (like 600) may block crawlers from reading the file.

FAQs About Adding llms.txt to WordPress or cPanel

1. Do I need both robots.txt and llms.txt?
Yes. robots.txt controls search engines, while llms.txt controls AI crawlers.

2. Can I block all AI bots?
Yes, use:

User-agent: *
Disallow: /

3. Where should llms.txt be placed?
Always in the root directory (e.g., /public_html/).

4. Will adding llms.txt affect SEO?
No, it only affects AI crawlers—not search engines.

5. How often should I update llms.txt?
Every few months or when new AI bots emerge.

6. What happens if I don’t add llms.txt?
AI crawlers may interpret it as permission to use your content for training.

Conclusion: Keep Your Website AI-Friendly and Future-Ready

Adding llms.txt to WordPress or cPanel is a simple but powerful way to manage how AI interacts with your website. By defining clear policies for AI crawlers, you protect your content, maintain data control, and ensure ethical AI engagement.

For further reading, check out https://llmstxt.org — a reliable resource for managing AI crawler settings.