How a post-hack crawl restriction can accidentally block an XML sitemap
When was the last time you audited your website’s robots.txt file? For many marketers, it is a set-it-and-forget-it technical detail. But a single line can inadvertently keep Google from retrieving an important resource, even when that resource opens normally in your browser. We were helping a partner figure out why one of their VIP XML Sitemaps was unreachable by GSC. They had put up several security measures in place in response to a website hack. This was the culprit and created a lingering crawlability issue. Pro tips below on why verification matters as much as the fix.
What Is Robots.txt, and Why Does It Matter?
A robots.txt file sits at the root of a website and tells compliant crawlers which paths they may request. You can normally view it at yourdomain.com/robots.txt. It is useful for managing crawler access to areas such as administrative paths, but the way each rule matches URLs matters. Robots.txt is a crawling control, not an indexing guarantee. A disallowed URL can still be discovered and sometimes indexed through links. By contrast, a noindex directive tells a search engine not to index a resource when it can actually retrieve that directive. Robots.txt should never be treated as a security control for private content.
An Illustrative Audit: A Sitemap That Looked Fine but Could Not Be Read
Imagine auditing a prospective client’s WordPress site after an earlier compromise. The owner had taken steps to contain unwanted URLs created during the incident. Months later, Google Search Console showed “Sitemap could not be read – General HTTP error” for a child sitemap at /page-sitemap.xml. In a browser, that XML sitemap displayed a normal list of pages. The main sitemap index was also accessible. Yet Search Console’s last successful read date for the child sitemap was months old, and they had just relaunched their site! A live inspection exposed the critical clue: the inspection crawler reported “Blocked by robots.txt.”
| Sitemap Diagnostic | Illustrative finding |
| Sitemap index | Accessible and recently read |
| Page sitemap | Displays page URLs in a browser |
| GSC Sitemaps report | General HTTP error; older last successful read |
| GSC live inspection (before fix) | Crawl blocked by robots.txt; page fetch unsuccessful |
A browser’s ability to display XML is not evidence that Google can retrieve it. A sitemap may be blocked for crawlers, intermittently unavailable, or served differently by a firewall, CDN, or web server.
Why Was the Rule Added? A Previous Website Hack
In this scenario, the restrictive rule originated in an earlier security response. The site had been compromised, creating unwanted URLs. To limit crawler access to those paths, a broad robots.txt directive was introduced. The intention was understandable, but the long-term effect was not fully reviewed.
User-agent: *
Disallow: /page
The rule is not a noindex instruction. It tells compliant crawlers not to request paths beginning with /page. That distinction is important after a hack: robots.txt alone cannot reliably remove hacked URLs from Google’s index and may prevent crawlers from seeing a noindex directive placed on those pages. A sound post-hack recovery plan includes eliminating malicious content, fixing the vulnerability, using appropriate HTTP status codes for removed URLs, considering temporary removals in Search Console where warranted, and reviewing any temporary crawl restrictions afterward.
The Hidden Problem: Prefix Matching
Without an end-of-URL anchor, the rule Disallow: /page can match more than a page called /page. For the relevant user-agent group, it also matches paths such as:
- /page/2/
- /page-example/
- /page-sitemap.xml
That final match is the problem. A crawl rule aimed at unwanted paths accidentally includes the legitimate XML sitemap. The result is a misleading situation: humans can open the sitemap, but the crawler’s access test is blocked. The fix is to remove or narrow the unintended broad rule, not to blindly delete every Disallow directive. Review user-agent groups, wildcards, and more-specific rules before editing.
The Troubleshooting Sequence: WordPress, Cache, and Hosting
1. Check which robots.txt file is actually being served
A WordPress site may expose a virtual robots.txt generated by WordPress or plugins, while a physical robots.txt in the site root can take precedence. Before changing settings, determine which source controls the public URL.
2. Don’t assume an SEO plugin edit reached the public file
In the scenario, the expected Yoast SEO File Editor was not available. That can reflect permissions or configuration; it does not by itself prove why the file was blocked. The team needed to investigate the server-side file rather than assume a plugin setting had updated the live response.
3. Clear relevant caches, but don’t keep purging blindly
The site used LiteSpeed Cache. A purge was reasonable as an initial test, but the old robots.txt directive persisted. That shifted attention from ordinary page caching toward the hosting file location and other delivery layers.
4. Correct the file through the hosting environment
After locating the relevant file through the hosting provider, the unwanted /page directive was removed while preserving the other intended rules. The public robots.txt URL was checked again to confirm that the line was gone. In a real audit, back up the original file and avoid replacing unrelated restrictions.
Why GSC May Still Show the Old Error After the Fix
Google caches robots.txt, generally for up to 24 hours, and sometimes longer when retrieval fails. Even when a corrected file is visible to you, Google may still be working from a previously fetched version. Search Console’s historical Sitemaps status also does not necessarily update at the instant the site-side correction is made. In Search Console, open Settings → Crawling → robots.txt, check the last fetch date and the content Google retrieved, and request a recrawl if needed. Once the new rules are available, run a fresh live URL inspection of the affected sitemap.
The Breakthrough: Successful Fetch, but a Noindex Message
After the robots.txt restriction was corrected, a subsequent live inspection reported “Page fetch: Successful.” That is meaningful confirmation that the Inspection Tool can now retrieve the sitemap. It is not, by itself, proof that the background Sitemaps report has completed a new successful read.
The same inspection may show “No: noindex detected in X-Robots-Tag HTTP header” under indexing allowed. That is not automatically an XML sitemap problem. Yoast commonly attaches a noindex header to XML sitemaps because the sitemap URL itself does not need to rank in search results; Google can still use a successfully fetched sitemap to discover its listed URLs.
| GSC Result | How to interpret it |
| Crawl allowed: Yes | The inspection crawler is not blocked by robots.txt |
| Page fetch: Successful | The live inspection retrieved the sitemap |
| Indexing allowed: No (X-Robots-Tag noindex) | Often expected for the sitemap URL itself |
| Sitemaps report still shows an older error | Monitor for a new successful background sitemap read |
Do not remove a normal sitemap noindex header merely to make every URL Inspection field say “Yes.” The appropriate goal is for the sitemap to be retrievable and processed, not for the sitemap file itself to appear in search results.
How to Audit Robots.txt After a Security Incident
- Open the live /robots.txt file and preserve a copy of its current directives.
- Review every User-agent, Allow, and Disallow rule for unintended prefix or wildcard matches.
- Check whether sitemap files, important pages, and rendering resources are crawlable.
- Compare your sitemap index and child sitemaps in Search Console, including statuses and last successful read dates.
- Use live URL Inspection as a crawl/fetch diagnostic; use the Sitemaps report for sitemap-processing confirmation.
- Identify the serving layer—physical file, WordPress, SEO plugin, host, CDN, or security system—before editing.
- After a correction, confirm the public robots.txt changed, review Google’s fetched copy, and monitor a new successful sitemap read.
What This Means for AI Search Visibility
Crawlability also matters when teams evaluate AI search visibility. Google’s AI search experiences depend on its wider search systems, while other AI services have separate crawlers and different purposes. Review access policies intentionally rather than allowing or blocking every AI crawler indiscriminately. Allowing a crawler does not guarantee that a brand will appear in an AI answer or earn a citation. But accidentally restricting important content can limit discovery and undermines the technical foundation of both SEO and AI search optimization.
The Bigger Lesson: Security Fixes Need SEO Follow-Up
A website hack is a security incident first. But the emergency changes made during recovery can continue affecting search visibility long after the immediate threat is removed. Broad crawl restrictions deserve particular attention because they can match resources that were never intended to be blocked.
For site owners and marketing teams, the process should have three distinct milestones: identify the cause, implement the correction, and verify the outcome. The final verification is especially important when Google’s live inspection and its background reporting update at different times.
At Simple Search Marketing, technical SEO audits look beyond what a page displays in a browser. We examine crawler access, XML sitemaps, indexing directives, hosting behavior, and the signals that can affect organic and AI search visibility. If you are recovering from a website compromise—or simply have not reviewed robots.txt recently—a post-recovery technical SEO audit can reveal overlooked restrictions before they undermine your search efforts.
Contact Simple Search Marketing to discuss a technical SEO audit.
Source Notes
Google robots.txt specification: https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec
Google sitemap troubleshooting: https://support.google.com/webmasters/answer/7451001
Google Search Console robots.txt report: https://support.google.com/webmasters/answer/6062598