Discover
See the current public robots and sitemap state.
Crawler instruction check
Two small files can decide what search engines see.
Check whether robots.txt or sitemap URLs are blocking indexing, pointing crawlers to an unexpected host, or exposing suspicious spam-looking paths.
Try it now
Enter a public domain
Running outside-in integrity check
Following redirects, reading public signals, and checking safe exposure paths.
HTTP & redirects
SEO spam signals
SSL, headers, robots, sitemap
Public domains only. Internal IPs, localhost, custom ports, and unsupported schemes are rejected.
robots.txt
Disallow rules and sitemap declarations
sitemaps
Common XML locations and suspicious URLs
Crawler control files
A single line can create a search visibility problem.
This checker looks for the public signals that matter most: unavailable robots.txt, Disallow: /, external sitemap declarations, external URLs, and spam-looking sitemap entries.
robots.txt
User-agent: * Disallow: / Sitemap: https://unknown-host.test/sitemap.xml
sitemap.xml
/normal-page /brand-wallet-replica https://external-host.test/page
One-time check vs monitoring
The first check is a snapshot. The risk is the next change.
Understand
Find external hosts, broad disallow rules, and spam terms.
Monitor
Create a baseline and get alerts when these public files change.
Keep robots.txt and sitemaps under watch.
Ambastly tracks these public files as part of external website integrity monitoring.
Why it matters
Crawler files are easy to ignore until traffic changes.
A broad disallow rule can hide pages from search. A compromised sitemap can invite crawlers to spam URLs. An external sitemap declaration can send attention to a host you do not control.
Disallow: /
Can tell compliant crawlers not to index the whole site.
External sitemap host
Can point crawlers away from the expected website.
Suspicious sitemap URL
Can expose spam paths to search engines before humans notice.
Problems this catches
Robots and sitemap problems are often invisible in the browser.
A visitor may never open robots.txt or sitemap.xml, but search engines and crawlers use those files to decide what to discover, ignore, or index.
robots.txt blocking all crawlers
A broad Disallow: / rule can create search visibility issues if it appears unexpectedly.
Sitemap points to another host
An external sitemap declaration can send crawlers away from the expected website.
Sitemap includes spam URLs
Suspicious paths in sitemap XML can expose injected pages to search engines.
Important crawler files changed
A one-time check shows the current state; monitoring helps catch future drift.
Robots and sitemap checker FAQ
Questions people ask when indexing suddenly changes.
What does robots.txt do?
robots.txt gives crawler instructions about which paths should or should not be crawled. A sudden broad disallow rule can affect search visibility.
Why check sitemap.xml?
Sitemaps help crawlers discover URLs. If a sitemap contains spam URLs, external hosts, or unexpected paths, those signals can create search reputation problems.
Can robots.txt or sitemap changes indicate compromise?
They can be publicly visible compromise signals, especially when they point to external hosts, include spam terms, or differ from the expected website state.
Why monitor robots.txt and sitemaps continuously?
These files can change after a deployment, CMS plugin issue, or compromise. Continuous monitoring helps detect suspicious changes instead of relying on occasional manual checks.