Amazonbot now respects robots.txt, ending years of webmaster complaints

As of May 14, 2026, Amazonbot began respecting robots.txt rules, allowing website owners to block its crawls [xeiaso]. For years, webmasters complained that Amazon’s crawler ignored these standard directives, often overloading servers with requests to disallowed pages.

The robots.txt protocol, defined by the W3C, lets site operators declare which paths crawlers should avoid. Googlebot and other major bots have followed it for over a decade [Google Developers]. Amazonbot’s adoption brings it in line with industry norms—belatedly.

This change matters for three concrete reasons. First, server load: sites can now block Amazonbot from crawling high-cost endpoints like search or API routes, reducing infrastructure strain. Second, control: webmasters can protect non-public sections—such as staging environments or admin panels—from being scanned. Third, parity: with Amazonbot complying, the major holdout in ethical crawling is gone.

One catch: Amazon has not published a formal statement or documentation update confirming the shift. The change was verified by independent tests showing Amazonbot honoring Disallow rules it previously ignored [xeiaso].

The delay raises questions. While Amazon joins Google, Bing, and others in respecting decades-old web standards, its late adoption suggests a pattern: the company builds first, conforms later. That approach works internally—but when it impacts third-party infrastructure, it shifts costs onto others. Now that Amazonbot obeys robots.txt, the baseline expectation for responsible crawling is met. But webmasters shouldn’t need to wait years to enforce basic boundaries.