# Allow all AI crawlers to access all pages except paginated variants User-agent: * Allow: / Disallow: /*_page=