# Hamster welcomes search engines, AI crawlers, and archivers.
# Please feel free to crawl and index this site; there is no paywall on public pages.
# The marketing pages load their JS and CSS from /_site/_next/, which stays
# crawlable so Googlebot can render them; those responses carry
# X-Robots-Tag: noindex (next.config.ts), which keeps the chunks out of the index.
# /_next/ on this host is Studio's. Studio sends no such header, so it stays
# disallowed until Studio or the proxy sends one.
# Every /methods and /skills page offers its pack as a zip. The route builds
# the archive on each request and the file is not a page, so crawling them
# costs the site work and Google nothing: 175 of these URLs were 404s in
# Search Console during the September outage and 44 more sit in "crawled,
# currently not indexed". The links carry rel="nofollow" as well.
User-agent: *
Allow: /
Disallow: /_next/
Disallow: /_site/api/download/

Sitemap: https://tryhamster.com/sitemap.xml