Tag: adidas robots txt

nike.com robots.txt — just crawl it.: ProgrammerHumor – Reddit Press J to jump to the feed. Press question mark to learn the rest of the keyboard shortcutsFound the internet! Log in or sign up to leave a comment level 1My favorite easter egg went something likeUser-agent: * Disallow: /harm/human Disallow: […]