1
0
mirror of https://github.com/funamitech/mastodon synced 2024-12-13 22:29:07 +09:00
YuruToot/public/robots.txt
nightpool a5992e5883 Change robots.txt to exclude only media proxy URLs (#10038)
* Revert "Change robots.txt to exclude some URLs (#10037)"

This reverts commit 80161f4351.

* Let's block media_proxy

/media_proxy/ is a dynamic route used for requesting uncached media, so it's
probably bad to let crawlers use it

* misleading comment
2019-02-14 03:11:47 +01:00

5 lines
137 B
Plaintext

# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
User-agent: *
Disallow: /media_proxy/