YuruToot/public/robots.txt

User-Agent: *
Disallow: /users/*/followers
Disallow: /users/*/following
Disallow: /@*/media
Disallow: /@*/with_replies
Disallow: /@*/tagged/*
Disallow: /media_proxy/*
Disallow: /emoji/*
Disallow: /packs/*
Disallow: /sounds/*
Disallow: /system/*
Disallow: /avatars/*
Disallow: /headers/*
Change robots.txt to exclude some URLs (#10037) - Exclude static assets - Exclude uploaded files - Exclude alternate versions of the profile page - Exclude media proxy URLs 2019-02-14 05:28:18 +09:00			`User-Agent: *`
			`Disallow: /users/*/followers`
			`Disallow: /users/*/following`
			`Disallow: /@*/media`
			`Disallow: /@*/with_replies`
			`Disallow: /@/tagged/`
			`Disallow: /media_proxy/*`
			`Disallow: /emoji/*`
			`Disallow: /packs/*`
			`Disallow: /sounds/*`
			`Disallow: /system/*`
			`Disallow: /avatars/*`
			`Disallow: /headers/*`