CloudToys/elk
1
0
Fork 0
Code

fix: syntax for robots.txt disallow rules (#1925)

Browse Source
This commit is contained in:
Baptiste Girardeau 2023-03-26 17:11:22 +02:00 committed by GitHub
parent e062fb5e52
commit 881a49e0d9
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 15 additions and 15 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

30
public/robots.txt
View File

@ -3,21 +3,21 @@ Allow: /
# Disallow authenticated pages
Disallow /intent
Disallow /settings
Disallow /blocks
Disallow /bookmarks
Disallow /compose
Disallow /conversations
Disallow /domain_blocks
Disallow /favourites
Disallow /home
Disallow /mutes
Disallow /notifications
Disallow /pinned
Disallow /search
Disallow /settings
Disallow /share-target
Disallow: /intent
Disallow: /settings
Disallow: /blocks
Disallow: /bookmarks
Disallow: /compose
Disallow: /conversations
Disallow: /domain_blocks
Disallow: /favourites
Disallow: /home
Disallow: /mutes
Disallow: /notifications
Disallow: /pinned
Disallow: /search
Disallow: /settings
Disallow: /share-target
# Wait 1 second between successive requests.
Crawl-delay: 1