robots.txt and a cool HN thread
| Posted on: |
03/01/2022 01:50:50 PM |
| Last edited: |
03/04/2022 12:52:31 PM |
the hn thread
- https://news.ycombinator.com/item?id=30299731
cool urls i found in comments
- https://www.nike.com/robots.txt
- https://www.youtube.com/robots.txt
- https://www.cloudflare.com/robots.txt
- https://www.last.fm/robots.txt
- https://www.bloomberg.com/robots.txt
- https://meta.wikimedia.org/robots.txt
lolol, my robots.txt is merely 2 lines!
And I don’t disallow anything. Some of these are pretty creative, I should add
some ascii art myself.
Tags: