Hints & Easter Eggs

Many a times tools can only do so much that you have to rely on your instinct and do some investigation on your own. These are some of the things that you can do besides the results that you may find in a tool to find new ways to either infliterate or elevate your privilege within a system.

Robots.txt

Robots.txt files are present on most of the websites and applications. This file tells any website crawler, such as the spiders from Google, the list of URLs that it should stay away from. So instead of using a fuzzer or dirb, you can quickly check the robots file for any special URLs.

Robots file may not be accessible from the browser directly, so modify the HTTP header to make it appear to be sent by a google/yahoo/bing bot,

#Add/Modify the http request

User-Agent: DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html)
X-Robots-Tag: googlebot: nofollow