Using wget to crawl your website

April 2, 2018

If you are looking to crawl your website for something like cache warming you can do so with wget quite easily.

The following wget command will crawl a site and leave nothing behind on the local filesystem afterward.

wget --mirror -q -e robots=off -p -r --delete-after -nd http://www.isleofmtv.com
–mirror crawls the entire site
-q prevents wget from writing output to the buffer
-e robots=off ignores directives in robots.txt
-p get all page assets
-r recursively request pages
–delete-after clean up any local files after running wget
-nd prevents wget from writing a directory structure that gets left behind