Using wget to crawl your website

April 2, 2018

If you are looking to crawl your website for something like cache warming you can do so with wget quite easily. The following wget command will crawl a site and leave nothing behind on the local filesystem afterward. wget –mirror -q -e robots=off -p -r –delete-after -nd –mirror crawls the entire site -q prevents