I'm a big Civ IV fan, and I'd like to download all of the Sample Game threads on this Civfanatics site: https://forums.civfanatics.com/threads/sample-game-directory.258697/
I'd like to have all of the posts and images from each sample game, each of which is a forum thread with multiple pages, posts, and images hosted from elsewhere.
Here's the wget function I tried (which I mostly stole from here https://letswp.justifiedgrid.com/download-discourse-forum-wget/) :
wget --mirror --page-requisites --convert-links --adjust-extension --compression=auto --reject-regex "/search" --no-if-modified-since --no-check-certificate --execute robots=off --random-wait --wait=1 --user-agent="Googlebot/2.1 (+http://www.google.com/bot.html)" --no-cookies -r https://forums.civfanatics.com/threads/sample-game-directory.258697/
It gave me an index file that was way too small, and while it did take me to the index page, seemingly locally on my system, clicking on any of the forum threads took me to the actual website instead of a local mirror. Also I don't think any of the images (which are linked from elsewhere) downloaded.
I'm mostly just interested in a subset of the sample games, so I'd be fine downloading each sample game forum post separately, but I haven't been able to do that either.
Any help would be appreciated!
Edit: I also just tried httrack with no results either:
httrack "https://forums.civfanatics.com/threads/all-leaders-challenge-game-strategy-session-montezuma.168755/" "+https://forums.civfanatics.com/threads/all-leaders-challenge-game-strategy-session-montezuma.168755/" -v
If it is just for your own (offline) reading pleasure, and it is not a too large a number of pages, then SingleFile is an easy way to archive pages in a semi-automated fashion: You can configure it such that it automatically saves every page you visit based on a pattern or just for a tab, so you'll just have to page through the threads to save them
Thanks, I'll try this next!
Yeah, it's just for me. I go back and read these threads once every five years or so, and I'm just worried they'll be gone one day.