I'm a big Civ IV fan, and I'd like to download all of the Sample Game threads on this Civfanatics site: https://forums.civfanatics.com/threads/sample-game-directory.258697/
I'd like to have all of the posts and images from each sample game, each of which is a forum thread with multiple pages, posts, and images hosted from elsewhere.
Here's the wget function I tried (which I mostly stole from here https://letswp.justifiedgrid.com/download-discourse-forum-wget/) :
wget --mirror --page-requisites --convert-links --adjust-extension --compression=auto --reject-regex "/search" --no-if-modified-since --no-check-certificate --execute robots=off --random-wait --wait=1 --user-agent="Googlebot/2.1 (+http://www.google.com/bot.html)" --no-cookies -r https://forums.civfanatics.com/threads/sample-game-directory.258697/
It gave me an index file that was way too small, and while it did take me to the index page, seemingly locally on my system, clicking on any of the forum threads took me to the actual website instead of a local mirror. Also I don't think any of the images (which are linked from elsewhere) downloaded.
I'm mostly just interested in a subset of the sample games, so I'd be fine downloading each sample game forum post separately, but I haven't been able to do that either.
Any help would be appreciated!
Edit: I also just tried httrack with no results either:
httrack "https://forums.civfanatics.com/threads/all-leaders-challenge-game-strategy-session-montezuma.168755/" "+https://forums.civfanatics.com/threads/all-leaders-challenge-game-strategy-session-montezuma.168755/" -v
Discourse is kind of a moving target at the best of times, and I wouldn't bet on any precanned command older than a couple of months working properly.
The server may have flagged your wget command as an unacceptable bot—as written, it's ID'ing itself as a bot, can pull a new page every half a second if you get unlucky with --random-wait, and isn't respecting robots.txt
The thing with the local index page pointing to the forum threads on the server matches what the wget man page says about --convert-links where the target file wasn't actually downloaded. Is it pulling any files at all other than the index? Do the files' names and extensions match what shows up in the index if you don't pass --convert-links?
Also, I'm pretty sure there's no way to get those images if they aren't hosted on the same server. (This is just common sense, or an offsite link could result in wget trying to mirror the entire Internet.)