I'm a big Civ IV fan, and I'd like to download all of the Sample Game threads on this Civfanatics site: https://forums.civfanatics.com/threads/sample-game-directory.258697/
I'd like to have all of the posts and images from each sample game, each of which is a forum thread with multiple pages, posts, and images hosted from elsewhere.
Here's the wget function I tried (which I mostly stole from here https://letswp.justifiedgrid.com/download-discourse-forum-wget/) :
wget --mirror --page-requisites --convert-links --adjust-extension --compression=auto --reject-regex "/search" --no-if-modified-since --no-check-certificate --execute robots=off --random-wait --wait=1 --user-agent="Googlebot/2.1 (+http://www.google.com/bot.html)" --no-cookies -r https://forums.civfanatics.com/threads/sample-game-directory.258697/
It gave me an index file that was way too small, and while it did take me to the index page, seemingly locally on my system, clicking on any of the forum threads took me to the actual website instead of a local mirror. Also I don't think any of the images (which are linked from elsewhere) downloaded.
I'm mostly just interested in a subset of the sample games, so I'd be fine downloading each sample game forum post separately, but I haven't been able to do that either.
Any help would be appreciated!
Edit: I also just tried httrack with no results either:
httrack "https://forums.civfanatics.com/threads/all-leaders-challenge-game-strategy-session-montezuma.168755/" "+https://forums.civfanatics.com/threads/all-leaders-challenge-game-strategy-session-montezuma.168755/" -v
Many websites are detecting tools like curl, wget and httrack and either rate limit or block them. Sometimes one can get around it when using more detailed configurations that mimick like a human using a real browser, but that is basically tricking them. The reason why they are blocking such automated tools is multifolded (if that is even a word): a) downloading pages and data in short amount of time creates huge stress for the servers, meaning its like an attack that halts their systems, b) that in turn also leads to higher server costs for them, c) they don't get visitors for the advertisements and engagement, as the tools download autonomously, d) is also sometimes seen as theft from Ai companies, so they do try to block automated mass downloads.
Besides these tools there are other ways, more sophisticated and requires more work to mimick real browsers. In some cases even using real browser engines, but changed the browser in a way to make automated work possible, such as https://www.selenium.dev/ . Especially needed when JavaScript and other dynamic content is involved. Maybe even programming Python with custom solutions. There is also a way to directly program in Typescript using https://deno.com/ . But these solutions are a bit advanced and only recommended if you want go deeper.
Soo, having all said that, this might not help directly solving your issue, but I hope its still helpful in why its difficult. Especially nowadays, because companies have better detection and blocking, because of Ai...