this post was submitted on 03 Aug 2026
24 points (100.0% liked)

Linux

14607 readers
291 users here now

A community for everything relating to the GNU/Linux operating system (except the memes!)

Also, check out:

Original icon base courtesy of lewing@isc.tamu.edu and The GIMP

founded 3 years ago
MODERATORS
 

I'm a big Civ IV fan, and I'd like to download all of the Sample Game threads on this Civfanatics site: https://forums.civfanatics.com/threads/sample-game-directory.258697/

I'd like to have all of the posts and images from each sample game, each of which is a forum thread with multiple pages, posts, and images hosted from elsewhere.

Here's the wget function I tried (which I mostly stole from here https://letswp.justifiedgrid.com/download-discourse-forum-wget/) :
wget --mirror --page-requisites --convert-links --adjust-extension --compression=auto --reject-regex "/search" --no-if-modified-since --no-check-certificate --execute robots=off --random-wait --wait=1 --user-agent="Googlebot/2.1 (+http://www.google.com/bot.html)" --no-cookies -r https://forums.civfanatics.com/threads/sample-game-directory.258697/

It gave me an index file that was way too small, and while it did take me to the index page, seemingly locally on my system, clicking on any of the forum threads took me to the actual website instead of a local mirror. Also I don't think any of the images (which are linked from elsewhere) downloaded.

I'm mostly just interested in a subset of the sample games, so I'd be fine downloading each sample game forum post separately, but I haven't been able to do that either.

Any help would be appreciated!

Edit: I also just tried httrack with no results either:
httrack "https://forums.civfanatics.com/threads/all-leaders-challenge-game-strategy-session-montezuma.168755/" "+https://forums.civfanatics.com/threads/all-leaders-challenge-game-strategy-session-montezuma.168755/" -v

you are viewing a single comment's thread
view the rest of the comments
[โ€“] hakase@lemmy.zip 4 points 1 week ago (1 children)

Thanks so much - FFF looks like it might be exactly what I'm looking for, so I'll see if I can get it to work tomorrow!

[โ€“] solrize@lemmy.ml 4 points 1 week ago* (last edited 1 week ago)

FFF won't work on that site right out of the box, because of the bot challenge. You'll need a workaround. FFF has the same problem with fanfiction.net (FFN) which is one of the biggest fanfic sites. So there are a number of FFF github issues and doc entries related to FFN and looking at those might help. I do know that FFN is scrapable using browser orchestration. https://github.com/FicHub/fichub.net may have some code for that, but getting your own instance running will be quite a bit more headache than just running fanficfare.