Use ZFS
Linux
From Wikipedia, the free encyclopedia
Linux is a family of open source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991 by Linus Torvalds. Linux is typically packaged in a Linux distribution (or distro for short).
Distributions include the Linux kernel and supporting system software and libraries, many of which are provided by the GNU Project. Many Linux distributions use the word "Linux" in their name, but the Free Software Foundation uses the name GNU/Linux to emphasize the importance of GNU software, causing some controversy.
Rules
- Posts must be relevant to operating systems running the Linux kernel. GNU/Linux or otherwise.
- No misinformation
- No NSFW content
- No hate speech, bigotry, etc
Related Communities
Community icon by Alpár-Etele Méder, licensed under CC BY 3.0
The Btrfs and ZFS filesystems support compression.
This is how I do it. I put my Steam directory in an external BTRFS drive with compression turned on. Works great, and allows me to take my library between computers.
Have you measured the difference? Is it huge? I’m curious to understand what the trade-offs are and how many more games can you store on the same drive.
My main goals are efficient space usage and portability, so I've never measured anything. I'll get some stats for you.
I would appreciate even not very accurate comparison just to get the idea. A brief search meanwhile showed me it’s a great idea actually. But having someone who uses that to comment would be great too. Thanks.
As noted by others, I'm pretty sure many different solutions exist. But, FWIW, I've had good results with DwarFS.
Btrfs disk with periodical dedup.
Does dedup no longer shred the file system?
Never did with my weekly cronjob. But i have ext4 on the boot partition, btrfs on the 1.5 TB game partition.
If you make any changes to the game files in Steam, including compression, then Steam will update the files and redownload them. And for online games, you could even get a ban if an Anticheat system thinks you try to cheat because the files are tempered (changed).
However there is a transparent compression available on filesystem level. Meaning files are saved compressed and look and work like uncompressed files. Something similar what Windows has I believe. EXT4 does not have transparent compression, but I often read Btrfs and ZFS do. I have no experience with that, so cannot assist. But at least you should be able to search for this now. Have in mind that compressed filesystem would be slower.
I would suggest Btrfs for regular use. ZFS is better for server use and redundancy in systems with critical data you don't want to lose. It also uses all your ram as cache and will give it back to you if you need it, but still has a harder performance hit on things you run. Be overkill for just wanting to save a little space. A lot of new gaming distros like Bazzite suggest using Btrfs on all secondary drives.
Have in mind that compressed filesystem would be slower.
Often the opposite is true, depending on case. Compressed files load faster, so if you have the cpu power to spare (which you usually have in games while loading) and loading speed is the bottle-neck then compression speeds things up, often considerably.
And even in the age of ssds processing data and moving it through ram is much faster than the disk, so even for writing some amount of transparent compression is possible without affecting speeds.
If you have a fast CPU adding compression can actually improve performance. https://www.phoronix.com/review/btrfs-zstd-compress
with fast nvme not that much.. not like the 'olden days' of msdos and stacker where it was actually noticeable on some systems.
Hmm, looks like it can be slower on write, but faster on read depending on the CPU and drive probably. Games mostly read files, so this might even improve performance.
Aren’t most games assets files already compressed?
With compress=zstd ZFS reports a 1.30 compression ratio on my Steam dataset, compressing it from ~1.2TB (reported by the file manager) down to 904GB (reported by zfs list -o name,usedds). Pretty good I think.
Obviously it depends on the games. AAA crap will probably add tens/hundreds of GBs of pre-compressed asset blobs, while indie and older games will often have more loose file structures with config files, scripts, runtimes etc. that all compress extremely well. And with older games, even when compressed the algorithms are often far from ideal and zstd can still get a few more percent out of them.
Btrfs here, with zsdt:3, on my game data the compression ratio is a bit worse than that, but it does save some space to my surprise.
with default btrfs zstd compression here, i save a whole 2 percent on game install directories..but 48 percent elsewhere (trixie kde, ~ 2800 dpkg + flatpaks. 'documents' are on a diff filesystem)
There are about a million ways to do what you’re asking. Which one is the right choice depends on the particular hardware you’re using and what you want to accomplish.