About Lemmit

Poor lemmy server went from constant 80-100% cpu load to gently coasting.

That thing must have been dragging things down for months, if not years.

Update: (Sunday 23:00) And we're back. 2 million posts have been deleted, and their communities have been disabled in the bot. That should keep things going a little longer.

Update 2: (Monday 10:00) All the communities have been synchronized and are up to date again. That's great news! Before the purge it would take at least 24 hours to handle all communities after an outage, and even then it'd be lagging behind.

The bot is currently paused on account of a huge cleanup taking place.

The amount of post has once again outgrown the size of the server, so a lot of communities and posts will be purged, focusing first and foremost on those with the least amount of interaction. The server had been running flawlessly for the past few months, but apparently we hit a bottleneck at around 7 million posts. Which is nothing to be sneered at.

Anyway, for the upcoming week or more, a huge cleanup action will take place (deleting nearly 2 million posts), during which the bot will not run. This should give some breathing space, and also slow down the growth going forward.

So for all the Lemmit users: sorry for the delay. Also sorry if your favorite community got wiped, and no, this is not negotiable.

Server space is getting crowded, so I'll be deleting a lot of communities that have tons of posts, but very little followers. And when I say tons of posts, I mean about 660.000 - a little under 10% of the total Lemmit posts. To put it into perspective - the next biggest Lemmy instance, LemmyWorld, has about 550k posts. I'll be deleting more posts than they have in total. Talk about a hobby getting out of hand...

Since the Lemmy server model was never designed for this use case (rightfully so), and I'm running this on a teensieweensie server - this is asking a lot of the database. Therefor the site will be blocked from regular traffic to lessen the load. Expect this maintenance time to last through the weekend, maybe longer.

With Vultr's upcoming price hike, I'm planning to migrate this instance storage bucket to a more affordable alternative. Unfortunately, it's rather large, (over 700 GB in thumbnails), so it's going to take a while to transfer.

Long story short: the bot will be not be posting new content for a couple of days - probably about a week.

All migrations and updates have completed. The server is now all up to date again, and the bot is playing catch up.

As my title says can you remove the “[OC]” identifier in the post titles if they exist? I don’t really think it’s worth going overboard with all of the variations but it seems like that’s the most common format people on reddit use to identify their original content. If not it’s not a big deal.

We've had a bit of a conversation, over in the big NoStupidQuestions community:
https://lemmy.dbzer0.com/post/37540045

While I have my own opinions on lemmit.online, I think it's relatively uncontroversial, that copying content from amateur and indie creators is unethical.
I'd like to request differentiating between the regular Reddit content, and amateur pornography plus OF creators and their original content. And deactivating the bridging for subreddits that contain a decent amount of the latter.

My rationale is more or less that it's not very Robin Hood to take things from people who aren't well off in the first place. And that more or less regular people have the right to decide what happens with pictures of their naked bodies, and we can't just spread them across the internet without their consent or ability to closely control their intimate stuff.

The good news: The migration is complete, and I've even managed to update the version to 0.18.5 (was stuck on 0.18.4-beta8 for the longest time).

~~The sad news: Cloudflare is having some issues, so nobody is able to access the new server at this time. Oh well. It'll probably be fixed Saturday morning, and I'll turn the bot back on.~~

Migration complete, and the bot has caught up on the 24 hour gap that it was offline. It only took like 12 hours this time, while in the past it was closer to taking an entire day. It probably helped that the new VM is dual core, even though the bot itself only ever makes 1 request at a time, so I didn't expect this much of an improvement.

Since its inception, the Lemmit instance has been controversial. That might be an understatement, but let's roll with it for now. One of the major issues people have with the bot, is the cross-posting of "interactive" Reddit posts, ie posts where the value lies in interacting with the OP, like AskMen, AskWomen, and AmITheAsshole. Personally, I fully agree with that viewpoint, but I didn't feel like interfering with supply and demand - in the sense that AmITheAsshole, for some reason, is the most subscribed community on this server.

That might change though. Earlier this week, I disabled the posibility to request new subreddits. This weekend I will follow that up by disabling the scraping of so-called interactive communities. So in order to facilitate that, I created a list of all the communities on this server (posted separately in !about@lemmit.online), and I will check each of them to see if they should be disabled. The goal is to keep a list of "content only" (or at least "content mostly") communities, where the value lies in the link that's provided or in the body of the self-post - not in the comment section. I'm sure this is going to be a disappointment to some people, but I do agree with the sentiment that this is better for Lemmy as a whole.

Edit: It is done. All 816 communities have been checked and and 110 of those have been purged from updates. I am sure some mistakes were made - that some communities have been disabled or left intact when they shouldn't have. If that's the case, reach out to me, and I'll fix it.

Hi,

I don't post this to be malicious or rude, but simply out of concern.

I believe this bot is killing community growth on Lemmy. I keep coming across would-be-interesting communities only to find a wall of bot posts with zero comments, zero votes, zero engagement. And I walk away disappointed instead of getting into a new community. As, I'm sure, may other users are doing.

Sure, you say, "just block the bot" if I don't like it. But that doesn't stop this thing from stifling any real engagement and growth in communities. Surely if someone can "just...", you can "just" go back to reddit if you want to read reddit content that badly.

I admire the engineering you put into making this thing work. It's impressive, and honestly very cool. But I really think it's actively disengaging users, when Lemmy has enough of a hurdle to overcome in growing new communities.

Thats just my 2 cents. I'm not sure it will mean much, but I felt I had to share it. Again, no ill intent against what you've accomplished in creating this. Best wishes.

As discussed here, I have implemented a minimum level of upvotes that a post needs to have on reddit, as well as a minimum ratio of upvotes to downvotes.

Right now I have those configured to require at least 5 upvotes, and more upvotes than downvotes (0.51). At first glance this already seems to be great improvement. There might be some tweaking later.

As a side note I have now switched from using the reddit RSS feed, to using the JSON feed. This was required in order to get easy access to the upvote/ratio properties. So there might be some new and interesting new bugs introduced because of that. It's a brave new world.

Needless to say, the first thing I'll do after releasing this, is plop down on the couch with a beer, and hope this doesn't crash. Fingers crossed!

I'd like to hear some feedback on this, or approach vectors.

Right now the bot is rather spammy. I was hoping that by using Reddits HOT feed, it would return have some level of quality control (I know, right?). Unfortunately, it seems that in most cases, it will just return anything that's new. The downside of this is that a lot of garbage gets through, and the bot spends a lot of time scraping the underlying page to get the details.

I propose to only archive reddit posts that have a karma score of 5 or higher. In case of subs that hide the karma scores of posts for a certain time, they'd have to be at least 2 hours old, so that the Reddit moderators can weed out garbage on our behalf.

Do you folks have any thoughts on this?

Secondly, I want to put sticky comments on each community, with links to native Lemmy communities that cover the same subject. For this I would need some kind of API, or a master list of... oh, I see sub.rehab has just the thing I need. So expect that somewhere this week :).

See you on the other side!

So the update is done, but the bot was offline for 6 hours, and needed to catch up.

Unfortunately, another update slipped through, which switched the default feed from www.reddit.com to old.reddit.com, which has the side effect of changing all the urls in the posts as well. On one hand this is great, because new reddit sucks. On the other hand, this is terrible, because for every post the bot encounters, it checks if it already exists on lemmit... based on the url.

So for every post the bot encountered, it went like "old.reddit.com/r/blabla/123? Haven't seen that one yet, there's an www.reddit.com/r/blabla/123, but that must be something completely different, let's post it again!"

This also meant that the bot took over a minute and a half to update each community because it takes a couple of second per post. When I went to bed last night, I figured it was just posting a lot of content because it had so much catching up to do. But this morning I figured something was off because it still hadn't caught up.

Anyway, the fix is out now. Sorry for all the duplicates. I need coffee now.

ChatGPT, write a post for the stuff that I have in my head and want to get out as an update.

Hmm. No brain implant yet. Guess I'll have to write this the hard way.

Syncing update

It has been an eventful week. I successfully deployed the initial version of smarter content syncing, and have made some adjustments to algorithm since then. Most notably, communities with only 1 subscriber (the bot) will no longer receive updates, and communities with fewer than 5 subscribers or with a low posting frequency will only be updated twice a day. Furthermore, for the highest update priority (every 10 minutes), a community must have a minimum of 50 subscribers. Implementation details can be found in the decide_interval() method over here.

Being a developer is fun

Meanwhile... Damnit, bot is stuck again.

2023-07-08 10:13:39,945 - utils.syncer - INFO - Scraping subreddit: bustynaturals. Last time  2:30:48 ago, interval 120 minutes
2023-07-08 10:13:40,653 - utils.syncer - INFO - 'latina bodies are the best' at https://www.reddit.com/r/BustyNaturals/comments/14twww8/latina_bodies_are_the_best/ updated: 2023-07-08 07:14:13+00:00
2023-07-08 10:13:45,324 - utils.syncer - ERROR - Error trying to retrieve post details, try again in a bit; Couldn't retrieve post detail page
2023-07-08 10:13:46,333 - utils.syncer - INFO - Scraping subreddit: bustynaturals. Last time  2:30:54 ago, interval 120 minutes
2023-07-08 10:13:48,581 - utils.syncer - INFO - 'latina bodies are the best' at https://www.reddit.com/r/BustyNaturals/comments/14twww8/latina_bodies_are_the_best/ updated: 2023-07-08 07:14:13+00:00
2023-07-08 10:13:51,227 - utils.syncer - ERROR - Error trying to retrieve post details, try again in a bit; Couldn't retrieve post detail page
...

1 bugfix and deployment later:

2023-07-08 10:46:42,836 - utils.syncer - INFO - Scraping subreddit: bustynaturals. Last time  3:03:51 ago, interval 120 minutes
2023-07-08 10:46:43,573 - utils.syncer - INFO - 'latina bodies are the best' at https://www.reddit.com/r/BustyNaturals/comments/14twww8/latina_bodies_are_the_best/ updated: 2023-07-08 07:14:13+00:00
2023-07-08 10:46:48,327 - utils.syncer - ERROR - Couldn't find post on https://old.reddit.com/r/BustyNaturals/comments/14told8/latina_bodies_are_the_best/, skipping.

Defederation

Meanwhile, the folks at https://lemmy.world/ reached out to me to tell me they're defederating Lemmit. They are not fond of high volume of posts made by the bot, and the fact that there are now (quick check) 462 communities on this server all being moderated by a single person. They have already received a couple of complaints about spam, and it didn't help that some requests for NSFW subreddits were not marked as NSFW. Occasionally, those subreddits had explicit thumbnails that appeared in the 'All feed' without warning.

I had a good talk with the LemmyWorld admin, wherein they explained their point of view, and I explained mine. I understand their decision to disassociate with Lemmit, and appreciate their attempt to contact me. Other instances like Beehaw, and some smaller ones have also reached the same decision.

This does mean that you will no longer be able to get new community updates on those servers. So make sure to check the blocked instances list on your home server if you were subscribed to Lemmit. At the same time I have removed all the subscriptions of users from those servers, in order to not affect the sync priority mentioned above. This does mean, that if LemmyWorld, Beehaw, etc ever decide to connect to Lemmit again (however unlikely), you will need to un- and re-subscribe from there.

Meanwhile, I've added a feature in the bot that will remove request posts for NSFW subreddits, if the post itself is not marked for NSFW. This should prevent explicit thumbnails showing up where they are not wanted.

Server growth

Last night I got an alert from my server monitoring that the disk is 80% full. Unfortunately, the disk is only 60 GB, so that doesn't leave much room for expansion. On the bright side, a good chunk of that is from Lemmys very verbose logging (like, 4 GB a day, which gets cleaned up daily), so it should last throughout the weekend if I tune that down. Furthermore, most of the storage growth is from from pictrs, the image upload part of Lemmy, and that can utilize an S3 bucket, rather than using the VM's storage like it is now. Using an S3 bucket offers a cost-efficient solution for expanding storage. Initial estimates indicate a monthly cost of around $5 for 1000 GB of storage, which should be sufficient for a while *fingers crossed*.

In the early days of Lemmit (literally, as the server is less than a month old) image uploads were limited to a default setting, which was something around 40 megabytes. That did add up quickly (thanks to half-minute porn gifs), and so I had to limit the max filesize to 1 MB, and later 0.5 MB. Once the server has switched to S3 storage, I can probably up that limit a little, although not too much.

Finally, Lemmy v0.18.1 has been released, and it contains even more performance boosts compared to v0.18.0, so if there's time left this weekend (and I can verify the Lemmit Bot is compatible), I will probably perform the upgrade.

In particular, posts to NSFW videos hosted on v.redd.it don't work on the www version. The links take you to the comments page, which blocks NSFW content, and nags you to go to the app.

old.reddit.com links just work without logging in.

I want to follow many of the NSFW subreddit-communities from this account and probably request more, but the communities page doesn't show NSFW when not logged in (making an account on lemmit itself just to see the list seems like overkill and could be confusing about what's logged in where, if it's even allowed)

Okay, this one took me a bit longer than I planned (mostly due to sql fun and trying to use integers as minutes, WEEEE!).

Backdrop: Last week I disabled the mirroring of a couple of subreddits to the database, because they were initially requested but the nobody subscribed to them. At the same time, the bot was just crawling in a loop, starting at todayilearned, ending at latestsubreddit. As more subreddits were requested, this loop took longer and longer (21 minutes before I rolled out this update). This wasn't sustainable.

So here's the new situation. The more popular a community is, the more often it will be updated. In this case popular means a mixture between number of subscribers and the amount of posts it receives per day (Link to relevant snippet of source code).

In short, the most popular subs will be synced every 10 minutes, the next tier ever 30 minutes, 120 minutes and the content with either no posts per day or no subscribers (other than the bot), will only be synced every 12 hours. I hope this will hit a good distribution of updates vs popularity, but it will most likely be refined at some point in the future.

Speaking of distribution, we now have over 300 communities on this server 🥳, and their update intervals are spread out as such:

• Every 10 minutes: 22
• Every 30 minutes: 39
• Every 60 minutes: 55
• Every 120 minutes: 143
• Every 720 minutes: 44

With this update running live (I started typing after I deployed it, and it has now gotten through the backlog of 'abandoned' subs), I'm going to step back from feature development for a few days. Any bugs that cause the bot to crash will of course continue to be addressed.

Have a blast!

For example, it'd be nice for anyone that looks at/finds https://lemmit.online/c/dota2 to also find that https://discuss.tchncs.de/c/dota2 or https://lemmy.world/c/dota2 is an actual community that corresponds to that with user content, not bot content.

I'm sure there's lots of equivalents for other communities as well where that would make sense.

Before was running on the cheapest model (1 core / 1GB mem / 30GB storage) at $12/month. The machine was running pretty low on memory, causing it to start swapping, which in turn caused the cpu to get too busy, and everything to slow down.

Now it has a whopping 2GB of memory, and things seem to have calmed down - cpu is back to around 10-15% usage, and swap is down to 0. Happy times all around.

Because of the amount of subs being archived, it now takes about 15 minutes between updates for each sub (was 18 before I updated the VM).

I'm planning to build some kind of scoring system, based on the amount of posts per subreddit (per day?), and amount of subscribers on the lemmy community. That way communities with little subscribers or that don't see many posts per day, will only be updated once per hour.

At the same time, I feel that subs like AskReddit, OutOfTheLoop and other "question-based" subreddits shouldn't be archived by Lemmit. In my opinion those kind of posts are useless without those answers, but please let me know if you disagree.

• Fixed a bug where posts would not be submitted because the title didn't contain long enough words.
• Fixed a bug where posts would not be submitted because the url was too long.
• Fixed a bug where posts would not be submitted when it was linking to a /user subreddit.
• Fixed a bug where the bot would think Every Post Everywhere was a subreddit request, and would reply to it.
• Fixed a bug where the bot would crash without recovering whenever something went wrong during new subreddit requests

A fruitful day all in all, I'd say.

That the replies-everywhere-bug was just because I forgot to include a variable in the bot deployment? 🤦

Long story short: I messed up with the domain registration for this instance, and never replied to a mandatory email. The domainname (lemmit.online) got put in suspension, causing disconnects all over the fediverse.

I fixed it as soon as I found out, but it will probably take a few more hours for the issues to be fully fixed.

So ehm. Whoops. Hope this explains and fixes the federation issues we've been having today.

Most importantly that the bot no longer crashes (and does nothing all night while I sleep 😛) when trying to create a community that has already been requested.

Furthermore mostly making the code prettier and adding tests.

Fixed a couple of bugs today:

• Nasty one that made the bot get stuck in an infinite when trying to add a post by a deleted user, which kept the bit offline for most of last night.
• Another creative one that, when posting certain links, would actually work, but the lemmy gateway would respond with a timeout. It only happens on certain links, but consistently. Which would make the bot think it was unsuccessful, which would make it try to post again the next time. Causing a duplicate post each time (technically it was a cross post to itself... Which is interesting in a whole new way).

TLDR: right now there is a workaround in place that assumes a timeout post to lemmit was actually successful. This might cause it to drop posts in the future, but seeing that the server is barely breaking a sweat at this time, it should be good until a better fix is implemented.

Also got some great feedback from users, which I added to the TODO.