admin

joined 2 years ago
MODERATOR OF
 

Before was running on the cheapest model (1 core / 1GB mem / 30GB storage) at $12/month. The machine was running pretty low on memory, causing it to start swapping, which in turn caused the cpu to get too busy, and everything to slow down.

Now it has a whopping 2GB of memory, and things seem to have calmed down - cpu is back to around 10-15% usage, and swap is down to 0. Happy times all around.

Because of the amount of subs being archived, it now takes about 15 minutes between updates for each sub (was 18 before I updated the VM).

I'm planning to build some kind of scoring system, based on the amount of posts per subreddit (per day?), and amount of subscribers on the lemmy community. That way communities with little subscribers or that don't see many posts per day, will only be updated once per hour.

At the same time, I feel that subs like AskReddit, OutOfTheLoop and other "question-based" subreddits shouldn't be archived by Lemmit. In my opinion those kind of posts are useless without those answers, but please let me know if you disagree.

 
  • Fixed a bug where posts would not be submitted because the title didn't contain long enough words.
  • Fixed a bug where posts would not be submitted because the url was too long.
  • Fixed a bug where posts would not be submitted when it was linking to a /user subreddit.
  • Fixed a bug where the bot would think Every Post Everywhere was a subreddit request, and would reply to it.
  • Fixed a bug where the bot would crash without recovering whenever something went wrong during new subreddit requests

A fruitful day all in all, I'd say.

 

That the replies-everywhere-bug was just because I forgot to include a variable in the bot deployment? 🤦

1
submitted 2 years ago* (last edited 1 month ago) by admin@lemmit.online to c/about@lemmit.online
 

In the short time since this instance and bot launched, I've been seeing the same questions resurface multiple times. This is totally understandable, since the concept of a Fediverse is still new to most (myself included), and this server is not like the others.

Q: What is Lemmit?

A: Lemmit is a Lemmy instance specifically designed for archiving Reddit content. Users can request new subreddits to be included in the archiving process by posting in the !requests@lemmit.online community. It is powered by an open source python bot, which periodically checks the request list, adds new requests to the queue, and continuously monitors the Hot feed of those subs for new posts to cross-post here.

Q: Does it synchronize comments?

A: No, that would be impossible. Considering there are thousands of posts already on Lemmit, many of them having at least several hundred comments on Reddit, often buried in deep layers, it simply wouldn't be feasible to index those for more than a few posts, let alone keep them up to date.

Unfortunately, this means that archiving certain subreddits, such as Ask Historians/Men/Women/Hyperintelligentshadesofthecolourblue-type subs, is going to be rather pointless.

Q: Can it send comments back to Reddit?

A: No, it cannot. The purpose is to help bootstrap the Lemmy platform, not to serve as a bridge between the two networks. Also, see the answer about synchronizing comments.

Q: Can I request any subreddit?

A: ~~Technically, yes. However, as the list of subs grows, the time it takes to update all of them will also increase. I do not have strict guidelines in place for this, so I'm relying on your common sense (hoooo boy). At some point, I will probably have to either stop accepting new requests or disable scraping for very low-traffic communities.~~ Nope. Automatic requests for new subreddits are closed. ~~You can send me a PM if you think a specific subreddit would be a valuable addition to the Lemmyverse, but you'll have to make a strong case.~~ Going forward, I only expect the amount of subreddits to go down, since I don't intend to let the server grow endlessly large.

Q: Does this use the API? Will it keep working after July 1st?

A: Nope, it uses a combination of the public feed and scraping old.reddit.com. So, as long as those are still available, it will continue working. And even if they close those sources, there will probably be new ways to achieve the same effect. "Content, eh, finds a way."

Q: This is spam, can you stop?

A: First of all, I apologise for the inconvenience. All you have to do is block @bot@lemmit.online, and none of its posts will ever show up on your instance. If you you don't want anyone else on your server to be exposed to this bot/instance, you should convince your admin to defederate from lemmit.online. Since there are no other users on here, there will be no harm done.

Obviously I could stop, because running this server and software is only ever going to cost me time and money. But for the reasons listed above, I still think this server is a useful addition to the lemmyverse at this time. But I'm looking forward to the day where I can turn the bot off because it's no longer needed.

Q: What started this?

A: Okay, nobody asked this, but I'm going to tell you anyway. After Reddit made it clear that they are effectively killing third-party apps and implementing plenty of other anti-end user decisions, I realized that I would either have to accept not being able to access my time-wasting content or have to do so in a rather uncomfortable way (either through the official app or old.reddit.com for as long as they'll allow it to exist).

Being a stubborn developer, naturally, I chose option C: Have my own Reddit. With blackjack, and hookers. This way, I would still be able to access my beloved content without being beholden to Reddit's mood swings and abusive relationship tendencies.

Besides that, I also know that Content is King. So I'm order to counter the network effect (No users because no content, No content because no users), I figured it would be better to have some inorganic content to bootstrap the adoption of Lemmy.

Q: Are NSFW subreddits allowed?

A: Absolutely. Like I said: Blackjack and hookers.

Q: My request isn't picked up by the bot!

A: That isn't a question. ~~But yeah, the process isn't flawless yet. I'm trying to iron out all the bugs as I encounter them. In the meantime, feel free to re-request the subreddit by making a second post. No harm done.~~ New subreddit requests are closed for the time being.

Q: No new posts are showing up at all on Lemmit

A: If no posts are appearing on the Lemmit Frontpage (sorted by NEW), it's possible that the bot has crashed or is stuck on something. Since no software is flawless, this sometimes happens. I usually fix this as soon as I'm aware, and I'm happy to say that these kinds of fatal errors are becoming less and less frequent. However, they may still occur, and as a human with needs of sleep and other responsibilities, I'm not always able to fix them immediately.

Q: Posts aren't showing up on my instance, what's up?

A: Due to the spammy nature of the bot, some server admins choose to block this server, and that is completely understandable. So first of all, make sure to check the instances link in the footer of your home server. If Lemmit is the Blocked Instances list, you're out of luck.

When you have verified that Lemmit is not blocked on your instance, try unsubscribing, waiting a little, and then re-subscribing. That tends to fix things.

 

Long story short: I messed up with the domain registration for this instance, and never replied to a mandatory email. The domainname (lemmit.online) got put in suspension, causing disconnects all over the fediverse.

I fixed it as soon as I found out, but it will probably take a few more hours for the issues to be fully fixed.

So ehm. Whoops. Hope this explains and fixes the federation issues we've been having today.

 

Most importantly that the bot no longer crashes (and does nothing all night while I sleep 😛) when trying to create a community that has already been requested.

Furthermore mostly making the code prettier and adding tests.

1
Bug fixes 19-06-2023 (lemmit.online)
submitted 2 years ago* (last edited 2 years ago) by admin@lemmit.online to c/about@lemmit.online
 

Fixed a couple of bugs today:

  • Nasty one that made the bot get stuck in an infinite when trying to add a post by a deleted user, which kept the bit offline for most of last night.
  • Another creative one that, when posting certain links, would actually work, but the lemmy gateway would respond with a timeout. It only happens on certain links, but consistently. Which would make the bot think it was unsuccessful, which would make it try to post again the next time. Causing a duplicate post each time (technically it was a cross post to itself... Which is interesting in a whole new way).

TLDR: right now there is a workaround in place that assumes a timeout post to lemmit was actually successful. This might cause it to drop posts in the future, but seeing that the server is barely breaking a sweat at this time, it should be good until a better fix is implemented.

Also got some great feedback from users, which I added to the TODO.

view more: ‹ prev next ›