/kbin server update - or how the server didn't blow up

Currently, on the main instance, people have created 40191 accounts (+214 marked as deleted). I don't know how many are active because I don't monitor it, but once again, I greet all of you here :) In recent days, the traffic on the website has been overwhelming. It's definitely too much for the basic docker-compose setup, primarily designed for development use. I was aware of the possible consequences of the situation happening on Reddit, but I assumed that most people would migrate to one of the Lemmy instances, which already has an established position. I hoped that a few stray enthusiasts would find their way to kbin ;)

The first step was to upscale the VPS to a higher version (66.91EUR). It quickly turned out that it wasn't enough. I had to enable CF protection just to keep the website responsive, but the response times were still very slow. At this stage, the instance was practically unusable. The next step was a full migration to a dedicated server (100EUR, the current hardware). It can be done relatively quickly, so it resulted in a 5-minute technical break. Despite the much higher parameters, it didn't get any better. It became clear that the problem didn't lie there. I'm really frustrated when it comes to server administration. That was the moment when I started looking for help. Or rather, it found me.

A couple days ago I wrote about how kbin qualified for the Fast Forward program. To be honest, I did it out of pure curiosity and completely forgot because a lot was happening during that time. During the biggest fire incident, Hannah ( @haubles ) reached out with a proposal to help. I outlined the situation (in short: the server is dying, I don't even know what I need, help! ;). She quickly connected us with Vlad ( @vvuksan ) and Renaud ( @renchap ). I was probably too tired because I don't know if the whole operation lasted 60 minutes or 6 hours, but after a series of precise questions and getting an understanding of the situation, the guys themselves adjusted the entire job. I love working with experts, and it's not often that you come across individuals so well-versed in the fediverse. Thanks to Hannah's kindness, we will be staying there a bit longer. Currently, fastly.com handles the caching layer and processes images. Hence those cool moving thumbnails ;)

Things were going well at that point. I could disable Cloudflare protection. Probably thanks to that, many of you are here today, and we got to know each other a bit better :) However, even then, when I tried to enable federation, the server would stop working.

Around the same time, Piotr ( @piotrsikora ), whom I already knew from the Polish fediverse, contacted me. He is the administrator of the Polish Mastodon instance pol.social, operates within the ftdl.pl foundation, and specializes in administering applications with a very similar tech stack. I made the decision to grant him server access. It only took him a few moments, and he came back to me with a few tips that allowed us to enable federation. In the following days, there was more of it, and we managed to reach the current level. I think it's not too bad.

Nevertheless, managing the instance has taken up about 60% or more of my time so far, which prevents me from fully focusing on current tasks. That's why I would like to collaborate with Piotr and hand over full care of the server to him. Piotr will also take care of the security side. Now I have to take this much more seriously. We still need to work out the terms of cooperation, but I want you to know the direction I intend to pursue.

We also need to migrate to a new environment because one server will sooner or later become insufficient. This time, I want to be prepared for it. This may be associated with transient issues with the website in the coming days.

The next two updates will still be about project funding (I still can't believe what happened) and moderation. The following ones will be more technical, with descriptions of changes and what contributors are doing on Codeberg. I would like to be here more often, but not as an admin, just as myself.

Thank you all for this.

P.S. In private messages, I also received numerous offers of help that I didn't even have a chance to read and respond to. You are the best!

/kbin logotype
Whirlgirl9 ,
@Whirlgirl9@kbin.social avatar

hi all, you can buy ernest a cup of coffee aka give him some cash for his incredibly hard work! https://www.buymeacoffee.com/kbin

I_Miss_Daniel ,
@I_Miss_Daniel@kbin.social avatar

That 100 Euro server also needs some coins inserted :)

freebrick ,
@freebrick@kbin.social avatar

Done! Thanks @ernest! This site is so cool!

phantomslave ,
@phantomslave@kbin.social avatar

Done!

DrChickenbeer ,
@DrChickenbeer@kbin.social avatar

Yassss I'm supporter 666🤘

BigVault ,
@BigVault@kbin.social avatar

🐐

GreenPlasticSushiGrass ,
@GreenPlasticSushiGrass@kbin.social avatar

Done! Keep up the good work!

HipHoboHarold ,
@HipHoboHarold@kbin.social avatar

Unfortunately someone just tried to use my bank account the other day, so I'm waiting for a new card to come in, but I got this bookmarked for when I do. I think it's worth the cost of a cup of coffee every now and then.

RealM ,
@RealM@kbin.social avatar

What payment options are there? This seems to be only credit card.

FoodDude ,

Yes, done!

aeternum ,

Ernest should set up a patreon. I'd pay a monthly sub to him for this great website.

It's hard ot believe it's only a month or two old.

tbird83ii ,

100%

51 ,

If I was Ernest I’d be scared stupid

cordlessmodem ,
@cordlessmodem@kbin.social avatar

Know wut i mean, Vern?

Upvotes_Kills_Birds ,
@Upvotes_Kills_Birds@kbin.social avatar

Easiest $15 I've spent this month. Let's keep Kbin going!

haubles ,
@haubles@fosstodon.org avatar

@ernest @vvuksan @renchap @piotrsikora Thank you, Ernest 💞 we're so glad we can help and support you, and the project.

Nougat ,

I hoped that a few stray enthusiasts would find their way to kbin

They did. "A few" as a portion of the whole, where the whole is much larger than you ever expected it would be.

One thing I have learned in my own technical career is that you can't do everything. Even if there was enough time in a day for you to do everything, you can't know everything, at least not sufficiently enough to be most effective. You have to depend on other experts, and delegate to them what you are not qualified to do. This requires a pretty high level of trust, and makes it so that you need to develop people management skills.

Lots of technical people are short on people management skills. I don't know where you sit on that spectrum, but you may need to consider bringing on an "overseer," kind of like a project manager, to keep tabs on all of the technical resources involved - yourself included. This will help ensure that concerns are prioritized appropriately, and that communication and messaging about those priorities are consistent and clear.

I've been in that kind of position, and I take a bit of pride in my use of words. I'm happy to give any advice you like.

bourbonmakesitbetter ,

I sense an opportunity here: managed hosting of Fediverse services. I don't mean a managed host where you can install/run the services. I mean a top-to-bottom setup, management, backup, upgrades, monitoring, etc. so the only thing you need to do is administer the community. I'd love to set up several Fediverse services for my local community, and I know there is an audience that would also love that, but I also know I do not want to invest the time it takes to manage the technical side of that.

Having easier to set up instances would help in relieving the pressure from the more popular current instances. We're starting to see those hosting options come on line for some of the services, but not fast enough for my liking.

letsroll ,
@letsroll@kbin.social avatar

Love it. I'm sure there are some spam-evaluation services up or starting soon that use AI, but that will also soon be important as we become more of a target.

bourbonmakesitbetter ,

I've been thinking recently that machine learning models could be used as a first-line defense for moderation, e.g. identify obvious spam/violations, but also identify borderline cases that require human intervention. So you could reduce the burden on moderators, and perhaps even shield them from some of the more extreme things, although I think those tend to be more image/video which I imagine will be a lot harder to really effectively harness ML for.

grahamsz ,

Obviously that's a tried and tested model with email, but i'm not sure there's a great way to implement that on federated servers without keeping the model fine tuning pretty secret. Any spam detection AI model that's public can simply be used to train better spam.

My past experience with this sort of thing suggests it's probably better to focus on identifying some kind of humanness score. Since kbin instances are responsible for moderating their own user population (I believe) that means they could quite easily keep a good running score of how viable an account is. Some of that could be ML that picks up on both the information content and uniqueness of a post, but you can also infer a good amount by how much interaction it gets with other users who also have good scores.

There's also some interesting stuff in the upvote structure. If you draw a directed graph of who-upvotes-who then spammers and trolls tend to form much more distinct islands than regular users do.

themadcodger ,
@themadcodger@kbin.social avatar

Check out Grow Your Own Services. It doesn't have everything (e.g., kbin, Lemmy, Calckey) but it does offer what I think you're looking for with a number of the other more popular services.

bourbonmakesitbetter ,

Thanks! Hadn't come across that particular site yet. Unfortunately, still the same list of hosts. I'm really looking for Frendica/Mastodon and maybe Pixelfed and Writefreely. So, basically all the ones that aren't yet available in a turn-key solution.

Like I said, slower than I'd like. But I'm sure we'll get there.

keith ,

@bourbonmakesitbetter We're a small independent "mom & pop" managed hosting provider focused on Mastodon and we're currently testing out kbin, send us an email at keith[@]thunderhost[.]com with the size of your community and we might be able to assist

bourbonmakesitbetter ,

Thanks. I need to talk to a few other people first, but my current hypothesis is that Frendica is a better fit. The target audience is "old" (i.e. my age) and really likes Facebook. Some (most?) of them don't even know what Reddit is. I'll need to demo kbin for a few people to see if we think the idea has legs.

LollerCorleone ,
@LollerCorleone@kbin.social avatar

Something like this already exists for Mastodon (check out masto.host, and there are also a few others). But a service supporting all fediverse softwares would be cool.

bourbonmakesitbetter ,

Yeah, I've had no trouble finding hosts for Mastodon. Being the oldest (?) and most popular service (at least until now, not sure what the current stats are) has its advantages.

cendawanita , (edited )
@cendawanita@kbin.social avatar

@LollerCorleone it's not online yet, but Chris Trottier is developing spacehost with that purpose iirc.

@haubles @vvuksan @renchap @piotrsikora @ernest @Nougat @bourbonmakesitbetter

Sam_uk ,
@Sam_uk@kbin.social avatar

@bourbonmakesitbetter elest.io provide a service like this kbin should be added in a few days

@haubles @vvuksan @renchap @piotrsikora @ernest @Nougat

bourbonmakesitbetter , (edited )

Nice, thanks! They have Friendica now, which I think is what I'll really need based on my potential audience.

Badabinski ,

Hey @Ernest and @piotrsikora,

I haven't looked too closely at how kbin is architected yet, but would it benefit from horizontal scaling? I do full-time development of tooling to administrate very large k8s clusters for a company that you've probably interacted with today without knowing it. Not sure if k8s is the right orchestration system for you, but I'd be more than happy to provide some input on a potential migration to k8s (if kbin is a good fit there). I know there's a community on Matrix as well — I'll try to reach out there too, although it may be a bit.

VerifiablyMrWonka ,
@VerifiablyMrWonka@kbin.social avatar

If the post is anything to go by it's using the included "mostly for dev work only, mostly" docker-compose files. It would absolutely be able to be scaled out since at it's core it's just a webapp with workers. The app is already configured to use Redis for session storage so should be able to go super wide.

Only limitation is how performant you could make your postgres cluster.

ernest OP ,
@ernest@kbin.social avatar

Bullseye

bumbly ,
@bumbly@readit.buzz avatar

I was thinking the same thing. Shouldn't this be one of the cases where k8s shines with a horizontal autoscaler? Wouldn't want to manage your own k8s though, so I imagine managed k8s is the best option. If it's the cheapest option is another question.

@Babinski do you know if there are other horizontal autoscaling options besides k8s?

BiggestBulb ,
@BiggestBulb@kbin.social avatar

I'm not familiar with the architecture of the app (nor where it's hosted), but if it happens to be on AWS then you should be able to spin up an ECS cluster (especially since it's already containerized) and load balance it that way with an ALB configured during setup. Imo that would be the fastest way to do it (again, assuming this is on AWS)

Trebach ,

I read somewhere that it's on Hetzner.

Badabinski ,

As @BiggestBulb said, most cloud providers have container platforms that support horizontal scaling, although generally not as elegantly as k8s (imo, others may disagree). Also totally agree about managed providers. EKS, AKS, and GKE weren't suitable for what we use k8s for (very large shared clusters) until recently, so we've been administrating our own custom k8s distro. The managed stuff has gotten a lot better, and I'd definitely recommend that for running kbin. Running k8s yourself is hard, etcd is an evil bastard. I've had plenty of chances to see what works and what doesn't in my role, however. There are some development/deployment patterns that are robust, and there are many that are not.

piotrsikora ,
@piotrsikora@pol.social avatar

Hi @Badabinski

K8S is one of option, but we decided to use some mix of bare-metal and docker swarm.

Almost everything is prepare to grow horizontally. Only (like always) problem is in database, and also we want to have flexible software that run on big cluster and small node without changes in code.

Give us few days, and after that we will show something ;)

@haubles @vvuksan @renchap @ernest

Calcharger , (edited )

I'm so confused. We only have 3500 accounts on Kbin? So we are miles and miles less than Lemmy?
edit No, I just need coffee

lixus98 ,
@lixus98@kbin.social avatar

Nope, we are at 35k+

Calcharger ,

I need coffee

ernest OP ,
@ernest@kbin.social avatar

Edited ;)

Calcharger ,

Hah, thanks Ernest!

DarkThoughts ,

That might also need an indicator to avoid abuse.

MeowdyPardner ,
@MeowdyPardner@kbin.social avatar

It's cool to see renchap helping out here in addition to his work with mastodon.social! The amount of cross-pollination and coordination within the fediverse is so cool to see.

Rycaut ,
@Rycaut@mastodon.social avatar

@MeowdyPardner @haubles fascinating reading comments like this and having to figure out the context for “here” (as I read it via Ivory with my mastodon.social account)

One of the things that will have to evolve in our discourse online in the fediverse is the distributed nature of “here”

nicetriangle ,
@nicetriangle@kbin.social avatar

[Thread, post or comment was deleted by the author]

  • Loading...
  • ReCursing ,
    @ReCursing@kbin.social avatar

    Sometimes the world is divided into before and after. For Earnest this must be one of those moments

    BaroqueInMind ,
    @BaroqueInMind@kbin.social avatar

    Hell yes, ernest. Keep up the momentum, stay humble of your shortcomings, and don't burn yourself out.

    We all love this community you developed, and hopefully we can attract the kinds of people who are as open, humble, courageous, and intelligent as you seem to have gravitating towards you in your personal and professional life.

    Friend ,
    @Friend@kbin.social avatar

    Look how far we've come in just a week :)

    eric ,

    thanks for proving the community this - really enjoying this place <3

    jesus_fish ,

    funding...moderation...why does this sound familiar?

    NotTheOnlyGamer ,
    @NotTheOnlyGamer@kbin.social avatar

    Same song, new instruments, that's all. Just brace for the ads and the monetization. Be ready to be a customer rather than a user, or be ready to move on.

    Deref ,

    Do you think Ernest should run and develop this whole site out of his personal income?

    NotTheOnlyGamer ,
    @NotTheOnlyGamer@kbin.social avatar

    You know, I'm honestly not very concerned about that. As long as it's not money or energy I have to expend, I'm fine with whatever needs to happen, including the 'move on' option.

    bourbonmakesitbetter ,

    The Fediverse needs to get a handle on advertising and revenue, and what's considered acceptable and not acceptable. It's easier to do this now than after a bunch of bad actors have shown up and ruined things. My initial idea: a standard API that provides the contract between the service and its users. It would include things such as how, where and when advertising is used in that instance, whether you can opt-out, and how to do so, operating costs, revenue sources and relative amounts, etc. This API could be leveraged to aggregate lists of instances that end users could then sort and filter to find those that meet with their individual values. It could also become part of the standard UI so that users could easily locate where to support the operator of various instances, instead of relying on posting messages and hoping somebody knows where to donate (and that they are providing the true information).

    Yes, I do realize there's a bit of a "herding cats" aspect to this. And there are those who are just completely opposed to any sort of "monetization" at all, but to me that's just an argument in favour of making it as transparent and discoverable as possible.

    Kierunkowy74 ,
    @Kierunkowy74@kbin.social avatar

    This is a slippery slope. Reddit used to be open-source...

    bourbonmakesitbetter ,

    First, just because reddit fucked it up doesn't mean that's the only path forward.

    Second, if the Fediverse community doesn't address advertising, somebody will. Probably somebody with deep pockets, and it won't be in the way anyone other than the advertisers wants it. Sure you can defederate them, but that's most likely going to result in a fragmented Fediverse with 95% of the users in a corporate walled-garden and 5% of users in a free (however you want to define "free) but content-poor open world.

    I'm not suggesting that advertising should be pushed between instances. Quite the opposite: advertising (if allowed) would be up to the individual instance to decide if, when and how users of that instance (i.e. those that are browsing that instance using that instance's UI/application/website) would see advertising. Don't like ads? Sign up with a different instance. The API I've suggested is not for publishing ads. It's purely informational to enforce transparency about how/where the funds to run an instance come from, and, if you so desire, tell you the official way to directly provide financial support. That information could also be used when subscribing across instances, to warn of potential bias, or about their advertising policies if you decide to visit directly. I'm basically suggesting a way to encode the suggestions in https://www.eff.org/deeplinks/2022/11/fediverse-could-be-awesome-if-we-dont-screw-it, point #6 in a machine-readable way.

    Kierunkowy74 ,
    @Kierunkowy74@kbin.social avatar

    First, just because reddit fucked it up doesn't mean that's the only path forward.

    Hundreds of thousands of people on Fedi are yet to be convinced to this. Your opinion is unpopular, and any try do combine fedi with funding by advertisements could likely break the fediverse rather than make it more sustainable.

    Upvoted your comments inter alia for courage.

    bourbonmakesitbetter ,

    Hundreds of thousands of people on Fedi are yet to be convinced to this. Your opinion is unpopular, and any try do combine fedi with funding by advertisements could likely break the fediverse rather than make it more sustainable.

    I'm not sure what "opinion" you are referring to. I'm not advocating that I want advertising or commercial entities running the majority of the Fediverse instances. Or is it my suggestion that it's possible to avoid that fate, even while suggesting that it's inevitable that advertising will eventually arrive?

    Really my main point was we need to have some way of ensuring that instances are able to pay their operating costs. At least until unicorn farts and rainbows are accepted in exchange for hosting, bandwidth and technical services.

    ernest OP ,
    @ernest@kbin.social avatar

    I receive support from the people here, and I also want to clearly communicate and report expenses. I feel a sense of responsibility. Many magazines have been abandoned, and that also needs to be addressed.

    NewBorn ,

    I have some magazines already. Just saying, I will take care of it and contribute a lot. But better moderation tools are soon needed. I can't even delete someone's posts in my magazines. I feel useless :v. I contribute a lot to lemmynsfw.com as of now

    VerifiablyMrWonka ,
    @VerifiablyMrWonka@kbin.social avatar

    post > more >moderate > delete?

    NewBorn , (edited )

    It only throws to the error page. It's not fixed yet, I guess. Even marked as NSFW didn't work anymore.

    Edit: I can delete thread now

    Nepenthe ,
    @Nepenthe@kbin.social avatar

    D:

    I've only ever deleted my own posts, so i assumed it was in working order. If that's the current case, I may rethink how much I've been wanting subscribers. That's a recipe for disaster atm

    Badabinski ,

    I've said it before — I'm not sure if you have it roadmapped, but I'd be really happy to contribute $5-25 on a monthly basis using Patreon/Open Collective to help pay for ongoing development and infrastructure costs. I really love what you've built here, and I'm super excited to see where it will go.

    ernest OP ,
    @ernest@kbin.social avatar

    I tried with Open Collective, but my application was rejected. I will try again at some point in the future.

    Badabinski ,

    Drat! That's super disappointing. I don't know why they rejected your application, since you seem like their target demographic (FOSS developer running a community-funded alternative to a proprietary platform). Some gentle community outreach might get them to reconsider, but I'd also be worried about people being jerks.

    Well, it's monthly coffees for now! I'll make sure I have something in my calendar.

    hillbicks ,
    @hillbicks@feddit.de avatar

    Liberapay could be another alternative, lemmy is set up there as well.

    EROLoLICON ,
    @EROLoLICON@kbin.social avatar

    I would also like a subscription type of donation like badabinski suggested. Preferably in EUR.

    themadcodger ,
    @themadcodger@kbin.social avatar

    Don't listen to the naysayers. The vast majority of us are appreciative of what you're doing here, and thankful for the work you've done so far. 600+ donations to back that up! We're happy to help contribute specifically because we're not the product.

    Kichae , (edited )

    Or spin up your own instance. Pay the $6/month it'll cost you, and never worry about ads and admins exploiting you for profit. You always have an out here.

    Plus, it's not actually that expensive. Centralized services have a problem because not only do they need to cover hosting costs, but also development costs, moderating costs, and then they need to answer to shareholders and provide a return on investment. Here, everything's being done by volunteers. Development, hosting, etc. None of it is being done for profit, by businesses.

    Look at it this way. If there are, say, 30k active users on the server, and hosting costs are 100 Eur/month, then it just takes 20 people donating 5 Eur/month to keep things going. And if every server has similar costs and similar benefactors, then this space is golden.

    Costs, of course, will be higher over time, but there are ways to share them. Media sharing is already a thing, and if more media shares crop up, costs can be kept pretty low and people with resources and a love of community can keep things going indefinitely.

    You just have to trust a small fraction of users to actually support not being the product.

    taigaman ,
    @taigaman@kbin.social avatar

    Yeah, don't even get me started about all the ads on Wikipedia. /s Servers cost money. Money has to come from somewhere, that doesn't mean ad dollars, but likely donations/grants. Moderation is a given for any public space because people could start posting CP, and I don't think anyone wants that.

    Kichae , (edited )

    Infrastructure costs money. Communities need to remove bad actors. These are just facts of life.

    Communities require continual investment. This is as true online, as it is for in-person communities, so it's important for people to donate when and if they can, if they find value in a community. The key thing is, a community needing to be invested in is very different from needing, or wanting, to turn a profit. In fact, community and profit-seeking are kind of mutually exclusive.

    The profit motive inevitably strangles the community.

    Badabinski , (edited )

    There are no free lunches. You want a nice platform? You've gotta pay for it somehow. You want a platform not filled with spam? You gotta keep it clean somehow. Funding and moderation. I really hope that kbin can survive off of donations, and I have already financially contributed fairly heavily in pursuit of that. Ernest is doing the right thing by communicating about these issues early, and often. Part of why Reddit got so fucked up was that there was no community engagement when it came to raising money. That's the nature of all for-profit platforms IMO.
    This is where federated, FOSS platforms can do better.

    KnittingTrekker ,
    @KnittingTrekker@kbin.social avatar

    Thank you!

    Saturdaycat ,
    @Saturdaycat@kbin.social avatar

    Thank you for the update @ernest ! Do you have a webpage for donations ?

    Whirlgirl9 ,
    @Whirlgirl9@kbin.social avatar
    Saturdaycat ,
    @Saturdaycat@kbin.social avatar

    Thanks!

    Girlparts ,
    @Girlparts@kbin.social avatar

    This link needs to be on the sidebar or something

    melroy Admin ,
    melroy avatar

    Doesn't accept debit cards..

    CantBeBothered ,

    Setup a virtual card linked to your debit, safer anyway.

    Zana ,
    Saturdaycat ,
    @Saturdaycat@kbin.social avatar

    Done!

    trent ,
    @trent@kbin.social avatar

    Thank you ernest... and fastly... and welcome aboard Piotr!

    OSPs make me so happy :)

    brandonsh ,
    @brandonsh@kbin.social avatar

    Thanks for the update, and keep up the amazing work!

  • All
  • Subscribed
  • Moderated
  • Favorites
  • random
  • kbinMeta@kbin.social
  • updates
  • testing
  • tech
  • drbboard
  • programming
  • til
  • wanderlust
  • bitcoincash
  • Sacramento
  • All magazines