One foot planted in “Yeehaw!” the other in “yuppie”.

  • 4 Posts
  • 29 Comments
Joined 1 year ago
cake
Cake day: June 11th, 2023

help-circle






  • yeah…

    They asked for easy, or newbie friendly - and didn’t particularly mention privacy concerns.

    Other than that, if they don’t have a port 80/433 ingress from their ISP there are scarce simple solutions that don’t require another server that also needs management, either by them or a corporate entity.

    back when i was on a DOCSIS modem, i noticed concurrent downloads would disrupt uploads and vice versa. i think this may depend on the type of connection OP has.

    I used to work at a cable company, that was either a problem that people with low SNR had. Either from external factors (tree branch on a cable line) or in-home ones (bad splitter). A modem will ramp up it’s gain in order to offset this (to a point), and in so doing, create a lot more interference between channels. OR they were hitting their ingress rate limit (which is quite agressive on residential plans because DDOS’es). It’s surprisingly easy to hit your ingress rate limit for modern http/https webservers hosting complex web apps. Lots of concurrent connections open up to try to download all the resources when you go to any website in a modern browser and while it’s not a TON of data, the short period of time causes the traffic to easily hit the PPS/BPS rate limit that ISPs employ.

    But yeah, it all depends on the ISP.


  • I’d argue that the cloudflared daemon is even easier to use than a static wire guard or openvpn tunnel. It’s basically set and forget. The downside is that you must use cloudflare. This may, or may not be a big deal depending on OPs needs.

    I moved from a place with symmetrical gigabit to “gigabit cable” with 30mbps upload, it definitely wasn’t good enough for my small family. Photos are quite large these days - not to mention videos. Though it likely has a lot more to do with the bandwidth shaping my ISP does than the 30mbps rate.

    Also agree that it’s not perfect, but very likely the most newbie friendly solution at the moment. Especially from a deployment scenario vs going piecemeal.



  • The best “bang for the buck” in your use-case is to use Nextcloud - Nextcloud Talk is your Jitsi replacement, and the files feature can be extended with the Nextcloud Photos plugin (https://github.com/nextcloud/photos).

    As for your domain question:

    1. You should use any computer you’d like that meets the Nextcloud recommendations, the key is of course isolating this machine on your home network so any “funny business” stays on the server. You can do this with VLANs or an entirely separate LAN connected to a different WAN (ISP).

    2. Many places, I like porkbun.com for real custom domains for cheap, but for your use case, you might be able to use a Dynamic DNS provider for free. It just likely won’t be an easy to remember URL (or at least, as easy as a root domain only). If you have a newer ASUS or Netgear router/modem they both have Dynamic DNS built in and you can select from a few different providers that have both free and paid tiers. ALSO it might be better to use Google Domains (now squarespace domains) since, IIRC, many DynDNS configs for routers support Google Domains too. Cloudflare can also be a decent registrar, and I’d recommend using them if you use any other cloudflare services (see below).

    3. Other things to consider: Your ISP may block port 80, meaning lots of issues. If this is the case, you might want to use a tunnel of some sort. Cloudflare has a great solution here. Even if they don’t block port 80, they may aggressively throttle and shape your incoming traffic - causing issues. Again, the tunnel is a good solution here. And, of course, your upload bandwidth matters a lot, you’ll need something around 100Mbps upload for a decent experience when accessing your stuff over the internet. The 30Mbps that’s typical of DOCSIS modems won’t cut it. Outside of these concerns it’s all about making sure you isolate your server from your “home stuff” to keep things secure.







  • On a technical level, user count matters less than the user count and comment count of the instances you subscribe to. Too many subscriptions can overwhelm smaller instances and saturate a network from the perspective of Packets Per Second and your ISPs routing capacity - not to mention your router. Additionally, most ISPs block traffic traffic going to your house on Port 80 - so you’d likely need to put it behind a cloudflare tunnel for anything resembling reliability. Your ISP may be different and it’s always worth asking what restrictions they have on self-hosted services (non-business use-cases specifically). Otherwise going with your ISP’s business plan is likely a must. Outside of that, yes, you’ll need a beefy router or switch (or multiple) to handle the constant packets coming into your network.

    Then there’s a security aspect. What happens if you’re site is breached in a way that an attacker gains remote execution? Did you make sure to isolate this network from the rest of your devices? If not, you’re in for a world of hurt.

    These are all issues that are mitigated and easier to navigate on a VPS or cloud provider.

    As for the non-technical issues:

    There’s also the problem of moderation. What I mean by that is that, as a server owner you WILL end up needing to quarantine, report, and submit illegal images to the authorities. Even if you use a whitelist of only the most respectable instances. It might not happen soon, but it’s only a matter of time before your instance happens to be subscribed to a popular external community while it gets a nasty attack. Leaving you to deal with a stressful cleanup.

    When you run this on a homelab on consumer hardware, it’s easier for certain government entities to claim that you were not performing your due diligence and may even be complicit in the content’s proliferation. Now, of course, proving such a thing is always the crux, but in my view I’d rather have my site running on things that look as official as possible. The closer it resembles what an actual business might do, the better I think I’d fare under a more targeted attack - from a legal/compliance standpoint.



  • “Your application” - the customers you mean. Our DB definitely does it’s own rate limiting and it emits rate limit warnings and errors as well. I didn’t say we advertised infinite IOPs that would be silly. We are totally aware of the scaling factors there and to date IOPs based scaling is rarely a Sev1 because of it. (Oh no p99 breached 8ms. Time to talk to Mr customer about scaling up soon)

    The problem is that the resulting cluster is so performant that you could load in 100x the amount of data and not notice until the disk fills up. And since these are NVME drives on cloud infrastructure, they are $$$.

    So usually what happens is that the customer fills up the disk arrays so fast that we can’t scale the volumes/cluster fast enough to avoid stop-writes let alone get feedback from the customer in time. And now that’s like the primary reason to get paged these days.

    We generally catch gradual disk space increases from normal customer app usage. Those give us hours to respond and our alerts are well tuned. It’s the “Mr. Customer launched a new app and didn’t tell us, and now they’ve filled up the disks in 1 hour flat.” that I’m complaining about.


  • It is definitely an under provisioning problem. But that under provisioning problem is caused by the customers usually being very very stingy about what they are willing to spend. Also, to be clear, it isn’t buckling. It is doing exactly The thing it was designed to do. Which is to stop writes to the DB since there is no disk space left. And before this time, it’s constantly throwing warnings to the end user. Usually these customers tend to ignore those errors until they reach this stop writes state.

    In fact, we just had to give an RCA to the c-suite detailing why we had not scaled a customer when we should have, but we have a paper trail of them refusing the pricing and refusing to engage.

    We get the same errors, and we usually reach out via email to each of these customers to help project where their data is going and scale appropriately. More frequently though, they are adding data at such a fast clip that them not responding for 2 hours would lead them directly into the stop writes status.

    This has led us to guessing what our customers are going to end up at. Oftentimes being completely wrong and eating to scale multiple times.

    Workload spikes are the entire reason why our database technology exists. That’s the main thing we market ourselves as being able to handle (provided you gave the DB enough disk and the workload isn’t sustained for a long enough to fill the discs.)

    There is definitely an automation problem. Unfortunately, this particular line of our managed services will not be able to be automated. We work with special customers, with special requirements, usually fortune 100 companies that have extensive change control processes. Custom security implementations. And sometimes even no access to their environment unless they flip a switch.

    To me it just seems to all go back to management/c-suite trying to sell a fantasy version of our product and setting us up for failure.


  • That is exactly what we do. The problem is that as a managed service offering. It is on us to scale in response to these alerts.

    I think people are misunderstanding my original post. When I say that customer cluster will go into stop writes, that does not mean it is not functional. It is an entirely intended function of the database so that no important data is lost or overwritten.

    The problem is more organizational. It’s that we have a 5 minute SLA to respond to these types of events and that they can happen at any random customer impulse.

    I don’t have a problem with customers that can correctly project their load and let us know in advance. Those are my favorite customers. But they’re not most of our customers.

    As for automation. As I had exhaustedly detailed in another response, we do have another product that does this a lot better. And it’s the one that we are mass marketing a lot more. The one where I’m feeling all the pain is actually our enterprise level managed service offering. Which goes to customers that have “special requirements” and usually mean that they will never get as robust automation as the other product line.


  • Our database is actually pretty graceful. It just goes into stop writes status. You can still read any data and resolving the situation is as easy as scaling the cluster or removing old records. By no means is the database down or inoperable.

    Essentially our database is working as designed. If we rate limited it further then we have less of a product to sell. The main feature we sell of our database technology is its IOPS and resiliency.

    Further, this is just for a specific customer, it has no impact to any other customers or any sort of central orchestration. Generally speaking the stop writes status only ever impacts a single customer and their associated applications.

    Also, customers can be very stingy with the clusters they are willing to buy. We actually are on poor terms of the couple of our customers who just refuse to scale and just expect us to magic their cluster into accepting more data than its sized for.