Suggestion: Murena mirror in case of outages

Hi All,

Murena cloud has a huge advantage over Google etc. in that it is based on Nextcloud. It shouldn’t be hard to offer users a way to switch to an alternative Nextcloud server (mirror) in case of an outage. In fact, before I used murena, I used Lineage OS with my own Nextcloud server.
What do you think?

Regain your privacy! Adopt /e/OS the deGoogled mobile OS and online servicesphone

4 Likes

I don’t know if it feasible, but it sound appealing …

So, I’ve been holding my tongue because I didn’t want to come across as elitist or anything, because I know this outage is adversely affecting lots of people and I want to be sensitive to my fellow community members.

…but this outage hasn’t affected me in the slightest, because I use the amazing self-hosted implementation of /e/Cloud, which lives on my hardware and for which I handle my own backups, plus it allowed me to add in additional Docker containers to have my own metasearch engine and photo backup functionality. You noted that you ran your own Nextcloud instance in the past, and I would submit that you can still do so with /e/OS, in an excellent way.

In terms of having users doing their own replication, I love the idea, but I’m not quite sure how that would work in practice, primarily due to DNS. In the event of a failure, end users would need their phones to connect to their own respective instances, so that you see your own contacts, and just as importantly, you don’t see mine :slight_smile: . That’s a bit rough to do when phones point to murena.io by default, though there might be a way to add a failover address within the account setup, if it was something the team as interested in pursuing.

The bigger issue would be mail flow; it’s impossible to have the MX records make mail flow to 1,001 different users’ mail servers and then replicate back again. The email outage wouldn’t be solved in this way. At work, we handle this by routing mail through Barracuda’s email protection service, but that’s basically impossible to offer, at scale, for free, and there are other concerns on that front that would make it a near impossibility to leverage a similar store-and-hold service for Murena mail, especially if free tiers are still expectations.

If the implied solution is for Murena to implement more resiliency within their infrastructure, that’s a part of the reason why the restore process is taking so long - they’re using the unexpected down time to add more fault tolerance, but there are limits, and the choice was made to have one large outage rather than two somewhat-smaller ones, which would have likely been longer than the singular lengthy one here.

In conclusion, the ability to run my own /e/Cloud server is the single biggest selling point, and why I stick with /e/OS over other builds for my phones…for those that can spend some money for a VPS slot, or can run their own server in a rack somewhere, it’s amazing. In terms of DIY failover…that might be a bit too tall an order, and probably the worst of both worlds - Who wants to spend money running a replica destination, hoping it works because it’s impossible to test a controlled outage, and it’ll never be an email backup anyway?

Hope it brings some options and perspective.

7 Likes

Dear voyager529,
Your post is very interesting, thank you.
Do you know if there is somewhere on the web a page indicating how to install a self-hosted implementation of /e/Cloud on personal hardware? I would greatly appreciate that, and it certainly will be of interest to a large audience here.
All the best
ltdm

3 Likes

Do you know if there is somewhere on the web a page indicating how to install a self-hosted implementation of /e/Cloud on personal hardware?

Good morning friend!

I’ll agree that the self-hosted option should be more prominently discussed and linked on the website, but you can find it here: e / infra / ecloud-selfhosting · GitLab

If you’re new to Docker or Linux or DNS or certificates…prepare to spend a weekend or two learning all of that first, and if you don’t have a domain name or static IP service for the internet connection you’re using (assuming you’re not using a VPS since you stipulated ‘personal hardware’), get that sorted.

Also, if you’re using residential ISP service, you’ll probably need to talk to your ISP to get a few things cleared up: you’ll need to make sure you’re not behind CGNAT, and you’ll have to ask about getting ports 80 and 25 unblocked, and even then you’ll still probably need a relay from SMTP2Go or a similar service…

There’s a lot of preconditions for self-hosting your own /e/Cloud instance, which is a good part of the reason why the number of self-hosters in the community is a pretty small number (and of those that do, you’ll be the second I’m aware of who run it on their own hardware :slight_smile: ). I don’t say all this to dissuade you; just the opposite: I’d love for half the /e/OS community to be self-hosting their own instances. I say this because those are incredibly common pitfalls and because I want your self-hosted instance to be a success =).

Greetings from America, my French friend!

2 Likes

Good afternoon (here is CEST ! :wink:) my friend from over the Ocean,
Thank you very much for your answer and for taking your time while traveling from the future :wink:
I will give it a try, but certainly not any time soon. I understand from what you write that I have a lot to check and learn before I take the plunge. I’m not a newbie in linux and server administration, but I’m not a specialist either. And my ISP (orange-France) is not very receptive when users want to think outside the box
Greetings from Brittany (West of France)

1 Like

It’s GMT -5 here in New York, so I’m very much living in the past, rather than the future =).

I would submit, for your consideration, that there are a few intermediate steps that might be worth taking before you take the plunge (some of these, it’s entirely possible you’ve already done):

  1. start by spinning up a Docker host with Portainer CE.
  2. Go through the awesome self-hosted list and see if there are any applications that seem interesting to you. The list varies greatly in quality; some applications are very easy to deploy, others have an underdeveloped frontend…so, not every failure is a true failure. A few that I’ve used and have found easy to deploy are Gotify, The Lounge, and Technitium. The Reddit self-hosted community is also a good resource for getting started with Docker.
  3. Spin up another Docker host, without Portainer. Try installing some applications using Docker Compose. Two I have had good success with are KitchenOwl and Rocket Chat.
  4. Now, we shift to the network admin part. For starters, you’ll need to know if you’re behind a CGNAT or not. If you are, there are options, but they have tradeoffs. You can use Cloudflare Tunnels, which is free and the easiest way to set it up, but it also sends all your data through Cloudflare (Cloudflare states that the free tier enables them to better mitigate malicious traffic through more accurate training data, but it’s up to you if you believe they do more than that with all the traffic they capture). It’s possible to use a cheap VPS instance to do a point-to-point tunnel using Headscale, but now you’re adding an extra layer of complexity, as well as a monthly bill (and if you don’t trust Cloudflare, do you trust the VPS service?). Ensuring you can access the server is much more complicated than it used to be, especially if you know you’re CGNAT’d, but it’s a foundational step.
  5. If you’re still hosting Nextcloud (or other self-hosted services) at home, and you’ve got a reverse proxy installed, you must move your port forwards to the /e/Cloud VM. I mean, you might be able to get away with only giving it port 80 and letting your existing reverse proxy handle the real, public-facing certificates, but /e/Cloud gets real cranky when it can’t get replacement certificates, especially during install time. Also, get the domain part together - /e/Cloud must have its own domain, so ecloud .ltdm .com isn’t going to cut it - you’ll need to give it ltdm .com and make the necessary subdomains accordingly. Once your DNS settings look good (whatsmydns.net is a great resource for checking DNS queries).
  6. Assuming all that is sorted…you’re ready to run the scripts =). One other thing to note if you’re self-hosting on a residential or even some business ISP connections: edit the init-repo.sh script and comment out line 162. Unless you know you can edit your PTR record (which MUST be done by the ISP), you’ll want the script to continue despite the PTR record not matching, which it doesn’t do by default.

The number of self-hosters in this community is small, but we’re dedicated, and we’re happy to provide some friendly support where it’s needed.

4 Likes

Hi @voyager529 @ltdm,

My 2 cents below!

about PTR DNS record & home housing

In France, unless you have a “business” contract for Internet access, it’s very unlikely to get a customized PTR record.
Also, some ISPs are filtering out SMTP traffic.
Furthermore, the fight against spam and bad email actors it partially based on “reputation”, and using residential IP addresses can lead to serious difficulties.
@voyager529’s post Suggestion: Murena mirror in case of outages - #5 by voyager529 contains some very helpfull informations :slight_smile:

On the other hand, you can lease a cheap VPS as a starter, for example CX22 from Hetzner is only 4.55 €/month (including 20% VAT).

Please be aware of:

  • check the VPS provider against PTR editing capabilities (for example https://docs.hetzner.com/cloud/servers/cloud-server-rdns). Not all providers will allow editing.
  • Docker support: some providers won’t allow running Docker on their VPS (this is often related to the virtualization technology underneath), or can propose it only at extra cost.

about DNS

Whatever choice you make for hosting, you will need a DNS domain name.

Hetzner can also provide this, for example .eu for 12.60 €/year VAT included. Easiest choice if you also have your VPS from them (everything on the same bill).
There are plenty of actors here, for example LWS or Gandi in France.

Please be aware of:

  • prices! There’s no regulation here, and some may be very cheap for the initial registration, but become expensive from the second year on. Also, please be aware that the initial price, before discount, is not always the renewal price (for example, LWS .fr is listed at 6.99 4.99, and is to be renewed for … 8.99, not including VAT :wink: ). Hetzner is always the same price, and Gandi provides a clear listing for costs
  • length: please stick to 2 or 3 letters (for example .fr, .eu, .com, .net), 4 or more letters can cause trouble in daily use
  • cleaning: some providers will fill your DNS zone with useless records, the only you need to start your self-hosted are NS (SOA) records

about installing

I’d suggest to start with a VPS, allowing you to follow the install guide (link provided by @voyager529) and have your eCloud running, hassle-free.

Please be aware of:

  • better read the guide twice, understand the content, ask questions around, before doing anything!
  • DNS have delay for modifications (propagation time). I’d suggest to:
    – fire up your VPS (Create an Ubuntu server instance). The purpose is to get your public IP address
    – create your DNS records, change PTR, as instructed by the guide (Set your server with proper DNS settings)
    – take a break, take the dog for a walkout, get some sleep
    – then come back and continue your installation (Start bootstrap process). Install script will check your most important DNS records

Once your eCloud is proven running fine and meet your needs, you may consider migrating to home housing!

:heart_eyes_cat:

2 Likes

That has become more and more true over the last two decades!

So, if you have built 2 mail servers with a clean reputation for 23 years and a second one with 16 years – both servers had no IP address change ever – one would not want to compromise their reputation, by moving your mail server to a /e self-hosted on a VPS. Plus, having mail and e-cloud on the same server goes very much against best practice.

I have checked and asked in the past for a set up option "no mail server".
It is not needed in nextcloud, owncloud and never has been needed for them to work.
And as for integration with mobil devices, it is file sync, documents, contacts, calendar which most people are looking for, at least as far as my experience goes.

In short, I believe, it would be a good selling point for self-hosted e-cloud, to have it available without mail server integration. Less complicated, for new users and for experts who want to keep their mail server where they are.

2 Likes

So, @smu44 is the one who has been so, so invaluable in helping me get my /e/Cloud server online; he is as knowledgeable as he is patient, and I wouldn’t have gotten /e/Cloud up and running without him. there is a 100% chance that, in the process of spinning it up, you’ll run into an issue and make a forum post about it, and he’ll be on the thread. I will always thank him, publicly and privately, for the amount of help he has provided to me, and to the community as a whole, for his documentation, as well as the responsiveness with which he provides it. He is a gem of a person and a pillar of this community, and every post I have ever read of his has always been worth reading twice.

That being said, he and I have slightly different methods of achieving the same goal. Smu is a fan of VPS services, and it’s completely understandable why this is the case: for some €5/month, you can get a good-enough VPS slot to get /e/Cloud going, and to his point, he noted in his recommendation that a VPS is a good idea, even for a short term to go through the install process and get some experience in the whole process when you’re not fighting with DNS and ISP drama, then giving it another go on owned hardware later on if you’d like. I also trust his judgment regarding French ISPs and experience with attempting to self-host on a residential internet connection; it is entirely possible that the extra effort of avoiding ISP drama is a core tenet of why he prefers VPS service.

There are three reasons I’m “team hardware” rather than “team VPS”. First is fundamental: “My data is MY data” isn’t just a slogan, and principally speaking, Hetzner can technically cut off my access to the data, at which point, is it really my data?

But let’s move forward from the philosophical and look at the two practical reasons. First, is cost. Smu is correct that it’s around €5/month for a base tier for Hetzner. At that rate, it’d take 10 years to pay off the Poweredge server I bought, and that was from a refurb IT supplier, even if power and internet were free! The problem is that the referenced tier has 40GB of storage. That’s perfectly fine if most of the intended task is PIM data (email/calendar/contacts/tasks/passwords), but once you start adding photo syncing, 40GB is going to get filled up in no time - less if your wife also has an account on your /e/Cloud server and her /e/OS phone is replicating photos there, too :slight_smile: .

I’ve got 4TB of storage on my server for /e/OS (and, admittedly, Immich, which handles the syncing), and Hetzner’s 2TB storage tier is €107/month. Once that’s factored in, my server, its internet, and its power consumption is paid off in less than 8 months; half that if you include 5TB of storage :slight_smile: .

The third reason I like having my own hardware is that I also have my own UTM appliance in front of it, managing the internet. Sophos has an excellent, and free, ‘home’ license for their virtual UTM appliance. It can only use a certain number of cores, a certain amount of RAM, and is limited to 50 LAN IPs, but that’s perfectly fine because it does IDS/IPS, Geo-IP blocking, malware traffic scanning, spam filtering, and MTA relaying, giving my /e/Cloud server an extra layer of security that is difficult to replicate in a VPS environment.

Bonus round: Spamming is one of the things most VPS companies will actually terminate service for, but IP reputation fluctuates greatly. Personally, I’m spoiled by being able to use a pristine IP in an IP block that I know has an excellent reputation, but I’d recommend relaying mail through SMTP2Go or Mailjet or Sendgrid, all of which support relaying mail through a TLS-based connection on nonstandard ports, which should help get around barriers to SMTP traffic. This is a ‘bonus round’ because this setup is a good idea regardless if you’re VPS hosting or local hosting.

On his topic of using Hetzner for domain registration, it’s a solid choice. Personally, I’ve been extremely happy with Namecheap for registration; they’re pretty inexpensive to begin with, they run sales regularly, their support folks are fantastic to work with, their DNS replication time is pretty quick, and they have an excellent system for Dynamic DNS that may come in handy. You can’t go wrong either way, but since we’re name dropping, I’ll give a recommendation, too =).

And, I’ll take a minute to respond to @make-nz…I’ll certainly agree that a “mail-free” install would be a nice extension to the install script (I wonder if the Virtualmin GPL install script could provide some guidance on that?), but I think that it’s a relatively low priority, since the workaround to that would be “get DNS working once, let it go through the original install, then remove the mail-centric DNS names from the cert renewal list”. The alternate version is “spin up your own Nextcloud instance and tie your phone to that with Davx5”.

The number of self-hosters isn’t huge to begin with (most /e/OS users seem to be just fine trusting Murena with their own accounts), but the number who are unable or unwilling to work with an /e/Cloud iteration that assumes e-mail is such a small subset that…I’d almost submit it’s a bad idea to go down that road. The current self-hosted instance is still based on Nextcloud 26; the current release of Nextcloud is v31. Regression testing for the self-hosted version is already difficult enough, to have to test migration paths for both versions of an installation is going to make development time even longer, I’d assume. At first blush, I’m with you, and I do hope that one day, there are enough developers and enough self-hosters to justify spending more time making the self-hosted /e/Cloud platform the first class citizen it should be :wink: . Until then, I would submit that the alternatives (move your port forwards over for an hour, once, to handle installation, or use Nextcloud) are in a workable state and would be worth pursuing if required.

1 Like
  • would have to do that before and after every update I think
  • switching DNS, IPs… for a mail server can hit the reputation
  • /e cloud has full integration in /e – Nextcloud does not

And the numbers of self-hosted may be so low because it comes with the mail server part! Or why do you think there are so many people who have Nextcloud?
You mention it as a workaround together with Davx5… and then you add the email account you’ve many years in K9/e mail app, right?

Mail is important, complex and best left on separate instance, for so so many reasons.
When Postfix, Dovecot… are up and running, they can handle thousands of users and several million emails per month. Why should a self hosted system, mostly used for a very low number of users and emails, come with such a Goliath?

If you do mail server maintenance right, you need a lot of time to keep it maintained and when it fails, well where is you back up? If /e couldn’t fix the outage real quick and without losing emails, ask you self what this would mean for you and your self-hosted!?

When mail was easy, we had a lot of spam!

would have to do that before and after every update I think

Nope! The updates don’t do DNS checks, and there’s a separate update script to handle them.

switching DNS, IPs… for a mail server can hit the reputation

I wouldn’t think so; it’d only be 80 and 443 that would have to be moved so that the script can get the initial round of Let’s Encrypt certs, but after that, it can be moved back. It’d be for less than ten minutes, and port 25 would never need to be moved.

/e cloud has full integration in /e – Nextcloud does not

Agreed! …but part of that full integration involves a mail server.

And the numbers of self-hosted may be so low because it comes with the mail server part!

I submit that self-hosting an /e/Cloud instance is probably a minority group for some of the other reasons: the need to pay for a VPS or hardware, have a domain name, and then opt into administering those things explicitly for /e/OS services…instead of either using a free /e/Cloud account or paying for one as a way of supporting the project in a mutually beneficial way. The number of people willing to root and flash their phone already selects for a more technical audience (the rest are fine letting Apple or Google do it), but even so, I’m not entirely convinced that the mail server part is the sum total of the reason for the comparative minority of self-hosters.

Or why do you think there are so many people who have Nextcloud?

I would guess that the reason for this is that Nextcloud has a few major things going for it. First off, it’s super easy to install with even the slightest amount of desire. Synology and QNAP both have first party plugins for it. Softaculous has a one-click installer that works with most garden variety web hosts. There’s a Turnkey Linux build for it. DietPi can install it during initial setup. There’s a Proxmox LXC container script for it. There are pre-built AWS/Azure/GCC/DigitalOcean instances for it. There’s a way to have it installed, in five minutes, almost by accident, on everything from a Raspberry Pi to an IBM Mainframe. This somewhat underscores your point, in that I definitely agree that there are far fewer barriers to deployment for Nextcloud than for /e/Cloud…but I don’t think that the mail server is the sole barrier to entry there.
The second thing I would submit is the flip side of the same coin: Nextcloud is usable everywhere. An off-the-shelf iPhone 16 can sync certain data with Nextcloud through an app install, no problem. So can off-the-shelf Android. So can a PC. /e/Cloud has a far more narrow use case, and while one can probably use the /e/Cloud iteration with the standard Nextcloud client, the real benefits of the system are only really found with the vertical…so I think that the two combined lean heavily toward the numbers Nextcloud has.

If /e couldn’t fix the outage real quick and without losing emails, ask you self what this would mean for you and your self-hosted!?

A self-hosted instance will inherently involve self-reliance. Freedom always involves responsibility. To answer your question, I right now use a program called XSI Backup since my self-hosted instance runs on the free version of VMWare, but my to-do list involves migrating over to Proxmox, which has its own first party backup software, but I’m more likely to be using Veeam 12, which handles Proxmox backups even better (including offsite backup, which is the goal). If my /e/Cloud VM goes down, any incoming mail is held on my Sophos UTM appliance until get it up and running again. If there’s a hardware failure, well…you’ve inspired me to invest in a new-to-me server as a part of the migration process, leaving the old one to be the failover volume =).

When mail was easy, we had a lot of spam!

True, but now the overwhelming majority of people have all their mail handled by Google. I’m not quite sure the trade was completely worth it.

We are way off topic, we agree on how easy it can be to run Nextcloud at home, we agree e-cloud is mostly Nextcloud, we agree murena can not redirect your email in case of outages.

Where I disagree is that, per default, any one testing/starting with self-hosted should have to deal with a mail-server.
If one gets to “yeah, I got it to work” but still has it running on port 25, it’s just not ideal.
And those, who have the knowledge on running a mail-server well, will say “no thanks, I got my mail server already”.

And again, we are way off topic and I write all this in a good spirit.

Thank you, always a pleasure to help good people! :blush:

That was exaclty my point, answering to:

One more “selling point” for VPS: you don’t have to care about hardware problems (from simple hardware failure to complete disaster like house catching fire), electrical power problems and bills, etc. Very nice for a starter project :slight_smile:.

To be clear, I totally agree with “My data is my data” and having them at home. But often one guy have to find some kind of “balance” between costs, ease of use & maintenance, security, privacy … It’s all personal choices!

Thanks for your detailed posts @voyager529, they for sure be of help for anyone willing to house-host (does this make sens?) :smiley_cat:

One thing about Hetzner’s storage: you can use a Storage Box, for example 5TB is 13.08 €/month, VAT included :wink:


About self-hosting without email: of course, everything is possible with enough time and knowledge!
I won’t go into “pros & cons”, once again it’s all about personal choices and capabilities. BTW some users are very happy to get an “all in one” integrated solution with automated set-up and hassle-free administration :wink:

Anyway, having spent hours “under the hood”, it’s clear to me that the self-hosted solution has been designed from the very beginning with an integrated email. Including, but not limited to:

  • a central database for both email & Nextcloud accounts
  • a Welcome web site to ease account creation (as far as I know, this can be stopped without consequences)
  • some Nextcloud customization and apps by Murena, requiring (maybe even local) email server

Maybe the best solution would be to merge an existing email server into self-hosted. Interesting idea to explore!
With enough free time I could investigate on a “separated email” solution. Unfortunately, it won’t happen in next weeks, or months :confused: .

About storage reliability: please don’t compare self-hosted with murena.io. They are running at complete different scales, and use different solutions underneath.

2 Likes

I’m all onboard with that, but as the very nature of email is sending and receiving emails/data, it is one more reason to have the mail-server in its own instance.
MX DNS records exist form almost the beginning of the Internet/DNS in the 1980s

+1 from me
Also because updates to a “purely mail-server” are way easier, less frequent…

Btw: Is there any information on what they use?

To answer that… Unfortunately it would be more or less impossible, as Nextcloud does not have a mirror feature!
People often see the Federation feature and think this would be some kind of synced mirror, but it is not. It only manages access to/from other NC instances.

nextcloud - the server/client model in general - is the wrong direction to empower a majority of endusers, too heavy weight. It’s a result of thinking of devices as end nodes.

It’s inactive now, but I gleaned at the “decentralized sync” people at DecSync in the past. Just having a second device you can bootstrap from, another phone or laptop. No vps, no hobby admining - just files synced peer to peer.

This topic was automatically closed after 90 days. New replies are no longer allowed.