Feature Request: Bypass/Override DNS During Install

Hello /e/ Self-Hosters!

I attempted to perform an install of the eCloud self-hosted option as listed here: README.md · master · e / infra / ecloud-selfhosting · GitLab.

The problem I quickly ran into is that eCloud does DNS checks during the initial install to ensure that e-mail will flow normally. Doing checks makes perfect sense, however the install script errors out and seems unwilling to proceed if the DNS checks do not return the expected values. While I have all of the A records and CNAME aliases pointed properly, my residential ISP doesn’t let me set PTR records. The script refuses to proceed as a result.

In my case (and I doubt I’m unique), e-mail service is one of the least important aspects of my self-hosted instance. I was unsuccessful in setting PTR records in the HOSTS file, so my workaround for this matter is to spin up my own DNS server to give eCloud the correct answers to get past this point in the installer.

I would submit that DNS checking should not be a failure condition without a bypass option. A better solution would be to give the user the ability to retry the DNS test (and get a readout of which records query properly), as well as a “proceed anyway” option that allows the installer to continue anyway, perhaps with a warning that there may be connectivity issues.

Thank you for your consideration in this matter.

Hello,

I’m wondering: if you don’t care about email services, why don’t you just install a standard Nextcloud instance?

/e/ Cloud is designed to provide a full featured solution, including email, running on a cloud server with a public IP address and associated DNS domain entries (see README you linked). Running it without email capabilities is just a waste of time and resources.

Anyway, I agree with your “retry” proposal (not the “proceed anyway”): actually, in case of failure at DNS check step, the best solution is to throw away the whole /mnt/repo-base dir and restart from git clone…

Hi there Sylvian!

I completely agree that e-mail should be an available component in the /e/Cloud server install process. However, self-hosting is going to end up with cases like mine, in one form or another. None of the ISPs in my area will set PTR records on residential accounts, and only one allows them on the business accounts. Your solution of setting it up on a “cloud server” (i.e. VPS or IaaS provider like AWS or Azure) is understandable, but defeats at least a part of the reason why self-hosting is appealing: I want it running on my hardware. At least one practical reason for this is storage; I can handle my own storage costs without having to deal with large S3 bills. Another is principle: I’d like to use the integrated password manager and would rather my data not live on a server next to LastPass.

Other cases might involve using third party spam filtering; Miracast and Barracuda both require MX records pointing to them first; the mail flow may be set up perfectly but wouldn’t pass the DNS check without a change/run/change back shuffle, again, pointless work. DDNS services might be used that themselves do ultimately point properly, but the intermediate traffic flow would fail the validation despite passing traffic correctly. I might also want to set up mail flow to use a local relay or SMTP2Go or some similar service that can relay outbound mail.

In terms of “why without e-mail”, I never said I was opposed to having mail hosting as an option. I’m simply acknowledging that it’s not a primary consideration. Even if I made all the required DNS changes listed, including the PTR record in question, the DNS check doesn’t validate the SPF or DKIM records, so mail sent from such an instance is almost guaranteed to be flagged as spam by a recipient, limiting its utility anyway. I’m certainly unopposed to e-mail being an option, even a recommended option, but I submit that there are enough use cases where a disparity between “expected DNS records” and “working DNS records” will be present that a bypass option should be possible.

In terms of “just use Nextcloud”…okay fine, that’s an option, but to get the same OnlyOffice integration and branding and other functionality of an /e/Cloud instance, it’d probably take about as much work as setting up a DNS server to force the test to proceed. Similarly, the documentation for /e/OS alludes to this setup, but if there’s an explicit procedure list, I was unable to come across it.

Hopefully this makes at least some sense.

Hi,

Could you please elaborate on this? At end of a successful setup, a DKIM key is displayed to be inserted in DNS zone: https://gitlab.e.foundation/e/infra/ecloud-selfhosting/-/blob/master/scripts/postinstall.sh#L113.
I’m not aware of SPF/DKIM problems related to residential IP addresses and could not find any further information; if you have any feedback on this I’ll be happy to learn :slight_smile:
IMHO the biggest problem would be with residential IP address: some spam filtering solution would look at the originating AS number and flag messages if they come from residential ISP. Then your IP address is likely to be blacklisted :frowning:

As for spam fighting, if you use self-hosted /e/ Cloud as provided it comes with RSpamd, which is IMHO one of the best (maybe the best) Open-Source solution at this time. I recently proposed a MR to allow admins to manage blacklist, which was accepted and merged (tracking issue https://gitlab.e.foundation/e/backlog/-/issues/5292).

Back to your Cloud, at first please be aware that OnlyOffice is no longer included with /e/ Cloud (it’s been a while).

Assuming you already own a DNS domain, after reading your message (which make great sense :wink: ) I could see 3 solutions:
1/ hybrid solution: rent an entry-level VPS (i.e. CX21 @Hetzner will only cost you the price for a good beer, monthly); then install a standard /e/ Cloud and expand storage with your home-hosted (https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/external_storage_configuration_gui.html). As mentioned before no OnlyOffice OOTB, but it should be possible to connect an existing to your /e/ Cloud (never tested here …), please see 2/
2/ all-in-home solution: give up on self-hosted email, install a standard Nextcloud at home, then connect it with home-hosted OnlyOffice. You can apply some /e/ or Murena branding to your NextCloud if you like
3/ continue with the actual solution (standard /e/ Cloud at home). But I sincerely doubt that /e/ Team will accept your DNS check bypass request. Anyway it could be interesting to see if everything continue to work fine, through time!

For solution 2/ you may find difficult to connect a /e/ device to it. Simply ignore the integrated /e/ Accounts Manager and install NextCloud client app.
For solution 3/ you may see from docker stats that the eager RAM container will be mailserver, and this is due to clamd, which is useless to you…

About passwords storage: all serious solutions come with a client-side E2E encryption (see for example https://git.mdns.eu/nextcloud/passwords/-/wikis/Users/Encryption). So, IMHO, the problem is not where the database is stored, but how secure is your client (for example your client computer & browser).
However, as risk zero doesn’t exist, I’ll strongly advise not to mix password storage with “daily activities” storage. You may find something like this interesting: https://bitwarden.com/help/install-on-premise-linux/.

About NextCloud branding: if you want to apply /e/ or Murena branding to an existing Nextcloud, you may be interested with what happen “under the hood” when firing up a /e/ or Murena Nextcloud.
First, look at the Docker file: starting with a standard official image, some apps are added, then patches and customization. You may pick up some of them.
Note that the apps installation is done here: https://gitlab.e.foundation/e/infra/ecloud/nextcloud/-/blob/main/custom_entrypoint.sh.
Then, at last, some configuration is applied here: https://gitlab.e.foundation/e/infra/ecloud-selfhosting/-/blob/master/scripts/postinstall.sh. As for Docker file, many are related to /e/ Cloud infrastructure (i.e. email) and will be useless for a standalone NextCloud.

As a side note, I wrote something about securing your server: Securing your self-hosted using CrowdSec

Sure thing!

As well it should! I intentionally stipulated that these things were not a part of the initial DNS check script. The server software won’t refuse to install if you don’t have a DKIM key, but it will refuse to install if you don’t have a PTR record, even though the absence of both could adversely affect mail delivery. Yes, check. Yes, recommend. Yes, warn. …but ultimately, my refrain is that an “install anyway” option should be provided.

Yes, you’re correct, SPF/DKIM records don’t have explicit issues with residential IP ranges in the same way that such issues exist with PTR records. However, it’s possible to exchange e-mails without a PTR record, as it is possible to exchange e-mails without DKIM keys. We’re talking about a script designed in a way that lacks an override under circumstances that are ultimately optional.

We agree here.

I like RSpamD. I administer one as a part of a Mailcow server I maintain. The reason I bring up Miracast and Barracuda isn’t because I believe RSpamD is incapable, it was a readily-understandable example of a scenario where a requested DNS record would fail a check, but that traffic would flow properly. Perhaps a better example is putting the A records and CNAME records behind Cloudflare, but the point ultimately remains that all I’m asking for is an “proceed anyway” option.

From the original post I made in this thread:

It’s entirely possible I could be wrong, but let’s go down the hypothetical road of what would happen if I did option 4: create my own DNS server and point the /e/Cloud instance to it, in order to feed it whatever answers it wants to proceed with the install. We’ll assume that the listed A records and CNAME records are also correct for the mail, autoconfig, and autodiscover CNAME entries the script says it needs on the domain registrar.
What would happen? E-mail would be inconsistent-at-best, I grant you…but all of the other functions of the /e/Cloud server would function properly, right? Contacts, calendars, notes, tasks, passwords, decks, activity logs, and the system config all replicate normally, right? None of these things hinge on a valid PTR record or the ability to receive inbound e-mail, correct?

That’s what I’m looking for. Now, you addressed this in option #2, and I will get to that in a minute, but my point is that I’m not proposing anything absurd, I’m proposing an approximation of the following code be added at the end of a DNS check (sorry, my shell script syntax may be a bit off):

echo “DNS check failed. We recommend resolving these DNS issues prior to install. Retry DNS check, Abort Install, or Proceed Anyway (R/A/P)?”
read var01
if var01 == “R”
then GOTO dnscheck
else if var01 == “P”
then GOTO install
else exit 0

That’s literally all I’m looking for; I’m not looking to fundamentally change the install process.

Your recommendation for the alternative is a Nextcloud instance I build from the ground up and tie to /e/OS. This is certainly possible, and it’s not entirely off the table. However, my logic is that, while I’m at least fairly competent in this sort of set up, I’d have greater trust in the people smarter than me, who wrote the install script, to not only be more effective at integrating all the components, but also to ensure that updates and additional integrations don’t break things in a way that is more likely to happen if I’m running my own separate Nextcloud instance.

You’re fully on point in this section. We agree completely, and again, I’ll concede that my example was questionable in terms of complete consistency. I appreciate the link that seems like a viable means of keeping my data local while the code runs on a VPS (Hetzner seems really cool)…but forgive me sounding like a broken record…the reason this solution is being proposed is because of a break condition in the installer that doesn’t account for valid configurations that won’t pass it.

Your post is a font of information; I appreciate the provided links and Crowdsec is likely to be implemented in my instance, regardless of where it lives.

I do sincerely appreciate your time and thoughtful responses here.

1 Like

Good news! I found the solution.

If anyone else finds themselves in my scenario, the relevant lines are 150 through 154 in the init-repo.sh script. By using a text editor, I commented those lines out and was able to allow the script to proceed despite the absence of a PTR record.

I hope this helps someone else!

1 Like

As long as you don’t need email interaction with foreign DNS domains, yes they probably will.

Happy to read that you found a solution that matches your needs :slight_smile:

As a side note : you can use the integrated RainLoop as a client for a foreign domain, if you declare it in RainLoop Admin.