Install Issue: postinstall.sh + .env...phone doesn't sync photos?

Using /e/OS Self Hosting
1

Hi there friends!

It was a long journey, but I solved my problem and wanted to provide a bit of feedback that will help someone else, hopefully.

I created a completely new server from my last attempts. I learned that I since I was using DietPi, I needed to add salt-minion. I also needed to apt-get install AppArmor.

Once that was sorted, I found that the install script seemed to silently fail when it was creating the PostfixAdmin database. Long story, a rebuild, and a quick discussion with ChatGPT later, and it turned out that the /mnt/repo-base/.env wasn’t being loaded, which meant it had none of the password or system variables needed to actually populate the PostfixAdmin database.

So, I ran “source /mnt/repo-base/.env” and did an “echo $DOMAIN”, which then displayed the variable declaration from the .env file. So, I went into the post-install.sh file and manually ran the commands within the script, this time ensuring it got the data it needed, and I was able to log into PostfixAdmin to make mailboxes and Nextcloud accounts, by extension.

I then set it up on my OnePlus 8T, which was sitting at the /e/OS out-of-box wizard, and pointed it to the domain name with my account…and it did connect, and contacts and calendars do sync…but some weirdness remains…

Most notably, I took a photo, but it’s not showing up anywhere in the Files section. Photos are set to sync in the Account settings, and multiple manual sync operations were attempted.

Similarly, the Mail app doesn’t seem to want to configure with the server. What’s particularly interesting is that eM Client on my computer (also on the outside of the firewall) did work properly and configure IMAP/SMTP just fine, but Mail on the phone doesn’t connect no matter how many times I try to configure it.

Thanks, as always, for any and all assistance in sorting this out. As an aside, I don’t mind starting from scratch again if I need to.

Regain your privacy! Adopt /e/OS the unGoogled mobile OS and online servicesphone

2

Hi,

Are you aware of self-hosting requirements, here https://gitlab.e.foundation/e/infra/ecloud-selfhosting#requirements and here https://gitlab.e.foundation/e/infra/ecloud-selfhosting#create-an-ubuntu-server-instance?

What gives a check using https://mxtoolbox.com/emailhealth/ and your root DNS domain?

3

Hi there Sylvain!

So, the closest thing I have to a requirement that I’m not fitting is the requirement for Debian 11; I was of the assumption/hope that Debian 12 is supported at this stage. DietPi essentially is Debian 12, with some tools to streamline things like software installs and network config and system updates. If we really are pointing to some sort of oddball discrepancy with DietPi, I’ll rebuild on Debian 11 tonight.

In terms of the MX Toolbox link, the only warnings I got reflected my known issues with PTR records and an issue with the SOA records, but everything else comes back fine and I get no actual errors.

Thank you for your continued awesomeness!

1 Like
4

Okay, gave it another go with a bog standard Debian 12.4 netinst, installed with ‘standard system utilities’ and ‘ssh server’. After booting, I added a static IP, apt-get installed git and curl, installed salt-minion, and commented out the PTR record verification in init-repo.sh…and the bootstrap script failed again. It did so at the same point, silently erroring out at ‘Creating postfix database schema’, and ‘echo $DOMAIN’ not returning the variable contents of the .env file.

…so, we can at least assume that it’s not an issue with DietPi, and that it IS consistently reproducible as best as I can tell.

5

Debian 12 is untested for self-hosted, as stated in install guide:

Installation on Debian Bullseye (11) stable works as well.

Anyway, I’ll try to fire a Debian 12.4 instance and post back (may need a couple of days)…

6

Found some spare time :smiley_cat:

I installed a fresh Debian 12.4 (amd64/netinstall) with same options as yours, then:

Then self-hosted installer ran smoothly, and everything looks OK at first glance.
I’ll keep the instance available for a few days, if you need to compare some details please just ask.
Please note I do have a valid PTR, sparing me from editing init-repo.sh.

7

@smu44 , I tried it once again and had my same outcome - script ended after attempting to create the postfixadmin schema, and ‘echo $DOMAIN’ returned blank.

This time, I did a screencap of the whole process ,from creating the VM to the point of failure. I’d rather not post a link in a public forum, since I go into the firewall config and my domain registrar and a few other things I’d rather not post publicly; might I impose upon you to accept a DM to look through the video and see where I went wrong?

8

Of course you can, thanks for asking! :smiley_cat:

9

Hi @voyager529,
I watched tour video, and currently trying to reproduce… No luck so far!

What I’m quite sure, that’s you’re probably not facing a $DOMAIN variable problem.
Let me explain: the way we launch the scripts, all environment is local to them.
A simple test:

  • create a little script (let’s call it … test.sh :smiley: ) as:
#!/usr/bin/env bash
source /mnt/repo-base/scripts/base.sh
echo $DOMAIN
  • launch it with bash test.sh, it should display your DNS domain
  • however, once the script is terminated, the exact same echo $DOMAIN in the shell you used to launch the script won’t return anything

I didn’t dig further, but as my weak memory tells to me this is the normal behavior with scripts.
To export variables to the calling shell, we’d have to use some export commands, and/or call the script with . or source.
For example, calling our test script with . ./test.sh followed by echo $DOMAIN should output our value.

Back to your problem: waiting for me to reproduce successfully your problem, it could be useful to get logs right after the install script abruptly stops (at 37:31 in your video).
It can be done with (having you current dir to /mnt/repo-base/): docker-compose logs -t postfixadmin.
Don’t be afraid with the bunch of errors at beginning, they are expected during installation. Please rely on timestamps.

I’ll keep you posted when/if I can reproduce.

10

Well… Trying to reproduce exactly as you’ve done, I got some … interesting :wink: results, but script did not hang :confused:

The long wait you had when creating Postfix schema looks like a timeout to me.
If the VM is still available, could you please try this: Unable to create account self-hosted? - #13 by smu44?

Also, I noticed some inconsistencies with Salt Minion.
If you’re willing to re-install (little advice BTW: take a snapshot just after OS install and apt upgrade+update :wink: ), please run in the exact same order as listed:

  • apt -y install curl git
  • mkdir /etc/apt/keyrings
  • curl -fsSL -o /etc/apt/keyrings/salt-archive-keyring-2023.gpg https://repo.saltproject.io/salt/py3/debian/12/amd64/SALT-PROJECT-GPG-PUBKEY-2023.gpg
  • echo "deb [signed-by=/etc/apt/keyrings/salt-archive-keyring-2023.gpg arch=amd64] https://repo.saltproject.io/salt/py3/debian/12/amd64/latest bookworm main" | tee /etc/apt/sources.list.d/salt.list
    – you used commands for Ubuntu in your video
  • apt update (and, if needed, apt -y upgrade)
  • then the installation instructions for self-hosted, including your patch for PTR

Note: I also applied the PTR patch, and reset mine to default (not mail.$DOMAIN). But my instance still have a public IPv4 address…

11

Okay, so…MAYBE I’m making some progress?

Here are what appear to be the relevant entries in the postfixadmin logs that were shown as a result of running the command asked in post #8:

Invalid query: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away caused by UPDATE config SET value = :value WHERE name = ‘version’

That line show up no less than one hundred times in the log file.

To a lesser extent, I also saw these lines later on:

[core:warn] [pid 6] AH00111: Config variable ${APACHE_RUN_DIR} is not defined

apache2: Syntax error on line 80 of /etc/apache2/apache2.conf: DefaultRuntimeDir must be a valid directory, absolute or relative to ServerRoot

If you’d like me to Pastebin the full logs in a DM, please let me know.

12

Unfortunately, “MySQL server has gone away” is a very generic message, probably thrown by a previous error :frowning:

I can also see the Apache warning in a functional PostfixAdmin log, so we can ignore that.

Found something that may differ from a cloud VPS to a standalone VM: how the hosts file being populated.

Could you please check:

  • that your /etc/hosts file contains (with your actual domain instead of “domain.tld”) :
127.0.0.1       localhost
127.0.1.1       mail.domain.tld   mail

(if not, please change accordingly)

  • what is the ouput after:
source /mnt/repo-base/scripts/base.sh
curl -L https://mail.$DOMAIN/setup.php?debug=1

The curl should return HTML code for the setup page. You may have to reboot after modifying your hosts file.

Note: if your hosts file contains “ecloud” instead of “mail”, I’d advise to (“domain.tld” to be replaced by your real domain):

  • drop VM
  • change everything in your firewall from “ecloud…” to “mail.domain.tld”, especially DNS
  • create VM as “mail.domain.tld” (VMware tools may use this as hostname)
  • reinstall guest OS (Debian 12) as “mail.domain.tld” for hostname and “domain.tld” for DNS domain
    ** not sure about the hostname, depends of OS/distro… If, after install, hosts file is not as above please retry with short “mail” as hostname
13

Okay, so I recreated a VM and before I performed the install process, I did what you said and specified the hostname. The curl command does indeed return the HTML, so I don’t think this issue has to do with the hostname at a system level, based on that test.

Here’s what’s got me scratching my head:

root@mail:/mnt/repo-base# docker exec -it mariadb mysql -u root -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 94
Server version: 10.3.39-MariaDB-1:10.3.39+maria~ubu2004 mariadb.org binary distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> USE postfix;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [postfix]> SHOW TABLES;
+-----------------------+
| Tables_in_postfix     |
+-----------------------+
| admin                 |
| alias                 |
| alias_domain          |
| config                |
| domain                |
| domain_admins         |
| fetchmail             |
| log                   |
| mailbox               |
| quota                 |
| quota2                |
| vacation              |
| vacation_notification |
+-----------------------+
13 rows in set (0.000 sec)

MariaDB [postfix]> SELECT * FROM admin;
Empty set (0.000 sec)

MariaDB [postfix]> SELECT
    ->     table_name,
    ->     table_rows
    -> FROM
    ->     information_schema.tables
    -> WHERE
    ->     table_schema = 'postfix';
+-----------------------+------------+
| table_name            | table_rows |
+-----------------------+------------+
| quota2                |          0 |
| log                   |          0 |
| vacation              |          0 |
| alias_domain          |          0 |
| domain                |          0 |
| admin                 |          0 |
| vacation_notification |          0 |
| config                |          0 |
| domain_admins         |          0 |
| mailbox               |          0 |
| fetchmail             |          0 |
| alias                 |          0 |
| quota                 |          0 |
+-----------------------+------------+
13 rows in set (0.001 sec)

Now, please forgive me if I keep harping on this incorrectly…but it looks like the issue is that the script fails when attempting to populate the database tables.

Looking at the script, the command in the script that seems to be failing is:
curl --silent -L https://mail.$DOMAIN/setup.php >/dev/null

because the very next command is:
echo "Adding Postfix admin superadmin account"

but I never that echo, so the curl command seems to be the point of failure.

My limits are clearly showing here; I’m “Linux Literate” but I’m a “nano Linux user” rather than a “vi Linux user” and I wish I had a better idea of where to be helpful.

I hate to say this because I’m poignantly aware of the inelegance…what if I ran
sed 's/$DOMAIN/voyager529.com/' /mnt/repo-base/scripts/postinstall.sh

Do you think that, despite the inelegance, that could ultimately get past this installation issue?

14

You’re right, and that’s good news :slight_smile:

You’re right again! And I’m struggling with this too…

The install process is meant to be simple and not require expert knowledge :confused: You’ve done very well so far!

You can try that, but I’m pretty sure it won’t help.
You can make sure with grep "default_domain" /mnt/repo-base/volumes/nextcloud/data/appdata_snappymail/_data_/_default_/configs/application.ini : this file gets modified using $DOMAIN, just before the PostfixAdmin thing.

Currently digging into PostfixAdmin setup, I’ll keep you posted…

15

Good news! I discovered that the curl does … absolutely nothing! :crazy_face:

First, it’s OK to have 13 empty tables in “postfix” database: they are meant to be filled with configuration commands (the lines after the curl in postinstall.sh).

You may ask: “OK, but how does the database and the tables get created?”. That’s a very good question :smiley_cat:

Right now, the only thing you’ll really have to take care of, is the config file being properly populated.
Please issue cd /mnt/repo-base then docker-compose exec postfixadmin cat /var/www/html/config.local.php.
If you see something very similar to the following, you’re good to go!

<?php
                $CONF['database_type'] = 'mysqli';
                $CONF['database_host'] = 'mariadb';
                $CONF['database_port'] = '3306';
                $CONF['database_user'] = 'postfix';
                $CONF['database_password'] = '****';
                $CONF['database_name'] = 'postfix';
                $CONF['setup_password'] = '****';
                $CONF['smtp_server'] = 'mail.****';
                $CONF['smtp_port'] = '587';
                $CONF['configured'] = true;
$CONF['configured'] = true;
$CONF['encrypt'] = 'dovecot:SHA512-CRYPT';
$CONF['dovecotpw'] = '/usr/bin/doveadm pw';
$CONF['smtp_sendmail_tls'] = 'YES';
$CONF['domain_path'] = 'YES';
$CONF['password_validation'] = array('length_check' => function($password) { if (strlen(trim($password)) < 5) { return 'password_too_short'; } },);
$CONF['admin_email'] = 'drive@****';
$CONF['admin_smtp_password'] = '****';
$CONF['footer_text'] = 'Return to ****';
$CONF['footer_link'] = 'http://****';
                ?>

I paused the script just before the curl, and ran it manually while comparing database & config file contents before and after: as the base configuration has already been automatically done, the PHP code called by curl does nothing :grin:
For the reference, code is here, as well as the companion upgrade.php: https://github.com/postfixadmin/postfixadmin/blob/master/public/setup.php.

So, I suggest you restart the installation, commenting out the curl line in postinstall.sh before launching the bootstrap (as you do with PTR check).
If you have any doubt, please post back the config.local.php content (you can DM me if it’s easier to you).

Unfortunately, this doesn’t explain WHY you’re facing an abrupt stop on the curl :confused:
Anyway, commenting it out will execute the following lines, and we’ll see what happen!

1 Like
16

Please DM me your Paypal / Venmo. After over a week of back-and-forth, it’s WORKING!!!

TL;DR for anyone else having the same problem:

  1. Make sure your server’s hostname is set to its publicly accessible domain name. /etc/hosts should have a line in it that reads: 127.0.1.1 mail.voyager529.com mail (obviously, replace ‘voyager529.com’ with your own domain).

  2. Open /mnt/repo-base/scripts/postinstall.sh in a text editor. Delete or comment out line 122.

On a completely tangentially related note, while I did my testing with smu44 on a stock Debian install and achieved success there, I was ultimately able to do my working server build on DietPi. I prefer DietPi because of its shorter install time, its menu-based interface, and a number of other built-in tools for lots of common functions, but there are a few things to keep in mind if you do…

  • During the initial install, add Python3 to the software installation.

  • During the initial install, do NOT add Docker/Docker-Compose/Portainer.

  • After the initial install, run dietpi-config and choose option 6 (security options) to set your hostname as well as your static IP in option 7 (network adapters).

  • Follow the instructions to install salt-minion here. The Ubuntu instructions do NOT work properly with the Debian build that is DietPi, and since the lsb_release command doesn’t work properly, the script will NOT automatically install salt-minion, which will break everything else.

  • BEFORE running the script, apt-get install apparmor git curl. The script will NOT run properly if AppArmor isn’t installed prior to running it.

Thank you again, Sylvain, for your dedication to helping solve my problem!!!

17

I’m very happy to read it’s finally working, that’s the best reward for me :smiley_cat:

While testing the next version of Murena Cloud, I will discuss with the development team the removal of the unnecessary curl line…

And thanks for your feedback on DietPi!
Although it’s uncommon distro for cloud VPS, it should be great for anyone willing to run Murena Cloud at home.
And it may open the door to the ARM world :wink:

Have fun with your Murena Cloud!
You may find these useful as extra steps:

18

DietPi is a streamlined distro that is based quite heavily on the Debian-Netinst release, which is why 1.) it’s mostly a drop-in replacement for Debian in this context, and 2.) All the tweaks listed that differ from the regular Debian install involve installing packages or using the menu interface instead of editing text files. Of course, I understand the team not necessarily wanting to have an appendix for every Debian derivative, but I’ve found DietPi to be an incredibly useful starting point for my LAMP or Docker machines, so I was glad I was able to get it to work, and hopefully contribute back.

One thing worth noting here is that my setup is a bit different than what most self-hosters are doing. Instead of using a VPS service, I have a physical server in a datacenter, and my /e/Cloud server is running in a VM on a bare metal hypervisor, rather than running /e/Cloud on a VPS service.

Thank you so much for this recommendation! My initial plan was to use XSIBackup, but if there’s a concern regarding database corruption in such a context, then I’ll absolutely follow up with the procedures listed here as well.

I’ll definitely look into this also! For the moment, I’m using the free version of the Sophos UTM firewall that does Geo-IP blocking and IDS/IPS prevention at the router level, but I completely agree that some security functions on the guest OS are worth implementing as a second line of defense.

Thank you again for everything!!

19

Well… Self-hosted Murena Cloud is primarily intended to be run on a cloud VPS and mainstream Debian-based distro, the install guide leaves no doubt about that.
However, it’s an open-source project, and everyone is welcome to give feedback about a custom use! :smiley_cat:

For next release, I think that we could propose to include a DietPi install guide as a separate document, based on your experience with it. What’s your feeling about that?

Please be aware that the SQL backups not intended to replace any sort of VM backup, but to provide a usable SQL dump-style of MariaDB databases.
I didn’t know about XSIBackup. Unless you already use it and have licenses, I’d also suggest https://www.veeam.com/virtual-machine-backup-solution-free.html, it’s a well-know actor for VM backup.
Whatever VM backup solution you use, please be sure to quiesce the VM before snapshot: we also have some file-based components (i.e. Postfix/Dovecot, Nextcloud file sharing, …).
Best solution could be a software-based replication to a dedicated backup VM, that you could “freeze” for backups without any downtime for your working server.

Please feel free to ask anything (related to self-hosted) on the dedicated thread.
What is different with Crowdsec is it’s capability to block a bad actor, not only based on a central reputation database, but also on behavior based on your server’s logs.

You’re very welcome :smiley_cat: