[HOWTO] Self-hosting feature thought: integration of LibreX into the /e/Cloud setup

[smu44]

Hi @voyager529,

@voyager529

2.) add the LibreY container to a different, independent Docker host, in which case my only changes to the /e/Cloud environment would be adding the nginx config entry and the certificate.

Now, while I think the #2 option is easier (honestly, it’s pretty generic, since I’ve already got a second VM with a Docker instance for KitchenOwl, I figured it’d be worth exploring what it would take to integrate LibreY into the build, because, as we’ve already estabished, I’m the one weirdo who puts my data on my own server, rather than a VPS . If I can turn this thread into a tutorial for other users, I think there is value to be had in that.

You can, but please be aware that communication between nginx and libreY will be unencrypted (not really sure if this a concern for a search engine, anyway).

That being said, it looks to me as an overly complicated setup, having the data going back and forth between your client, the Murena Cloud reverse proxy, and your search instance.
Of course it will work, and using Murena Cloud nginx will spare you from reverse proxy and SSL headaches. But you may also want to start with a simple HTTP setup @home, skipping the nginx reverse proxy for a start.

@voyager529

So, if I understand the procedure correctly, it would be as follows (stay with me):

1. ssh into my /e/Cloud VM and docker-compose stop && nano /mnt/repo-base/librey.txt
2. enter the contents of the linked docker-compose file, along with the recommended changes from your list (I’ll probably opt out of watchtower for the time being).
3. cat /mnt/repo-base/librey.txt >> /mnt/repo-base/docker-compose.yml && docker-compose up -d

Do I have that right? The logic regarding making the librey entry a separate text file and then concatenating it is that when the /e/Cloud update comes (no, I’m not looking forward to Nextcloud 29, why do you ask? =) ), I’m assuming that the upgrade will overwrite my docker-compose.yml file, so the known-working config can be re-added after an upgrade by running the append command again…that’s my logic, anyway…I promise I won’t be offended if you tell me I’m way off base here and provide some corrections =).

(note: this is a setup for having the search engine at your Murena Cloud server, it won’t be needed if you install libreY on another server @home).

Murena Cloud upgrades never implied a complete overwrite of docker-compose.yml, so far (and I bet my pants that upgrade for NC 29 will stay this way, hopefully we’ll see that in a near future ). Please see here as an example: https://gitlab.e.foundation/e/infra/ecloud-selfhosting/-/blob/master/upgrade-guides/upgrade-to-26.0.8.23.md.
Personalty, I’ll go for a direct edit after a backup copy (you can always backup/diff/copy/paste the lines to a newer file, when upgrading).
Anyway it will work the way you describe, just pay attention to the number of spaces as beginning of lines, please.

@voyager529

Now, there are two related parts here I’m not completely clear on. First, I’d assume that I’d need to add a subdomain, say ‘search.voyager529.com’ that would point to the librey instance. I’d assume that I’d need to make a search.voyager529.com.conf file and put it in /mnt/repo-base/config/nginx/sites-enabled, as I’ve done with my Vaultwarden instance. Now, that’s something I’ve done before, but you said that I didn’t need to expose the port since I’m using the internal nginx instance. Where I’m a bit confused is that I’d assume that there’s a need to tell nginx to direct traffic to the separate container.

Please read here: |HowTo] Add a webapp as a sideload of /e/ self-hosted cloud.

The differences for an external container @home should be (untested):

• keep the port exposure: in this case we need access to the Docker container from outside of Docker. You can use anything you want, above 1024. Let’s say 8887
  – there’s no way you can proxy from one server to another, without exposing the target port to the Internet (unless you have some kind of VPN between these servers)
• you will have to pinhole or NAT this port in your home firewall, let’s say we’ll use 12345 as external port to keep everything clear here Of course you can also restrict source IP with your Murena Cloud’s
• now, you will need 2 DNS entries: one for the public entry point as HTTPS (let’s say search.voyager529.com) and one for your private libreY instance as HTTP over 12345 (let’s say search-engine.voyager529.com)
• then if you look at nginx service in your Murena Cloud docker-compose.yml, you fill find:

    ports:
      - "80:8000"
      - "443:4430"

Knowing that, in nginx config files we’ll have to use 8080 for external HTTP/80, and 4430 for external HTTPS/443.

• in my example (link above), you can find 2 server directives:
  – you should focus on server_name (the DNS name that nginx will know and reply for), and proxy_pass (files are not served locally, but by a proxied resource)
  – first directive is to redirect HTTP to HTTPS, you only have to change server_name to your public DNS (search.voyager529.com)
  – second directive is for regular HTTPS data, here change server_name to your public DNS (search.voyager529.com), and proxy_pass to your libreY instance as known from Internet (http://search-engine.voyager529.com:12345) (please note the http://)
  – in a “one in a box” setup Docker will implement an internal IPAM, and will provide some kind of low-level DNS resolution to containers; that’s why we usually use containers names to communicate between, for example in nginx conf files. In this case, we can’t
• you can delete the client_max_body_size thing, it won’t be of use for you

As for the option with having the search instance at your Murena Cloud server (please also read above for technical details):

• the (in)famous line 7 can be deleted: we won’t expose the service outside Docker
  – ports are managed for each Docker container, you can have multiple containers (even sharing the same source image) listening on the same port (couple “container IP address + port” is the criteria here)
• server_name will still need to be your public DNS name (search.voyager529.com)
• proxy_pass should be set up to your libreY container “protocol+name+port” (http://librey:8080)
  – I don’t recommend using static IP addresses here: container’s are managed by Docker! In Docker IPAM we trust
• you can also delete the client_max_body_size thing
• if your nginx complains about not finding the “librey” DNS name, you may have to play with “depends_on” Docker Compose directive to have librey started before nginx (most likely: add “librey” to “depends_on” nginx’s list)
  – as I could experience, each container maintain it’s own IPAM image acquired at startup, and Docker IPAM is populated only when a container starts. There is probably an option to refresh all containers IPAM image when starting a supplemental container, but I’m too lazy to find out…

@voyager529

Second, something I haven’t done before is to use certbot to get a Let’s Encrypt cert, and this seems like as good a time as any to attempt that. Having read through the installer scripts, here’s what I think the procedures are; please tell me what I’ve got wrong here (again, same logic - separate files for custom entries to allow simple concatenation commands after updates):

1. echo 'search.voyager529.com' >> /etc/repo-base/config/letsencrypt/autorenew/custom_domains.txt
2. cat /etc/repo-base/config/letsencrypt/autorenew/custom_domains.txt >> /etc/repo-base/config/letsencrypt/autorenew/ssl-domains.dat
3. in the nginx config file I make, set the ssl_certificate line to /certs/live/search.voyager529.com/fullchain.pem and a similar path to the .key file.
4. stop the nginx container.
5. run ./mnt/repo-base/scripts/ssl-renew.sh
6. check /certs/live folder that the cert and key were created. Assuming they were, start the nginx container again.

1. and 2. as for docker-compose.yml you can also directly edit this file, it’s very unlikely to be overwritten (in fact, most often we forget to clean it up)
2. to 6. it’s OK

(extra step) Bravo !

Hopefully I didn’t forget something… Please ask if unsure!
And please remember: one of the major benefits of Docker is the easy cleanup, contrary to traditional hosted installs