Securing your self-hosted using CrowdSec

smu44 · April 2, 2022, 8:20am

Hi there !

Introduction

Every self-hosted administrator has to deal with various attacks nowadays (script kiddies, automated scanners, …)
As we host several web services (http, mail, …) our server are targets of choice.

Of course /e/ Team has taken a particular care to the security, and attackers are unlikely to gain any kind of access.
But the logs may show lots of attempts. It is not reassuring, and pollutes the reading.
So a guy may want to install an IDS/IPS solution, which integrates in a standalone /e/ self-hosted. And also easy to install and maintain.
Of course, open-source !

Searching for a solution

I set out to find a system that would perform well and integrate efficiently in a self-hosted /e/ Cloud.
I searched a tried some well-known, including:

fail2ban (+ ufw) : primarily designed for shell attacks, not so easy to fit to a whole server
Suricata : known as good, but maybe to heavy (it is network-capture oriented)
Wazuh : also a good solution (used by /e/ if I remember well), but way too heavy (agent/server approach, with network capture)
…

Finally, after some trials and fails, I came up with CrowdSec:

light (no network capture, just log analysis)
easy to install, watch, maintain
collaborative (community database)
well maintained, with a strong community
easy to install, watch, maintain
modular, also with simulation mode
interacts with Docker logs (no need for journald )

Sometimes, a good drawing is best to explain “how it works” : https://doc.crowdsec.net/docs/intro

So, I propose to share this with you.

CrowdSec installation

First, was the decision to use a Docker image or not.
I found that not using Docker was the right choice : we also want to protect SSH and maybe some other non-Docker services. Also the Docker overhead is not wanted here.

Let’s roll !

Easy ! Just followed the install doc
For now, we don’t want to install a “bouncer”, which is the active part (IPS). Just stay relax and watch how it goes …

Quick tour:

main management text-only entry-point is cscli
logs, as usual, in /var/log (crowdsec*.log)

To easily monitor the activity, I created a Console account and registered my instance : https://doc.crowdsec.net/docs/console

For my own little comfort, I registered Bash completion with cscli completion bash | sudo tee /etc/bash_completion.d/cscli (more help about that with cscli help completion). Don’t forget to throw a new shell .

I also chose some additional components from the CrowdSec Hub. Below my full lists.

Collections (bundles of parser+scenarios)

COLLECTIONS
------------------------------------------------------------------------------------------------------------
 NAME                               📦 STATUS   VERSION  LOCAL PATH
------------------------------------------------------------------------------------------------------------
 crowdsecurity/apache2              ✔️  enabled  0.1      /etc/crowdsec/collections/apache2.yaml
 crowdsecurity/dovecot              ✔️  enabled  0.1      /etc/crowdsec/collections/dovecot.yaml
 crowdsecurity/mysql                ✔️  enabled  0.1      /etc/crowdsec/collections/mysql.yaml
 crowdsecurity/sshd                 ✔️  enabled  0.2      /etc/crowdsec/collections/sshd.yaml
 crowdsecurity/iptables             ✔️  enabled  0.1      /etc/crowdsec/collections/iptables.yaml
 crowdsecurity/base-http-scenarios  ✔️  enabled  0.5      /etc/crowdsec/collections/base-http-scenarios.yaml
 crowdsecurity/linux                ✔️  enabled  0.2      /etc/crowdsec/collections/linux.yaml
 crowdsecurity/nextcloud            ✔️  enabled  0.2      /etc/crowdsec/collections/nextcloud.yaml
 crowdsecurity/nginx                ✔️  enabled  0.1      /etc/crowdsec/collections/nginx.yaml
 crowdsecurity/postfix              ✔️  enabled  0.2      /etc/crowdsec/collections/postfix.yaml
------------------------------------------------------------------------------------------------------------

Parsers (information extraction from logs)

PARSERS
-------------------------------------------------------------------------------------------------------------
 NAME                            📦 STATUS   VERSION  LOCAL PATH
-------------------------------------------------------------------------------------------------------------
 crowdsecurity/geoip-enrich      ✔️  enabled  0.2      /etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml
 crowdsecurity/http-logs         ✔️  enabled  0.8      /etc/crowdsec/parsers/s02-enrich/http-logs.yaml
 crowdsecurity/postscreen-logs   ✔️  enabled  0.1      /etc/crowdsec/parsers/s01-parse/postscreen-logs.yaml
 crowdsecurity/whitelists        ✔️  enabled  0.2      /etc/crowdsec/parsers/s02-enrich/whitelists.yaml
 crowdsecurity/nextcloud-logs    ✔️  enabled  0.1      /etc/crowdsec/parsers/s01-parse/nextcloud-logs.yaml
 crowdsecurity/apache2-logs      ✔️  enabled  0.9      /etc/crowdsec/parsers/s01-parse/apache2-logs.yaml
 crowdsecurity/mysql-logs        ✔️  enabled  0.3      /etc/crowdsec/parsers/s01-parse/mysql-logs.yaml
 crowdsecurity/nginx-logs        ✔️  enabled  1.0      /etc/crowdsec/parsers/s01-parse/nginx-logs.yaml
 crowdsecurity/iptables-logs     ✔️  enabled  0.2      /etc/crowdsec/parsers/s01-parse/iptables-logs.yaml
 crowdsecurity/syslog-logs       ✔️  enabled  0.8      /etc/crowdsec/parsers/s00-raw/syslog-logs.yaml
 crowdsecurity/dovecot-logs      ✔️  enabled  0.3      /etc/crowdsec/parsers/s01-parse/dovecot-logs.yaml
 crowdsecurity/postfix-logs      ✔️  enabled  0.3      /etc/crowdsec/parsers/s01-parse/postfix-logs.yaml
 crowdsecurity/dateparse-enrich  ✔️  enabled  0.2      /etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml
 crowdsecurity/sshd-logs         ✔️  enabled  1.8      /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml
-------------------------------------------------------------------------------------------------------------

Scenarios (bad behaviour detection)

SCENARIOS
--------------------------------------------------------------------------------------------------------------------------
 NAME                                       📦 STATUS   VERSION  LOCAL PATH
--------------------------------------------------------------------------------------------------------------------------
 ltsich/http-w00tw00t                       ✔️  enabled  0.1      /etc/crowdsec/scenarios/http-w00tw00t.yaml
 crowdsecurity/http-path-traversal-probing  ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-path-traversal-probing.yaml
 crowdsecurity/http-probing                 ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-probing.yaml
 crowdsecurity/ssh-slow-bf                  ✔️  enabled  0.2      /etc/crowdsec/scenarios/ssh-slow-bf.yaml
 crowdsecurity/ssh-bf                       ✔️  enabled  0.1      /etc/crowdsec/scenarios/ssh-bf.yaml
 crowdsecurity/http-crawl-non_statics       ✔️  enabled  0.3      /etc/crowdsec/scenarios/http-crawl-non_statics.yaml
 crowdsecurity/http-sqli-probing            ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-sqli-probing.yaml
 crowdsecurity/iptables-scan-multi_ports    ✔️  enabled  0.1      /etc/crowdsec/scenarios/iptables-scan-multi_ports.yaml
 crowdsecurity/http-backdoors-attempts      ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-backdoors-attempts.yaml
 crowdsecurity/nextcloud-bf                 ✔️  enabled  0.1      /etc/crowdsec/scenarios/nextcloud-bf.yaml
 crowdsecurity/http-generic-bf              ✔️  enabled  0.1      /etc/crowdsec/scenarios/http-generic-bf.yaml
 crowdsecurity/http-sensitive-files         ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-sensitive-files.yaml
 crowdsecurity/http-xss-probing             ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-xss-probing.yaml
 crowdsecurity/postfix-spam                 ✔️  enabled  0.2      /etc/crowdsec/scenarios/postfix-spam.yaml
 crowdsecurity/dovecot-spam                 ✔️  enabled  0.3      /etc/crowdsec/scenarios/dovecot-spam.yaml
 crowdsecurity/http-bad-user-agent          ✔️  enabled  0.7      /etc/crowdsec/scenarios/http-bad-user-agent.yaml
 crowdsecurity/http-open-proxy              ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-open-proxy.yaml
 crowdsecurity/mysql-bf                     ✔️  enabled  0.1      /etc/crowdsec/scenarios/mysql-bf.yaml
--------------------------------------------------------------------------------------------------------------------------

It’s very easy to manage them from cscli (for example, https://doc.crowdsec.net/docs/cscli/cscli_collections_install).

CrowdSec customization

Of course, now we’ll have to “feed the beast” !

Here is my /etc/crowdsec/acquis.yaml file (of course, I made a backup of the original ):

source: file
filenames:
  - /var/log/syslog
  - /var/log/kern.log
  - /var/log/auth.log
  - /var/log/ufw.log
labels:
  type: syslog
---
source: docker
container_name:
  - mailserver
labels:
  type: syslog
---
source: docker
container_name:
  - nginx
labels:
  type: nginx
---
source: docker
container_name:
  - postfixadmin
labels:
  type: nginx
---
source: docker
container_name:
  - welcome
labels:
  type: nginx
---
source: docker
container_name:
  - nextcloud
labels:
  type: nextcloud
---

Nothing fancy here, all is explained at https://doc.crowdsec.net/docs/data_sources/intro !
Most important part is the “type” label, which indicates how the log should be parsed.
You can live-test them, for example with cscli explain -d "docker://nginx" -t nginx.
Please be aware that parsing a long-life container logs may take … hours ! You may want to try with cscli explain -d "docker://nginx?since=1h" -t nginx (doc : https://doc.crowdsec.net/docs/data_sources/docker#dsn-and-command-line).

NB : I included “postfixadmin”, “welcome”, “nextcloud” containers logs for my tests, as our only http(s) entry point is “nginx” they are not mandatory.

Now, just restart CrowdSec service with systemctl restart crowdsec and wait for some bad guys … it shouldn’t take long !

Also, you can tell CrowdSec to scan the whole log from a source with, for example, crowdsec -dsn "docker://nginx" -type nginx -no-api (warning: heavy load !).
Take a look at your acquis.yaml file to get DSN & type.

Watch it

After some time, cscli metrics should reveal some interesting counters.
Here’s mine after some days:

INFO[02-04-2022 09:28:38 AM] Buckets Metrics:
+-------------------------------------------+---------------+-----------+--------------+--------+---------+
|                  BUCKET                   | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |
+-------------------------------------------+---------------+-----------+--------------+--------+---------+
| crowdsecurity/dovecot-spam                | -             | -         |            9 |      9 |       9 |
| crowdsecurity/http-backdoors-attempts     | -             | -         |            4 |      4 |       4 |
| crowdsecurity/http-bad-user-agent         | -             |        30 |          100 |    130 |      70 |
| crowdsecurity/http-crawl-non_statics      | -             | -         |         7419 |  10256 |    7419 |
| crowdsecurity/http-open-proxy             | -             |         4 |            4 | -      | -       |
| crowdsecurity/http-path-traversal-probing | -             | -         |            6 |      6 |       6 |
| crowdsecurity/http-probing                | -             |         9 |          680 |    871 |     671 |
| crowdsecurity/http-sensitive-files        | -             | -         |           63 |     67 |      63 |
| crowdsecurity/iptables-scan-multi_ports   |             4 | -         |        22778 |  29933 |   22774 |
| crowdsecurity/postfix-spam                |             1 |        14 |          879 |   1240 |     864 |
| ltsich/http-w00tw00t                      | -             |         3 |            3 | -      | -       |
+-------------------------------------------+---------------+-----------+--------------+--------+---------+
INFO[02-04-2022 09:28:38 AM] Acquisition Metrics:
+------------------------+------------+--------------+----------------+------------------------+
|         SOURCE         | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+------------------------+------------+--------------+----------------+------------------------+
| docker:mailserver      |      42066 |         8165 |          33901 |                   1249 |
| docker:nextcloud       |      23760 | -            |          23760 | -                      |
| docker:nginx           |      27678 |        27144 |            534 |                  11290 |
| docker:postfixadmin    |         98 |           96 |              2 | -                      |
| docker:welcome         |        166 |          156 |             10 |                     44 |
| file:/var/log/auth.log |       6071 | -            |           6071 | -                      |
| file:/var/log/syslog   |       4780 | -            |           4780 | -                      |
| file:/var/log/ufw.log  |      30298 |        30258 |             40 |                  29933 |
+------------------------+------------+--------------+----------------+------------------------+
INFO[02-04-2022 09:28:38 AM] Parser Metrics:
+------------------------------------+-------+--------+----------+
|              PARSERS               | HITS  | PARSED | UNPARSED |
+------------------------------------+-------+--------+----------+
| child-crowdsecurity/dovecot-logs   | 14797 |   6925 |     7872 |
| child-crowdsecurity/http-logs      | 82188 |  59766 |    22422 |
| child-crowdsecurity/nextcloud-logs | 47520 | -      |    47520 |
| child-crowdsecurity/nginx-logs     | 28524 |  27396 |     1128 |
| child-crowdsecurity/postfix-logs   | 29524 |   1240 |    28284 |
| child-crowdsecurity/sshd-logs      |  1890 | -      |     1890 |
| child-crowdsecurity/syslog-logs    | 83215 |  83215 | -        |
| crowdsecurity/dateparse-enrich     | 65819 |  65819 | -        |
| crowdsecurity/dovecot-logs         | 10861 |   6925 |     3936 |
| crowdsecurity/geoip-enrich         | 65818 |  65818 | -        |
| crowdsecurity/http-logs            | 27396 |  26544 |      852 |
| crowdsecurity/iptables-logs        | 30298 |  30258 |       40 |
| crowdsecurity/nextcloud-logs       | 23760 | -      |    23760 |
| crowdsecurity/nginx-logs           | 27942 |  27396 |      546 |
| crowdsecurity/non-syslog           | 51702 |  51702 | -        |
| crowdsecurity/postfix-logs         | 10528 |   1240 |     9288 |
| crowdsecurity/sshd-logs            |   210 | -      |      210 |
| crowdsecurity/syslog-logs          | 83215 |  83215 | -        |
| crowdsecurity/whitelists           | 65819 |  65819 | -        |
+------------------------------------+-------+--------+----------+

(API calls at bottom are to be ignored).

Here I owe you a little explanation:

“Acquisition” is the log reading, please refer to previous chapter
“Parser” is the log processing
“Bucket” is the positive results treatment, when they are “leaking” (overflows counters) then alert is raised and decision taken. More details here : https://doc.crowdsec.net/docs/scenarios/intro

You can also watch alerts & decisions, using cscli alerts list & cscli decisions list (only lists local decisions by default, use cscli decisions list -a to include community-driven, warning: big list).
You may notice that many alerts & decisions are from community, that’s what we expect !

Also, using the Console:

You can display details about an attacker using the integrated CTI:

What’s next ?

Once it is established that our own use is not detected as malicious, we can start to block these bad guys.

Just install the firewall Bouncer and enjoy !
This will install ipset as requisite, so we can simply monitor the blocked IP addresses with ipset list (warning: big list).

You may also watch decisions taken from your server using cscli decisions list (should not be huge …).

Then, enjoy the clean air, and go say Hi in CrowdSec forum or Discord chat !

As always, please notify me of any errors or omissions, and share your comments

Regain your privacy! Adopt /e/ the unGoogled mobile OS and online services phone

smu44 · April 2, 2022, 8:50am

As a side note, I also changed SSH listening port (22/TCP is far too much well-known).

A quick how-to

please ensure you have a working console access to your server
find an unused TCP port, using netstat -tlan and /etc/services file
edit /etc/ssh/sshd_config, ensure that you have:

Include /etc/ssh/sshd_config.d/*.conf

#Port 22

then create a file in /etc/ssh/sshd_config.d/, let’s say myserver.conf (replace 12345):

# My conf
Port 12345
PermitRootLogin yes
PasswordAuthentication no

# For IDS
LogLevel VERBOSE

then, from console, restart sshd using systemctl restart ssh. Verify using netstat -tlan.
you may have to allow your customized ssh through firewall, I used ufw for it’s ease of use:
– create a ssh app profile /etc/ufw/applications.d/openssh-server with (replace 12345):

[OpenSSH]
title=Secure shell server, an rshd replacement
description=OpenSSH is a free implementation of the Secure Shell protocol.
ports=12345/tcp

– then ufw enable, ufw allow OpenSSH
– note: ufw won’t interfere with iptables and Docker rules, it’s just a front-end (more here)

smu44 · September 30, 2023, 7:40am

Hi!

Forgot to mention about firewall bouncer: I had to enable the iptables “DOCKER-USER” chain in /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml (and set “disable_ipv6” to true, as we only use IPv4).
Also useful is “deny_log” to true

Don’t forget to restart bouncer with systemctl restart crowdsec-firewall-bouncer.service!