this post was submitted on 17 Feb 2024

90 points (97.9% liked)

Selfhosted

39253 readers

233 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 1 year ago

MODERATORS

[email protected]

What's Your Preferred Server Monitoring Method? (lemmy.world)

submitted 7 months ago by [email protected] to c/[email protected]

68 comments fedilink hide all child comments

How do you monitor your server containers, disks, load...?

Do you use an easy-to-use web interface? Do you do everything via SSH? Or maybe you've got a more complicated setup?

I want to change my setup and I'm looking for new ideas, I've been using Cockpit for some years and some of the plugins are really outdated (ZFS for example) and others are completely broken (docker-compose).

top 50 comments

sorted by: hot top controversial new old

[–] [email protected] 1 points 6 months ago* (last edited 6 months ago)

Used to not monitor at all, but after setting up crowdsec I got completeky into prometheus + grafana system resource and security monitiring on my dedicated hetzner server.

I also keep uptime kuma on another cheap vps to monitor the states of websites directly for issues and I have seted up watchtower to send ntfy notifications on updates so I can know an update is the one fucking everything up.

Recently also setup restuc backups so I made it so I also get backup health check logs as ntfy notifications on my phone, which really helps me keep everything runnig.

What I really need to also do is create prometheus/grafana alerts for additional things to get notifications on my phone for them also ( like when crowdsec starts to randomly not get any more acqusitions, so I have to restart it. Once passed over a week before I looked at grafana and caught that ).

Note: this is all a hoby, I also don't host anything at home out of a couple reasons, most important being internet and hardware is expensive af here so it's simply cheaper for me to play around with vps's and hetzner dedicated servers.

[–] [email protected] 1 points 7 months ago

i just have top running through ssh on an xterm window.

[–] [email protected] 1 points 7 months ago

For the physical hosts / bare metal I use fluentbit, with Loki as the backend. Grafana for visualization and alerts. This gives me utilization metrics and uptime monitoring. The app containers themselves I do not monitor.

[–] [email protected] 21 points 7 months ago

"Huh weird, I tried to use and it's not working. Welp, guess I better fix it..."

[–] [email protected] 1 points 7 months ago (1 children)

Um, Proxmox?

[–] [email protected] 0 points 7 months ago (1 children)

I am running everything in docker compose so I've never found a use to it that justifies the waste of power

[–] [email protected] 2 points 7 months ago* (last edited 7 months ago) (1 children)

docker-compose doesn't scale well and if you run it natively it is a little less secure.

Virtualization adds 1-2% of overhead at most and gives you way more control of how the hardware is used.

If you setup is small docker-compose might be easier to manage but as soon as you get more hardware it becomes the limiting factor. I still use docker-compose but now I run it in a VM

[–] [email protected] 2 points 7 months ago

I switched from docker compose to pure Ansible for deploying my containers. Makes managing config and starting containers across multiple hosts super easy. I considered virtualizing too but decided it didn't offer me enough advantages. If I ever have an issue with the host OS I just reinstall using a preseed file and then rerun my playbooks and it's ready to go.

[–] [email protected] 5 points 7 months ago* (last edited 7 months ago) (1 children)

I've been using uptime Kuma recently and it's great but works better outside of docker.

Inside docker I'd get a lot of false down positives from I assume docker throttling the checks.

Plus it works with email, telegram, and matrix chat alerts. I monitor all my clients sites with it, and it's bullet proof behind caddy.

[–] [email protected] 1 points 7 months ago (1 children)

For light touch monitoring this is my approach too. I have one instance in my network, and another on fly.io for the VPSs (my most common outage is my home internet). To make it a tiny bit stronger, I wrote a Go endpoint that exposes the disk and memory usage of a server including with mem_okay and disk_okay keywords, and I have Kuma checking those.

I even have the two Kuma instances checking each other by making a status page and adding checks for each other's 'degraded' state. I have ntfy set up on both so I get the Kuma change notifications on my iPhone. I love ntfy so much I donate to it.

For my VPSs, this is probably not enough, so I am considering the more complicated solutions (I've started wanting to know things like an influx of fali2ban bans etc.)

[–] [email protected] 2 points 7 months ago

I just do web hosting for clients sites and use Kuma to monitor uptime and SSL certificates.

Ive got multiple Kuma's running as well.

[–] [email protected] 7 points 7 months ago (2 children)

How has nobody in this thread said check_mk yet?

It's free, you host it yourself. It's built off of nagios, compatible with nagios plugins, supports snmp or agent based checks. It can email, SMS, slack or discord you when something breaks, you can write your own custom checks in any language that can output to a local console... I could never imagine even looking for something else.

[–] [email protected] 1 points 7 months ago

I started using Checkmk recently after it was mentioned here and I really like it. I'd used Zabbix a bit but was annoyed at how much work it took to get it to do what I want. Checkmk was a lot better right out of the box.

[–] [email protected] 2 points 7 months ago* (last edited 7 months ago)

+1 for check_mk.

It's got a scriptable config file that begs for automation like mgmtConfig and it does SNMP. For me, that's it. SNMP->MQTT->SNMP next year.

[–] [email protected] 1 points 7 months ago

I use Proxmox, so I just use the PVE web interface

[–] [email protected] 1 points 7 months ago

Prometheus + node/container exporters. + Grafana for dashboards I haven't touched zabbix in years but last time it didn't support very well dynamic scalations. Also all of them are focused on monitoring infrastructure, you need to pay if you want APM or UX.

Enterprise level, for APM I like datadog, much better than NR. For UX we use acoustic tealeef.

[–] [email protected] 1 points 7 months ago

Grafana, fronting information from Prometheus, Loki and Telegraf/influxdb since I'm used to that from work and has been a bit more set and forget compared to node_exporter. Easier to add in plugins as well instead of a new container/service to scrape.

[–] [email protected] 1 points 7 months ago* (last edited 7 months ago)

Cockpit, Cosmos Cloud, Portainer, Grafana, and a few other things. It’s not the most optimal solution but it kinda of works for now.

[–] [email protected] 1 points 7 months ago* (last edited 6 months ago)

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:

Fewer Letters	More Letters
MQTT	Message Queue Telemetry Transport point-to-point networking
SSH	Secure Shell for remote terminal access
SSL	Secure Sockets Layer, for transparent encryption
TLS	Transport Layer Security, supersedes SSL
VPS	Virtual Private Server (opposed to shared hosting)

5 acronyms in this thread; the most compressed thread commented on today has 16 acronyms.

[Thread #521 for this sub, first seen 17th Feb 2024, 17:25] [FAQ] [Full list] [Contact] [Source code]

[–] [email protected] 12 points 7 months ago

I just use homepage as my homepage :D

I can see simple CPU/RAM/storage stats and got widgets for almost all services, one of them is portainer so I can see if any service is stopped (most of them are running in docker). Also few services send notification on error or update

I know its not really a monitoring tool, but it works well enough for me

[–] [email protected] 1 points 7 months ago

I use collectd and graphs on my openwrt router. It can even use data from mqtt-connected thermometers and gather metrics from other collectd instances.

[–] [email protected] 0 points 7 months ago

systemd can be used to run whatever notification scheme you would like to use whenever some service fails. Here's an example of how to do it: https://www.baeldung.com/linux/systemd-service-fail-notification

[–] [email protected] 4 points 7 months ago

At home, nagios, at work colleagues. (I finally escaped the admin rat race)

[–] [email protected] 47 points 7 months ago (2 children)

My clients when they text me the server is down.

[–] [email protected] 22 points 7 months ago (1 children)

This has the same energy as my spouse yelling at me because jellyfin went down

[–] [email protected] 4 points 7 months ago

Or my partners greeting me in the morning "Home assistant went down again, so the lights are all manual"

Thankfully that one is mostly solved.

[–] [email protected] 3 points 7 months ago

So damn accurate ahhaha

[–] [email protected] 4 points 7 months ago

Prometheus and Altertmanager

[–] [email protected] 2 points 7 months ago* (last edited 7 months ago)

Btop and logwatch with logrotate. I use healthchecks to check if the server is unreachable and it notifies me.

load more comments