DevOps

umami.is & plausible.io

Professionally, I’ve spent the last year almost entirely focused on o11y, a numeronym for observability. IMO you want to run opentelemetry (aka otel) for a lot of this stuff. It’s a fantastic tool. We tell clients that if they don’t use otel, then they’re probably doing o11y wrong.

You can run it as a collector to scrape log files. If your apps are instrumented, they can emit telemetry via OTLP to otel instead. Then otel can process and export the data to various data backends like Minor (metrics), Loki (logs), and Tempo (traces). Then use Grafana for a UI. That particular set of tools is known as the LGTM stack. if you only want to handle logs, your stack could be simpler: otel, Loki, and Grafana.

A final thought is about a seeming want for metrics generated from logs. Otel can do that for you too.

I think you could also use awstats, which runs on the log files produced by your web server.

I did something very similar with Opensearch rather than grafana, but it’s definitely possible. My setup:

• fluent-bit installed on webserver to scrape and parse nginx logs, then forward them over TLS to the monitoring server
• on the monitoring server, a second fluent-bit service runs here to collect the forwarded logs and insert them into the correct index pattern. A filter also inserts geoip lookups into the records.
• opensearch & dashboards set up to exclude known “bot” user-agents from the analytics, and do some other basic data cleanup to make the dashboards pretty

It works well, but could be a bit simpler admittedly. You may choose to use Loki instead of Opensearch/Elasticsearch, and there are plenty of other log parsing tools out there.

Another, much simpler option is to just run Goaccess on your log files, either periodically to generate reports, or as a daemon to create a live dashboard.