SeeJayEmm

I've started encountering a problem that I should use some assistance troubleshooting. I've got a Proxmox system that hosts, primarily, my Opnsense router. I've had this specific setup for about a year.

Recently, I've been experiencing sluggishness and noticed that the IO wait is through the roof. Rebooting the Opnsense VM, which normally only takes a few minutes is now taking upwards of 15-20. The entire time my IO wait sits between 50-80%.

The system has 1 disk in it that is formatted ZFS. I've checked dmesg, and the syslog for indications of disk errors (this feels like a failing disk) and found none. I also checked the smart statistics and they all "PASSED".

Any pointers would be appreciated.

Edit: I believe I've found the root cause of the change in performance and it was a bit of shooting myself in the foot. I've been experimenting with different tools for log collection and the most recent one is a SIEM tool called Wazuh. I didn't realize that upon reboot it runs an integrity check that generates a ton of disk I/O. So when I rebooted this proxmox server, that integrity check was running on proxmox, my pihole, and (I think) opnsense concurrently. All against a single consumer grade HDD.

Thanks to everyone who responded. I really appreciate all the performance tuning guidance. I've also made the following changes:

1. Added a 2nd drive (I have several of these lying around, don't ask) converting the zfs pool into a mirror. This gives me both redundancy and should improve read performance.
2. Configured a 2nd storage target on the same zpool with compression enabled and a 64k block size in proxmox. I then migrated the 2 VMs to that storage.
3. Since I'm collecting logs in Wazuh I set Opnsense to use ram disks for /tmp and /var/log.

Rebooted Opensense and it was back up in 1:42 min.

I'd like to start doing a better job of tracking the changes I made to my homelab environment. Hardware, software, network, etc. I'm just not sure what path I want to take and was hoping to get some recommendations. So far the thoughts I have are:

• A change history sub-section of my wiki. (I'm not a fan of this idea.)
• A ticketing system of some sort. (I tried this one and it was too heavy. I'd need to find a simple solution.)
• A nextcloud task list.
• Self-host a gitlab instance, make a project for changes and track with issues. Move what stuff I have in github to this instance and kill my github projects. (It's all private stuff.)

I know that several of you are going to say "config as code" and I get it. But I'm not there yet and I want to track the changes I'm making today.

Thanks

I can't seem to find anything so I was hoping someone here has run into this.

Does anyone know if there's a way to get reporting on a per application key basis or per bucket. I periodically get threshold alerts (usually the download cap) but that doesn't give me any idea of what utilization is triggering the alert. The reporting I can find is pretty rudimentary and account wide.

I'm going to start off but saying I know that self-hosting email can be a bad idea. That being said, I'm trying to de-googlfy my life and would like to experiment.

I have a VPS and a domain that doesn't get used for much at the moment. I'd like to try configuring a full mail suite on that domain and see if I can make it work. I've been looking into the various options on this list and was hoping for some feed back on options that people have used. If this works out it would be fairly low volume.

Ideally I'd like a full solution that includes web administration if at all possible. I think I'm leaning towards mailcow but it might be overkill.

I'd appreciate any input on what has or hasn't worked for people. Thanks.