Amazon typically has a few vendors that specialize in refurbished Optiplexes and/or HP Elites in small or ultra small form factor sizes.
A word of caution about these refurbs though...the memory and storage they include are often dollar store brands (Kingfast) that I wouldn't even trust for a child's PC. It's worth purchasing your own after the fact.
You've clearly done your homework, and you've gotten a lot of good feedback already, so I'll just add a few points...
Storage options: Personally, I'd replace the existing drive with the highest capacity I could afford. In an ideal situation, I'd keep the host on another drive (NVMe or flash) and dedicate the large drive to a single partition of data storage.
In my own mini-PC (8th gen NUC), I've got a smaller NVMe for Proxmox and a single 8TB internal SSD for data.
Encryption: If you're going to bother with encryption, I wouldn't half-ass it. Why bother at all if you're fine using auto-decryption or a weak password that will be guessed with any sizeable effort? Just lock it down with a strong password and decrypt/mount the data drive after any reboot; making a shell alias or script for this is trivial. You're likely not rebooting the server more than once a week anyway.
Budget/Specs: I get the sense you don't have much budget right now, but knowing your hardware would help in suggesting solutions. Do you have an NVMe slot? What is the make/model of the motherboard and case?
Filesystem: For simple storage, this really doesn't matter and Ext4 will probably be fine. It's a mature, robust, no-frills filesystem which is perfect for bulk file storage (docs, music, videos, etc.), but Btrfs would be fine too if you want more options.
USB Docking Stations: I've had really good experiences with USB docking stations like this one, and I currently use it for attaching my backup HDDs each month. I wouldn't want to rely on them for realtime data access, but they do work wonderfully for backups and one-off drive access.