I mean for a hobby project that no one cares about sure. Otherwise the whole CI/CD process was invented exactly to avoid having devs push untested and untrackable crap on production servers. So once there are more than two people in a team and paying customers with access to a lawyer that's going to be a hard pass.
Anyway the main reason your CI/CD are slow is that you're using $5 workers with 1Gb ram. There's a reason the build is faster on your 12 core/64Gb laptop, the issue is usually not the process, the issue is being cheap on the infrastructure. The only good thing about GitHub CI workers is that they are cheap but performance wise they are garbage.