No need to beat yourself up, just make simple changes to improve. Write a one paragraph summary for yourself after each chapter, or list key points you want to remember. The next day, review it and take 10 minutes to think about how that chapter applies to your life or other things you've read.
Once you've finished the book, rewrite & combine your notes into one or two pages and flip back through the book to find a quote or two that stands out to you. Then set a reminder on your calendar one month in the future to review your summary page.
Note-taking, rewriting notes, and spaced repetition are all proven study techniques. It's a little extra time, but it's nothing compared to the time you're spending reading, and it'll make a big difference in how well you remember your takeaways from it.