I've just counted 100 communities I had to block to get a semi interesting "local" feed, in addition to 7 posters who were really not my taste. How many hentai/anime communities are really needed for example? So yes, downvoting makes sense as long as there is a local feed, that's how lemmy works.
And yes, I have witnessed a clear improvement since downvotes are back.