Depends strongly on the community. Every sub discipline has its own standards of respectability. Publishing outside of those constraints can cause articles to be ignored.

In part it's prestige, which for some might matter for promotion purposes, and at least personally I'm more like to cite journals for which I know I trust their judgement in peer review and submission acceptance. There are predatory publishers which abuse the open access concept to make money, and if I'm reviewing literature I don't want to have to also research if a journal can be trusted (unless of course the publication I want to include is novel or especially worthwhile).

Also, in many contexts open access requires payment by the authors; this may be fine if an author is in a large grant-funded lab or at an institution willing to fund the open access fee but for many of us non-research-track folks it's kind of a deal breaker.