At least part of it might also just be Federation snobbery.
None of the crew on the Enterprise complained of replicated food, and for the most part, the attempts at actual cooking tended to be dismal (Rikers scrambled eggs were inedible to human palates). The only complaint we had was Troi wanting a "real chocolate sundae", which the computer seemed happy to provide, if it didn't exceed her nutritional intake guidelines.
The Federation prides real things and real experiences over something they consider fake, and they might consider replicated food to be in that category, even if it is otherwise fine and near-indistinguishable.
We do also know that the computer will remove poisonous substances (like removing the seeds and stems from replicated apple slices) and make as best a balanced nutritional profile it can in what it creates, which might alter the taste a bit, similar to how some healthy-variant recipes aren't as nice as their counterparts, since they lack the salt and fat for flavour.