But it is particularly concerning for cheap, uncertified chargers. These often lack appropriate levels of protection and can be a fire hazard.
I mean, you could hypothetically have an unsafe charger that plugs into wall power, but I don't think that that's specific to chargers. Any electrical device that plugs into wall power could hypothetically be unsafe.
In the case of chargers, the power supply is external to the device being powered and uses a standard interface, so it's easy to examine and replace. I think that the only thing that comes close are external, semi-standardized power supplies with barrel plugs. So if you want to make sure that you have, say, all UL-marked chargers (in the US; a CE mark isn't really the same thing in the EU but is the closest analog that I'm aware of) you can do that fairly easily compared to ripping an internal power supply out of a device. But I'm not convinced that USB chargers in particular are especially problematic relative to other forms of power supply or wall-power-connected device.