I can't speak for NY, but...
In Los Angeles I used the busses, subway, and trains for two to three years going to work. For one year it was an hour each way riding 2 trains. After that, it was 2.5 hours each way switching between busses and trains five times.
While I truly appreciated the Metro, it was often not fun. Usually everything and everyone was fine. But, at times I'd be riding with drugged up dangerous acting people. Other times just super annoying people. Sometimes the trains would be packed shoulder to shoulder full of people. And sometimes, in the middle of LA, the train would stop, and say "everybody off" without an explanation, and everyone would exit the train and have to figure out where to go.
Once I was able to drive myself, I no longer had to worry about any of the issues I had before. All I had to deal with was traffic jams. Annoying, but I did feel safer.