Out of the games I've played, OSU. I am pretty average at rhythm games where it's like Project Sekai or the Miku Diva style games where all you have to do it wait and click a button or tap somewhere specific at a fixed location on screen, but I absolutely suck at the whole move the mouse and click thing. Just as bad with mouse as when I tried with my beginners tablet.
Most other games I play anymore are games I know I'm at least decent at, so I don't have many games I'd consider the hardest or even to compare those too. Though, while writing this and thinking about it, I'd say I might compare OSU to Vib-Ribbon in general, default songs or not, and possibly even give it a close second for difficulty. And that's despite it being more of a wait and click type rhythm game in my eyes.