Can you power it off from the command line without SSH? You may have a hardware problem that's keeping the board from responding to soft poweroff at all.
Provided the machine isn't writing to disk or holding unwritten data in a disk cache at the moment you press the button, you're unlikely to damage anything with a hard poweroff.