The anthropologist might see skeletal differences but they'd also pay attention to the manner in which the subject was buried or what possessions survived with them that could also serve as clues of the subject's identity in life.

Humans have sexual dimorphism, but it's a cultural thing that women wear skirts and men drink themselves to death instead of talking about their problems (both of these are jokes btw. I have a friend who wore kilts quite often and my mother drank herself to death)

Also, genetics is tricky, there a plenty of examples of people who do not fall into one category or another for these sexually dimorphic traits. There are people who have genetics from both sex, as well and differences in hormones distributions will causes these traits to appear or not appear.

Is a huge grey area.

You're confusing sex with gender