This Woman's Work lyrics

[Rolf.Peukert@Theoinf.TU-Ilmenau.DE (Rolf Peukert)]

Please excuse this probably silly question from a non-native
speaker, but in the lines

Ooh, it's hard on the man, 
Now his part is over. 

does "hard on" have the double meaning that i think it has?
(would make sense somehow)

ciao,
Rolf