The nation's three largest unions have backed the ACTU's call for Victorian construction union boss John Setka to resign for the good of the labour movement.
Their backing comes as a Labor MP has hinted Mr Setka's refusal to go is straining the relationship between the political party and his Construction Forestry Maritime Mining And Energy Union.
ACTU secretary Sally McManus on Thursday demanded Mr Setka resign as secretary of the Victorian CFMMEU branch.
Mr Setka is facing court later in June when he is expected to plead guilty to two criminal charges, including one of using a carriage service to harass a woman.
The outcry over his leadership has grown louder in the past week after he was accused of telling union colleagues that anti-family violence campaigner Rosie Batty's advocacy had led to men having fewer rights.
Mr Setka has denied making any remarks denigrating Ms Batty at the union executive meeting.
His union colleagues are demanding an audit of the phone records of those involved in the meeting to discover how it was leaked.
Ms McManus says he must quit the state branch leadership for the good of the whole union movement.
While Mr Setka won public support from the construction arms of the Victorian and NSW branches, Ms McManus has been backed in by the three largest unions in the country.
The Australian Nursing and Midwifery Federation, the Shop, Distributive & Allied Employees Association, and the Australian Education Union - representing largely female workforces - have a combined membership of more than 650,000.
The CFMMEU has about 130,000 members.
ANMF federal secretary Annie Butler said her organisation backed the ACTU's resignation demand.
"The ANMF has, and always will have a, zero tolerance toward any form of violence and we are working hard with our members to combat family violence," she told AAP
SDA national secretary Gerard Dwyer said Mr Setka should resign his leadership position in the best interests of his union.
"There are no acceptable degrees of domestic violence and it is inconsistent with trade union values," he said.
AEU federal president Correna Haythorpe and secretary Susan Hopgood said union leaders had to call out any behaviours that damaged the movement and its members.
The Victorian Trades Hall council met on Friday, but would not comment on any discussion about Mr Setka's fate.
It made a statement on Tuesday saying the council "condemns all forms of gendered violence" without mentioning the construction unionist by name.
Meanwhile, federal Labor frontbencher Michelle Rowland said further action should be considered if Mr Setka continues to dig in.
"It is incumbent on this individual to consider what that means for his union, what that means for the relationship between that union and the Labor Party," she told Sky News.
Asked whether that could involve Labor disaffiliating the CFMMEU Victorian branch, she said: "I think they should all be considered."
Labor leader Anthony Albanese is moving to expel Mr Setka from the party. The union boss has already had his membership suspended.
The NSW branch of the union called on Mr Albanese to drop this push, but he was standing firm on Friday.
"He (Setka) needs to listen to Sally McManus as the secretary of the ACTU," Mr Albanese told Nine's Today program.
"His ongoing presence as Victorian secretary of the CFMMEU is not in the interests of the union movement as a whole."
Australian Associated Press
