Tony Abbott is standing by his threat to withdraw $3bn in federal funding for Victoria if the new state government tears up the contract to build the East West Link toll-road project.

The prime minister said he would like the incoming Labor premier, Daniel Andrews, to rethink his opposition to the Coalition-backed project, but this request was swiftly rejected.

Andrews said he would not break his election commitment and called on Abbott to agree to reallocate the money to other Victorian transport priorities.

Abbott, who had previously cast Saturday’s state election as “a referendum on the East West Link”, told parliament on Monday the new government should “honour the contracts that were duly entered into by the legitimate government of Victoria” before the election.

“I do hope that the incoming premier of Victoria does reconsider his misguided commitment to tear up the East West Link contract … because it is pretty clear that the people of Victoria do want the East West Link,” the prime minister said.

“We are only too happy to continue to deliver $3bn to the government of Victoria, provided it goes ahead with building the East West Link that it is contractually obligated to build.”

Describing his phone conversation with Abbott on Monday morning, Andrews said: “We had a discussion about some important matters. He invited me to break my commitment to the Victorian people around East West Link and I indicated to him in a very respectful way that I would not be doing that.”

Andrews told 3AW he would fulfil his promise to “end the secrecy around this project” and release all the documentation, including the full business case and the contracts. This would allow the people of Victoria to “make their own judgment about whether this thing stacks up”.

Andrews said he had “lots of things to build” and hoped Abbott would agree to allow the $3bn in federal funds to support other Victorian projects.

“I think it would be unwise if a federal government, having cut health and education, then decided the best thing for Victoria was to cut infrastructure funding,” Andrews said. “I don’t think the people of this great state would warm to that – I think they would react very badly, and I would too. He [Abbott] wants to be an infrastructure prime minister. Let’s get on and build some infrastructure.”

The foreign affairs minister, Julie Bishop, who visited Victoria during the campaign, said polls showed the majority of voters supported the East West Link project.

Weighing in to the debate over the impact of federal factors in the Victorian Coalition’s defeat after a single term, Bishop said the federal government had to take some “pretty harsh measures” to repair the federal budget but it was “simplistic” to suggest any one issue dominated the campaign.

“It’s obviously a very complex result. Of course I’m very disappointed and I hope that the Victorian Liberals and of course the federal Liberal party will look very closely at the results,” Bishop told the ABC.

“I think it’s interesting to note that Mike Baird, the premier of NSW, is doing exceedingly well in the polls, so if this was all about federal government policy then why is the state Coalition parties in NSW so far ahead in the polls? It’s the same federal government.”

Bishop said the Liberal party had not been expected to win the 2010 Victorian election and “sadly they got off to a bit of a shaky start” and this was complicated by the instability caused by the Liberal-turned-independent MP Geoff Shaw.

Senior Victorian Coalition members have said federal factors distracted from the state government’s message.

The former Victorian Liberal premier Jeff Kennett said there was “no doubt” the federal government, in particular its budget measures, had caused considerable community concerns.

The election also caused tensions between Nationals. The state Nationals leader, Peter Ryan, who announced on Monday he would step down as party leader, criticised the federal deputy leader, Barnaby Joyce, for an “inappropriate and untimely comment” about the role of the party in the battle for SPC Ardmona funding. Joyce told the ABC he apologised if his remarks had an impact.