Anglo-French talks over a migration deal were deadlocked on Wednesday night, with negotiators haggling over how much Britain will pay towards the cost of policing small boat crossings.

Keir Starmer had hoped to be able to announce a returns deal – under which Britain would send back some asylum seekers once they had crossed the Channel – before the conclusion on Thursday of the three-day state visit by the French president, Emmanuel Macron.

But just hours before the start of Thursday’s bilateral summit, aides on both sides said there remained several significant hurdles to a deal, including money, potential legal challenges in France and opposition from other European countries.

One British official said the talks were “complex” and “fluid”, adding that they needed to be agreed not only with France but also in consultation with other EU members.

A Downing Street spokesperson said the prime minister was hoping to make “concrete progress” on a range of issues including irregular migration at the summit.

A French source, meanwhile, said Paris’s demands for more money to help with policing on its northern coast were “clearly very politically sensitive” for the UK.

Macron’s visit continued on Wednesday with talks between the two leaders in No 10, followed by a visit by both men to the British Museum to formally announce the loan of the Bayeux tapestry to the UK.

The visit has been symbolically important to Starmer, who wants to show that his EU “reset” has helped restore good relations with the UK’s main European partners.

But it is also fraught with complexity, not least on the issue of irregular migration, which has caused both leaders significant problems with their domestic electorates.

Macron and Starmer spent much of their time in Downing Street on Wednesday discussing illegal migration. A No 10 spokesperson said afterwards: “The leaders agreed tackling the threat of irregular migration and small boat crossings is a shared priority that requires shared solutions.”

Starmer has prioritised signing a “one-in, one-out” deal which would have led to Britain accepting asylum seekers who could show a family connection to the UK, but returning those who could not.

The French newspaper Le Monde reported on Wednesday that an initial pilot scheme would mean Britain would send back only 2,600 people a year – about 6% of the total number of crossings. But British officials hope they can increase those numbers after any pilot scheme ends.

With negotiations going on into Wednesday evening, aides on both sides said there remained significant areas of disagreement, one of which is money.

The UK signed a £480m deal two years ago to pay for additional border patrols and surveillance equipment such as drones and night-vision binoculars. Since then the French have agreed to intercept boats in the sea which are up to 300 metres from their shore, and are now asking for extra funding to pay for police officers, boats and drones to enforce that policy.

France has been looking for money to expand its Compagnie de Marche, a specialist unit of officers which operates on beaches to tackle the people-smuggling trade and intercept crossings.

Other obstacles to the deal include opposition from the southern European countries of Cyprus, Greece, Italy, Malta and Spain, all of which have experienced high levels of irregular migration.

This so-called Med 5 group is concerned that asylum seekers who are sent back from the UK to France could then decide to travel back through Europe to their countries.

Starmer, meanwhile, has sought to assuage Macron’s concerns about the UK’s hidden economy, which the French president believes acts as a “pull factor”, encouraging migrants to cross the Channel and find illegal work.

The PM has announced a number of policies to tackle illegal work, including more immigration enforcement raids. One government aide said: “If the French were asking us to do more illegal working raids, we would be fine with that.”

A No 10 spokesperson said Starmer had used Wednesday’s meeting to stress the action the UK is taking on this front.

“[He] spoke of his government’s toughening of the system in the past year to ensure rules are respected and enforced, including a massive surge in illegal working arrests to end the false promise of jobs that are used to sell spaces on boats,” the person said.

One final concern on the French side, diplomats said, is that their policy of stopping boats 300 metres away from shore may be susceptible to challenge in their domestic courts.

UK sources said on Wednesday the deal was still a possibility, although some European officials were pessimistic about the chances of it being agreed during this visit.

One French diplomatic source described the idea as “a hostage to fortune”, saying the UK had made few guarantees about how many people they would take and that agreement with the wider EU looked as remote as ever.

Britain and France also said on Wednesday they would be willing to mount a joint nuclear response to an extreme military threat – possibly from countries such as Russia and China – and that “any adversary threatening the vital interests of Britain or France” could be confronted by the “nuclear forces of both nations”.

The joint nuclear declaration, a first, forms part of a deepened Lancaster House defence cooperation agreement announced between the two countries, which also sees the two place orders for more Storm Shadow missiles, and begin work on a next generation anti-ship missile together.

The UK described the new agreement as an “entente industrielle” and said it would support more than 300 jobs at the Stevenage production line where Storm Shadow missiles are made by Anglo-French manufacturer MBDA. The number of new missiles being ordered was not made clear.