TUI Travel, Europe's largest tour operator, said last week it had received an approach regarding a potential nil-premium, all-share merger with the German company, which owns 56.4 percent of its shares.
TUI AG is focusing on tourism after selling a majority stake in container shipper Hapag-Lloyd, and has long been looking at a potential deal with TUI Travel to save on costs, sources close to the matter have told Reuters.
The German group said on Wednesday a share-based transaction was not attractive at current prices.
However, a person familiar with TUI AG's thinking said a merger was not completely off the cards for the future, though TUI AG first needed to become more efficient.
"The form it takes is of secondary importance. Basically, merging the two makes sense," the person said.
TUI Travel said the two companies had held detailed talks, but failed to reach an agreement, adding it was not in discussions with any other parties about being bought.
TUI Travel shares have surged 67 percent over the past six months, helped by problems at rival Thomas Cook, outperforming a 57 percent increase at TUI AG. The British-listed firm has a market value of 3.3 billion pounds ($5.2 billion) versus TUI AG's 1.9 billion euros ($2.5 billion).
At 1020 GMT, TUI AG shares were down 5.5 percent at 7.5 euros, while TUI Travel was 4.5 percent lower at 279.1 pence.
Equinet analyst Jochen Rothenbacher said a share-based deal at current rates was not in the best interests of TUI AG investors, but that another deal structure could yield benefits.
Sources familiar with the matter told Reuters last week that TUI AG's biggest shareholders, Russian tycoon Alexey Mordashov with 25 percent and Norwegian shipping magnate John Fredriksen on 15 percent, were interested in a tie-up via a so-called "reverse takeover" of TUI AG by TUI Travel.
It was not feasible for TUI AG to buy the 43.6 percent of TUI Travel it does not own because it lacks the funds and has little prospect of raising money on equity markets, given it trades at a hefty discount to peers, one of the sources said.
One of the sources also said cost savings from such a deal could reach 500 million euros, although some analysts have put the figure much lower, at around 100 million.
Any costs savings would be welcomed by TUI Travel, which returned to Britain's FTSE 100 benchmark stock index in December. Over the next few years analysts expect austerity measures to hold back demand for leisure travel across Europe.
"Our model assumes TUI Travel's tour package revenues remain essentially flat over the next five years," said Morningstar analyst Dan Su.
TUI Travel was formed in 2007 after the merger of TUI AG's travel business and Britain's First Choice. TUI AG has hotels and luxury cruise operations and is looking to sell its remaining interest in shipper Hapag-Lloyd.
The statement from TUI AG means that under UK takeover rules it is not allowed to make an offer for TUI Travel within the next six months. It had been given a deadline of close of business on Feb 13 to make a firm proposal or walk away.