Ant and Dec were once "strongly considering leaving ‘Britain’s Got Talent’" after feeling they were being "sidelined".
The TV presenting pair have admitted they held showdown talks with Simon Cowell, after feeling as though they "hardly seemed to feature" on the show, despite "very long" days of filming.
In their new autobiography ‘Once Upon A Tyne’, serialised in The Sun newspaper, Ant says: "It was on a trip to America, in 2012, where we had to have a serious conversation with Simon.
"The previous series of BGT had been especially tricky for us two.
"The audition days had been very long, which in itself we don’t mind, but after filming interviews and interacting with every single act, as well as making jokes, giving reactions and everything else we always do, we found that when the show went out on TV, we hardly seemed to feature in it at all.
"We began to feel that we were wasting our time doing the auditions because all the footage of the work we were doing was ending up on the cutting-room floor."
Dec adds: "And, what’s more, with Saturday Night Takeaway about to come back into our schedule, we could have been putting our efforts into that in January, when the auditions happen.
"The experience left us feeling like we were being sidelined."
The pair visited ITV bosses and told them they were thinking about departing the programme, but they agreed to have a meeting with Simon.
Dec adds: "We’d been to see the bosses at ITV and told them we were strongly considering leaving BGT.
"I remember waking up in Birmingham on the last day of the 2012 tour, opening my curtains and thinking, ‘Well, this is the last time I’ll be doing this’."
The trio met in an LA restaurant and sat outside "mainly so Simon could smoke", and the Geordie duo sat either side of Simon "so [they] could do the full pincer movement", before the pair "let him have it with both barrels".
Ant says: "We told him everything – that we weren’t being used, that we could be doing other stuff, that maybe someone else should take over."
Simon listened to the conversation "intently" for "four, maybe five cigarettes", and Dec says he promised the pair they would never feel that way again.
He adds: "He desperately wanted us to stay and we told him we’d do the next series and see how it went – and, to be fair to him, he was as good as his word, things did change."
Simon admitted he "respects" the pair for telling him to his face if they are "p***** off".
Also writing in the book, he says: "We had to come up with some kind of compromise, which now means we work earlier."
What’s more, Dec admits one of the main elements of a ‘BGT’ audition day is "waiting for Simon Cowell to turn up", and the music mogul says the pair "have a real issue with me being late and it used to drive them nuts".
Ant quips: "We’ve still got a real issue with him being late and it still drives us nuts."
The ‘BGT’ semi-finals will return on September 5th, but Simon will not be on the judging panel after recently breaking his back when he fell off his electric bike.