Cheryl, Louis and Mel face serious talks with Simon Cowell and other show bosses ahead of the live shows.
The talks are after this years series has rated well below last years run of The X Factor and has had millions of viewers less than when both Cheryl and Simon were on the panel back in 2010.
And for a third time in a row this year, last night saw rival show Strictly Come Dancing come out top in the battle of the ratings.
The Sun reports that Cheryl, Mel and Louis are in danger of being axed in the lead up to the live shows.
“This year we need to please everyone — the show needs to be bigger and better. We have to kill every other show. We have to do it because our necks are on the block,” Louis revealed.
He told the newspaper: “Simon will do whatever it takes to make the show big, and he will fire someone if they didn’t do a good job.
“There’s no room for passengers. Simon wants to be the No 1 show. No matter what he’s doing he always wants to be No 1.”
Louis admitted that there was pressure on the judges from Simon to find a ‘good’ winner who would actually have a successful music career.
He explained: “There has absolutely been disappointment about winners in the past. But then the public pick, and if they pick people like Sam Bailey or that awful Steve Brookstein, then there’s nothing we can do.
“But having Simon back has changed things. He wants people who will sell records and be famous. That means more pressure on us — but it is a good pressure.”
The X Factor continues next weekend.