Baker said leaving Doctor Who and the role of the Doctor was rough for him, saying he was happier when he was “being unreal” than real, which speaks to why Baker’s Doctor has the longest tenure of them all, with seven seasons.
Baker did follow up in saying that he’s not a big fan of reality in general, so there’s a bias but that he’s happier now in the present than he’s ever been.
Tom Baker left Doctor Who in 1980 after just one season of working with John Nathan-Turner, who remained the head of the show until 1989, when it was taken off the air until Paul McGann’s stint in 1996.
And no producers are going to come in and tell him otherwise.