There had been longstanding rumours throughout most of Mr Tillerson's tenure of friction between Mr Trump and his secretary of state, a former Exxon Mobile Corp chief executive officer.
He announced that deputy CIA Director Gina Haspel will become the new director, and the first woman so chosen.
But Trump fired him even as he was returning to the capital, and Tillerson reportedly learned of it from Twitter, where the President announced his successor. "President Trump has demonstrated yet again that he is the Commander-in-Chaos", said Sen. His political obituaries were already being written after Trump undermined him or sidelined him on several issues, including the Iran nuclear deal (which Tillerson supported). I wanted to break it or do something and he felt a little bit differently. "With Mike Pompeo, we have a similar thought process". I think it's going to go well. "The relationship is very, very good, and that's what I need as secretary of state".
A senior White House official said Mr Trump asked Tillerson to step down on Friday but did not want to make it public while he was on a trip to Africa.
State Department Under Secretary Steve Goldstein echoed that sentiment in a statement to CBS News on Tuesday morning, saying, "The Secretary had every intention of staying because of the critical progress made in national security".
Tillerson's head was on the chopping block - as is everyone's in the Trump circle - from literally the day he took the job, given the President's style and mercurial temperament.
Pompeo, however, will need to find a way to grapple with a boss who has shown little regard for diplomacy and no qualms about undermining Tillerson with Twitter posts, current and former USA officials said. Last summer as tensions rose when Kim Jong Un and Trump traded insults, Tillerson said the US wanted to "resolve this through talks", rather than the military options the White House said were on the table.
Pompeo, a former Republican congressman from Kansas, has already been confirmed by the Senate for his current role at the Central Intelligence Agency, making it extremely likely that he will be confirmed for the State Department role.
Trump's public frustrations with Tillerson began well before that.
CIA Director Mike Pompeo speaks at the FDD National Security Summit in Washington, DC, U.S., October 19, 2017.
Potential talks with North Korea were apparently the catalyst for booting Rex Tillerson from the State Department.
As of Tuesday morning, a meeting between Trump and Kim hadn't been scheduled. Now it's clear he had one move in particular in mind.
Working as a clandestine officer in Thailand in 2002, Haspel reportedly was involved in the interrogations of two suspects, Abu Zubaydah and Abd al-Rahim al-Nashiri, several news outlets reported previous year.
Before being tapped to run the CIA, Pompeo spent six years as a Republican representative from Kansas.
Reports had swirled since a year ago of a schism in the Trump administration between the commander-in-chief and his top diplomat.
The rumors came a month after a bombshell NBC News report revealed Tillerson privately railed against the president and called him a "moron" at a closed-door meeting at the Pentagon.
Last autumn, Mr Trump publicly undercut the former Texas oilman by tweeting that he was "wasting his time" trying to negotiate with North Korea.