Mr Trump tweeted that Mr Pompeo, "will do a fantastic job!" and thanked Mr Tillerson for his service.
Secretary of State Rex Tillerson in Brussels on December 6, 2017.
Gina Haspel, now deputy director of the CIA, will become the agency's first woman director of the CIA.
In a statement, Mr Trump said: "I have gotten to know Mike very well over the past 14 months, and I am confident he is the right person for the job at this critical juncture".
It was the latest shake up in Trump's administration.
Mr Trump last Friday asked Mr Tillerson to step aside, and the embattled top diplomat cut short his trip to Africa on Monday to return to Washington. Trump offered no explanation for the change. "His leadership has made America safer and I look forward to representing him and the American people to the rest of the world to further America's prosperity", Pompeo said in a statement.
Tillerson is also deeply unpopular within the State Department for his imperious management style and disregard for the traditions of diplomacy, which has led to a steady level of attrition throughout his tenure.
Last year, Tillerson said the United States should talk with North Korea, which Trump quickly shot down.
Trump and Tillerson, the former CEO of Exxon Mobile, have reportedly been at odds for months.