Mrs Merkel's conservative Christian Democratic Union was defending its 19-year hold on Hesse, previously a stronghold of the centre-left Social Democrats, the chancellor's federal coalition partners in Berlin.
What does Ms. Merkel's decision to give up the party leadership mean?
In the end, the move won't just be good for her, but for her party, too.
According to "multiple media reports", Merkel announced her intention to retire as the chief of the CDU.
She has led Germany as chancellor since 2005 - and wants to stay in that post.
Young people in Germany can scarcely remember a time before Angela Merkel, 64, was their Chancellor.
This position has allowed the so-called grand coalition which presently rules Germany, an alliance between the main centre-right CDU and main centre-left SPD, from parties which should, in theory, be opposed on government but are united in keeping other voices out of power.
At the moment, it is still too early to tell whether she will be able to govern effectively as a lame-duck chancellor, or if it will strengthen her coalition, said Thorsten Faas, political science professor at Berlin's Free University.
The country's far-right Alternative for Germany party on Sunday secured about 12% of the vote and will enter the region's assembly for the first time.
Merkel now governs Germany in a "grand coalition" of what traditionally have been the country's biggest parties-the CDU, its Bavaria-only sister, the Christian Social Union, and the Social Democrats. Political experts see this as Merkel responding to voters after recent elections and calming the nerves within her party. Her current coalition took office only in March, after six months of tortuous negotiations, but has become notorious for its squabbling.
The shock decision signals the beginning of the end for a chancellor who put her stamp on Europe and beyond defending moderation and liberal values that have increasingly come under attack.
Merkel said she won't seek to return as her party's leader when the CDU meets in December - setting off speculation over who in her party might try to replace her. Since then, both Merkel and her CSU allies have been criticized for their management of the influx.
Andrea Nahles, federal chairwoman of the Social Democratic Party SPD, gives a statement at the SPD headquarters in Berlin, on October 28, 2018.
A very poor performance in Hesse could embolden critics to push for the Social Democrats to leave the federal coalition, and endanger the job of party leader Andrea Nahles.
She said she hopes to open the way for "new success for the CDU" by letting it prepare for her departure as chancellor, and she won't interfere with the choice of a successor. Such an outcome would increase pressure from the SPD's rank-and-file for the party to pull out of the coalition with Merkel in Berlin.
There are two main names coming up as top contenders for Angela Merkel's place as head of the CDU, in Freidrich Merz and Annegret Kramp-Karrenbauer.