He was voted in by the German parliament, where his three-party coalition has a substantial majority, and given a standing ovation.
His centre-left Social Democrats will govern alongside the Greens and the business-friendly Free Democrats.
The handover of power marks an end to Mrs Merkel’s 31-year political career.
The German parliament, the Bundestag, backed his appointment by 395 votes to 303, and he was then formally appointed as the ninth federal chancellor by President Frank-Walter Steinmeier.
After the vote in parliament, he was asked by Bundestag President Bärbel Bas whether he accepted the appointment and said “yes”.
He later returned to parliament to take the oath of office. Unlike his predecessor, he left out the religious reference “so help me God”.
Since the election, Mr Scholz’s party has worked with the Greens and the Free Democrats on a coalition deal, which was finally signed on Tuesday.
The new government has ambitious plans to fight climate change by phasing out coal early and focusing on renewable energy, but their initial priority will be tackling the coronavirus pandemic.
Health authorities have recorded another 69,601 cases in the past 24 hours and a further 527 deaths – the highest number since last winter.
A known face but not Merkel No.2
This is the moment the career politician has long been waiting for. After 16 years of Angela Merkel, it’s a case of out with the old and in with the new-ish.
Mr Scholz is already a known and trusted face in Berlin and Brussels. He marketed himself to voters as sort of Merkel Mark Two, despite hailing from a different political party.
But it’s not all about continuity, and friends and trade partners of this rich and powerful nation will be watching closely.
Mr Scholz’s coalition government is a never-before-seen marriage of convenience. What unites them, they claim, is a determination to modernise Germany, while preserving the country’s treasured stability.
Immediate challenges for them are:
- Covid – Germany is in the midst of a pernicious fourth wave and considering mandatory vaccination
- The threat of a Russian invasion of Ukraine
- Angela Merkel was accused of putting trade before politics. Team Scholz is expected to be somewhat tougher on Moscow and Beijing despite the potential economic hit to German business.
Relations with Washington may improve as a result, though, and that is a declared priority of Olaf Scholz’s fledgling coalition.
Mr Scholz’s first foreign trip as chancellor will be to Paris and Brussels on Friday. He and Greens joint leader Annalena Baerbock, who becomes foreign minister, will have to respond to fears surrounding Russia’s military build-up near the border with Ukraine.
Although Russia has denied plans to invade its neighbour, Angela Merkel agreed with US President Joe Biden and the leaders of the UK, France and Italy late on Tuesday that they would adopt a joint strategy to respond by imposing “significant and severe harm on the Russian economy”.
One obvious economic measure would be to threaten Russia’s Nord Stream 2 gas pipeline to Germany, which has been completed but is still awaiting approval from the German energy regulator.
US officials say they have reached an understanding with Germany that the pipeline would be shut down, which would be a significant intervention.
In a message of congratulations to Mr Scholz, Russian President Vladimir Putin said he looked forward to constructive dialogue, and hoped that Germany continued to recognise “there’s no alternative to dialogue”.
But she then helped oust him and became Christian Democrat leader in 2000, before becoming chancellor herself in November 2005.
She will still have an office close to the Bundestag, in a flat that was once used by Margot Honecker, once dubbed the most powerful woman in communist East Germany.