The troupe, cheered by dozens of fans as their bus pulled up under tight security before the game, have been dispatched south as part of a North Korean charm offensive after months of fiery rhetoric threatening nuclear war and provocative missile tests.
Pence fired off a tweet calling the North's participation in the Games a "propaganda charade" and saying the world must not "turn a blind eye to the oppression and threats of the Kim regime".
Pence told reporters that the two leaders "spoke as friends" during his visit to South Korea, and their discussions including the meeting between Moon and the North Korean delegation, led by Kim Jong Un's sister, Kim Yo Jong.
"South Korea has some hard homework to solve regarding some countries", President Moon said in a speech before Pence arrived.
The North's official Korean Central News Agency also referred to Moon as president.
Many analysts are skeptical about Kim Jong Un's motivations for this sudden outreach, especially as he continues to assert his desire to press ahead with his nuclear weapons program to fend off the "hostile" United States.
A White House official said although Moon did not discuss the invitation with Pence, the South Korean president made it very clear that only when North Korea actually starts to take steps to denuclearise would anyone even consider beginning to take the pressure off.
"North Korea clearly appears to be winning the gold", said Kim Sung-han, who served as Korea's vice foreign minister in 2012-2013 and who now teaches at Seoul's Korea University.
Pyongyang conducted its largest nuclear test a year ago and in November tested its most advanced intercontinental ballistic missile that experts said has the range to reach anywhere in the United States.
Kim Yo Jong and Kim Yong Nam have invited Moon to visit Pyongyang for inter-Korean talks.
A Unification Ministry official in Seoul said that Pyongyang is believed to have said during discussions that it will not receive the fuel supply in order not to cause any trouble to South Korea.
U.S. Vice President Mike Pence, who attended Friday's opening ceremony along with the North Koreans, said "no daylight" existed between the United States, South Korea and Japan on the need to isolate North Korea.
Japan does not participate in the military exercises, but it is within reach of North Korean missiles and relies heavily on USA forces and their readiness to deal with that threat.
Prime Minister Shinzo Abe has left for South Korea to hold talks with President Moon Jae-in and attend the opening ceremony of the PyeongChang Olympics.
These officials said that while Pence did not greet Kim Yong Nam, he didn't deliberately skip over him.
The 19 other North Korean athletes participating in the games are three cross-country skiers; two short-track speed skaters; a pair of figure skaters; and 12 female hockey players who helped form the Koreas' first-ever joint Olympic team with South Korean players. The United States fought with South Korea and maintains tens of thousands of troops and an "ironclad" agreement to protect its ally.
Before being ejected, the two managed to tell an official their intention had been to show how the U.S. and North Korea could get along.
South Korea says North Korea has withdrawn a request that the South provide fuel to a North Korean ferry that transported more than a 100 artists for performances marking the Pyeongchang Winter Olympics.
Pence arrived late and stayed for just five minutes - and did not interact with the delegation from the North - at a Friday VIP reception for delegation leaders.