The invitation was extended by Kim's sister Kim Yo Jong during a lunch meeting with Moon at Blue House, the presidential palace in Seoul.
The meeting led to a series of reunions of families separated by the 1950-53 Korean War, as well as the launch of a joint factory park in the North's border town of Kaesong in 2004.
Moon did not immediately accept the invitation.
Washington insists that Pyongyang - which is under multiple sets of UN Security Council sanctions - must show a willingness to give up its weapons before any negotiations can happen.
Moon shook hands with both Kim Yo Jong and Kim Yong Nam at the Olympics opening ceremony and they cheered as athletes from North and South entered the arena together behind a unification flag showing an undivided Korean peninsula.
Correspondents say the invitation puts Mr Moon in a hard position as he campaigned on a promise to engage with the North, but his moves in that direction go contrary to the wishes of his ally the US.
"We have to be unified (with North Korea)".
The diplomatic honor bestowed upon North Korea was rare: Moon met Kim Yo Jong, the first time a member of the Kim dynasty has ever visited the country, four times during the visit.
"If the North was genuinely interested in improving relations with the South, it would.(have) refrained from conducting a military parade on the eve of the Olympics which is contrary to the Olympic spirit", Stangarone added.
"Although it's my first time here, it doesn't feel unusual".
"She (Kim Yo Jong) is not charming, she is a dictator's sister", Laila said."She is not looking after North Korean people".
The two teams marched together at an Olympics opening ceremony for the first time since 2006.
Kim Yo Jong, the younger sister of Kim Jong Un, "courteously" handed over a personal letter from the North Korean leader to Moon during talks on Saturday and also told Moon of her brother's "intention", KCNA said, without elaborating on what that was.
At the opening ceremony for the Games, Mr Pence, Kim Yo-jong and Kim Yong-nam were seated in close proximity to each other.
Pence fired off a tweet calling the North's participation in the games a "propaganda charade" and saying the world must not "turn a blind eye to the oppression and threats of the Kim regime".
Kim Yong Nam, the North's nominal head of state who was also at Saturday's meeting, said "even unexpected difficulties and ordeals could be surely overcome and the future of reunification brought earlier when having a firm will and taking courage and determination to usher in a new heyday of inter-Korean relations".
He made it clear from the start that he came to the Games with a message of "maximum pressure" on Pyongyang.