I think the second game was an improvement in gameplay over the first. IMHO games 1, 3 and 5 changed up the gameplay whereas 2, 4 and 6 refined the previous games. So if I had to choose one to have a "HD" upgrade, it'd be 2.
The funny thing about this meme is that this is almost the opposite of how they were developed. Civ II was written by Brian Reynolds in England, initially without access to the Civ II source, and even after he got it, he didn’t copy any code. Civ III, meanwhile, was based on SMAC, a game that was very similar to Civ II in its basic mechanics. Civ IV is the one big ground-up rewrite, which then formed the basis for Civ V and VI.
(Source: Designer Notes podcast, where Soren Johnson interviews other developers, including Sid Meier and Reynolds)
BTW, I disagree with the characterization of Civ IV as a polish. It is a much more fundamental reimagining than the iterative Civ III, which really only added strategic resources and great people to the formula.