It was a bit more than that.
https://en.wikipedia.org/wiki/Lawrence_Summers
He explicitly said that his best guess (in 2005) was that when it came to the radical difference in male/female faculty representation in the sciences, the 'largest phenomenon, by far' was 'issues of intrinsic aptitude', followed by 'what are in fact lessor factors involving socialization and continuing discrimination.'
He's making a much stronger assertion that the variability hypothesis is true; he's asserting that 'intrinsic aptitude' was the largest factor in play. Even if he believed the variability hypothesis to be true (though your wikipedia link doesn't appear to give much reason to think he did, nor that he should have) isn't a meaningful support for his claims, except to mean that they aren't obviously false.
Even at the time, that seemed to be unsupported by the evidence available. Having just skimmed your link, it seems like his claim is less-supportable now than it was then, if not clearly false.