For centuries, it remained as the one record that could never conceivably be challenged.
No matter how much faster, stronger, tougher people got, at the end of the day, he was just a freak of nature. No-one had come close to swimming right around the world before him, and after him, there was a marked continuance of no-one getting close to it.
His body was perfectly suited to the task, he trained hard, and while it took him weeks, he did it. Legitimately, honestly, fairly.
Just a shame it was before civilisation had developed and thus no-one knew about it.