CDNs make the download part much less of a problem.
And your server could easily send a fully rendered page on the first page load when it receives a full URL (one which was made by pushState and linked elsewhere) and still subsequently load pages via XHR. So it wouldn't have to parse any JS on first load -- subsequent loads would, but they'd be saving time from not downloading as much and not refreshing the entire page.
And your server could easily send a fully rendered page on the first page load when it receives a full URL (one which was made by pushState and linked elsewhere) and still subsequently load pages via XHR. So it wouldn't have to parse any JS on first load -- subsequent loads would, but they'd be saving time from not downloading as much and not refreshing the entire page.