In general, my suggestion is via webpack+babel to do two build configurations. One for "modern" browsers (async/await support an higher) and one for "legacy" browsers (everywhere else). Modern ones are fairly easy to detect server-side and inject the correct entry point(s) into your html delivery.
In the top of my "modern" entry, I do a test for async/await, and if it fails, will set a cookie and `location.reload()`. This forces legacy should it not have detected properly. In the end, without most of the polyfills or the corejs bits there's a huge savings in overall size, and load.
My current work is mostly internal applications, where I don't have to work around supporting legacy browsers fortunately. Even keeping firefox + chrome happy is enough work in some cases.