With new set implementation, this became really costy.  Optimize it.
There's more to be done, but this shaves off most of the fat.

Part of fixing https://bugs.chromium.org/p/chromium/issues/detail?id=782220