[cherry-pick] keep the size of symmetric padding is 2 for 2d and 3 for 3d. test=release/1.6 (#20903) (#20939)