This reworks the HTML parsing logic, to gracefully use `html5lib` on
non-compliant HTML 5 documents. This warning softens the failure mode
for users who are using commercial package index solutions that do not
follow the requisite standards and serve malformed HTML documents.

The earlier variant _returned_ an iterable object from a generator. This
did not properly handle the fallback, resulting in the html5lib code
path not being executed.

explicit{ely -> ly}

The html5lib library isn't strictly required as the same functionality
can be achieved through the stdlib html.parser module.

The html5lib is one of the largest uses of the six library. By dropping
this unnecessary dependency, the pip project is closer to dropping the
six library.

Additionally, html5lib maintenance has slowed down and the project has
rejected pull requests to drop Python 2 support.

For now, the html5lib code remains, but is gated behind a command
line option: `--use-deprecated=html5lib`. After a sufficient amount of
time has passed without any reported bugs, the vendored library and this
flag can be removed completely.

This serves as an opt-out from build failures causing the entire
installation to abort.

This behaviour is more forgiving when a source distribution cannot be
installed (eg: due to missing build dependencies or platform
incompatibility) and favours early eager failures instead of trying to
ensure that a package is installed regardless of the amount of effort it
takes.

This is just a preference, I'm approaching a new Nox ability.

Co-authored-by: NTzu-ping Chung <uranusjr@gmail.com>
Co-authored-by: NPradyun Gedam <pradyunsg@users.noreply.github.com>

These are newly vendored and this "upgrade" is the wrong verb to use.