Bug #5899, reported by Marko Tiikkaja.

Heikki Linnakangas, reviewed by Kevin Grittner and Dan Ports.

The MSVC compiler complains if a macro is called with less arguments
than its definition provides for. flex generates a macro with one
argument for yywrap, but only supplies the argument for reentrant
scanners, so we remove the useless argument in the non-reentrant
case to silence the warning.

we know what should be stored in there.

Strip leading and trailing whitespace and replace interior whitespace
by a single space.  This avoids problems with the index generator
producing duplicate index entries for terms that differ only in
whitespace.

Commit dca30da3 actually fixed all the
indexterm elements that would cause this problem at the moment, but in
case it sneaks in again, we're set.

The previous coding failed to account properly for the costs of evaluating
the input expressions of aggregates and window functions, as seen in a
recent gripe from Claudio Freire.  (I said at the time that it wasn't
counting these costs at all; but on closer inspection, it was effectively
charging these costs once per output tuple.  That is completely wrong for
aggregates, and not exactly right for window functions either.)

There was also a hard-wired assumption that aggregates and window functions
had procost 1.0, which is now fixed to respect the actual cataloged costs.

The costing of WindowAgg is still pretty bogus, since it doesn't try to
estimate the effects of spilling data to disk, but that seems like a
separate issue.

Per report from Andrew Dunstan.

Teach the program and script to deal with OID-array referencing columns,
which we now have several of.  Also, modify the recommended usage process
to specify that the program should be run against the regression database
rather than template1.  This lets it find numerous joins that cannot be
found in template1 because the relevant catalogs are entirely empty.

Together these changes add seventeen formerly-missed cases to the oidjoins
regression test.

Most of these cast DWORD to int or unsigned int for printf type handling.
This is safe even on 64 bit architectures because a DWORD is always 32 bits.

In one case a variable is initialised to keep the compiler happy.

Per experimentation with contrib/citext, whose hash function assumes that
it'll be passed a collation.

This test should now work in any database with UTF8 encoding, regardless
of the database's default locale.  The former restriction was really
"doesn't work if default locale is C", and that was because of not handling
mbstowcs/wcstombs correctly.

These functions should take a pg_locale_t, not a collation OID, and should
call mbstowcs_l/wcstombs_l where available.  Where those functions are not
available, temporarily select the correct locale with uselocale().

This change removes the bogus assumption that all locales selectable in
a given database have the same wide-character conversion method; in
particular, the collate.linux.utf8 regression test now passes with
LC_CTYPE=C, so long as the database encoding is UTF8.

I decided to move the char2wchar/wchar2char functions out of mbutils.c and
into pg_locale.c, because they work on wchar_t not pg_wchar_t and thus
don't really belong with the mbutils.c functions.  Keeping them where they
were would have required importing pg_locale_t into pg_wchar.h somehow,
which did not seem like a good plan.

Make use of the collation attached to the index column, instead of
hard-wiring DEFAULT_COLLATION_OID.  (Note: in theory this could require
reindexing btree_gist indexes on textual columns, but I rather doubt anyone
has one with a non-default declared collation as yet.)

Experimentation with contrib/btree_gist shows that the majority of the GIST
support functions potentially need collation information.  Safest policy
seems to be to pass it to all of them, instead of making assumptions about
which ones could possibly need it.

Using DEFAULT_COLLATION_OID in the comparePartial functions was not only
a lame hack, but outright wrong, because the compare functions for
collation-aware types were already responding to the declared index
collation.  So comparePartial would have the wrong expectation about
the index's sort order, possibly leading to missing matches for prefix
searches.

Since both tarballs and git now result in a "postgresql" directory
rather than a "pgsql" directory, adjust the example configuration to
look for the former.

This matches the maximum line length that pgindent uses.

This patch is almost entirely cosmetic --- mostly cleaning up a lot of
neglected comments, and fixing code layout problems in places where the
patch made lines too long and then pgindent did weird things with that.
I did find a bug-of-omission in equalTupleDescs().

Per report from Jeff Janes.

Without this, adding an attribute to a typed table with an inheritance
child fails, which is surprising.

Noah Misch, with minor changes by me.

This could cause failures when DROP OWNED BY attempt to remove default
privileges on sequences.  Back-patching to 9.0.

Shigeru Hanada

This syntax allows a standalone table to be made into a typed table,
or a typed table to be made standalone.  This is possibly a mildly
useful feature in its own right, but the real motivation for this
change is that we need it to make pg_upgrade work with typed tables.
This doesn't actually fix that problem, but it's necessary
infrastructure.

Noah Misch

If we find a DELETE_IN_PROGRESS HOT-updated tuple, it is impossible to know
whether to index it or not except by waiting to see if the deleting
transaction commits.  If it doesn't, the tuple might again be LIVE, meaning
we have to index it.  So wait and recheck in that case.

Also, we must not rely on ii_BrokenHotChain to decide that it's possible to
omit tuples from the index.  That could result in omitting tuples that we
need, particularly in view of yesterday's fixes to not necessarily set
indcheckxmin (but it's broken even without that, as per my analysis today).
Since this is just an extremely marginal performance optimization, dropping
the test shouldn't hurt.

These cases are only expected to happen in system catalogs (they're
possible there due to early release of RowExclusiveLock in most
catalog-update code paths).  Since reindexing of a system catalog isn't a
particularly performance-critical operation anyway, there's no real need to
be concerned about possible performance degradation from these changes.

The worst aspects of this bug were introduced in 9.0 --- 8.x will always
wait out a DELETE_IN_PROGRESS tuple.  But I think dropping index entries
on the strength of ii_BrokenHotChain is dangerous even without that, so
back-patch removal of that optimization to 8.3 and 8.4.

Per comment from Greg Stark, it's less clear that HOT chains don't conflict
with the index than it would be for a valid index.  So, let's preserve the
former behavior that indcheckxmin does get set when there are
potentially-broken HOT chains in this case.  This change does not cause any
pg_index update that wouldn't have happened anyway, so we're not
re-introducing the previous bug with pg_index updates, and surely the case
is not significant from a performance standpoint; so let's be as
conservative as possible.

The original coding assumed that such a case represents caller error, but
actually get_relation_info will omit generating an IndexOptInfo for any
index it thinks is unsafe to use.  Therefore, handle this case by returning
"true" to indicate that a seqscan-and-sort is the preferred way to
implement the CLUSTER operation.  New bug in 9.1, no backpatch needed.
Per bug #5985 from Daniel Grace.

It assumed that the lineno from the traceback always refers to the
PL/Python function.  If you created a PL/Python function that imports
some code, runs it, and that code raises an exception, PLy_traceback
would get utterly confused.

Now we look at the file name reported with the traceback and only
print the source line if it came from the PL/Python function.

Jan Urbański

Document why we do the missing new database check during the check
phase.

"People's Republic of China" locale on Windows was causing initdb to fail.

This fixes bug #5818 reported by yulei. On master, this makes the mapping
of "People's Republic of China" to just "China" obsolete. In 9.0 and 8.4,
just fix the escaping. Earlier versions didn't have locale names in bki
file.

If someone removes the 'postgres' database from the old cluster and the
new cluster has a 'postgres' database, the number of databases will not
match.  We actually could upgrade such a setup, but it would violate the
1-to-1 mapping of database counts, so we throw an error instead.

Previously they got an error during the upgrade, and not at the check
stage; PG 9.0.4 does the same.

Add C comment about why we throw an error if the pg_upgrade old/new
database counts don't match.

There can never be a need to push the indcheckxmin horizon forward, since
any HOT chains that are actually broken with respect to the index must
pre-date its original creation.  So we can just avoid changing pg_index
altogether during a REINDEX operation.

This offers a cleaner solution than my previous patch for the problem
found a few days ago that we mustn't try to update pg_index while we are
reindexing it.  System catalog indexes will always be created with
indcheckxmin = false during initdb, and with this modified code we should
never try to change their pg_index entries.  This avoids special-casing
system catalogs as the former patch did, and should provide a performance
benefit for many cases where REINDEX formerly caused an index to be
considered unusable for a short time.

Back-patch to 8.3 to cover all versions containing HOT.  Note that this
patch changes the API for index_build(), but I believe it is unlikely that
any add-on code is calling that directly.

This reverts commit 4b6106cc of 2011-04-15.
There's a better way to do it, which will follow shortly.

... for some value of "properly".  Instead of overriding REGRESS_OPTS,
set the variables ENCODING and NO_LOCALE, which is more expressive and
allows overriding by the user.  Fix vcregress.pl to handle that.

This is how build.pl treats it and how it's documented.

While "UTF8" is the correct name for this encoding, existing JDBC drivers
expect that if they send "UNICODE" it will read back the same way; they
fail with an opaque "Protocol error" complaint if not.  This will be fixed
in the 9.1 drivers, but until older drivers are no longer in use in the
wild, we'd better leave "UNICODE" alone.  Continue to canonicalize all
other inputs.  Per report from Steve Singer and subsequent discussion.