While fetching the pg_class or pg_type tuple using index, perform sanity
check to make sure tuple intended to read, is the tuple index is
pointing too. This is just sanity check to make sure index is not broken
and not returning some incorrect tuple to contain the damage.

This commit makes sure while accessing gp_relation_node through its
index, sanity check is always performed to verify the tuple being
operated on is the intended tuple, else for any reason index is broken
and provide bad tuple fail the operation instead of causing damage.
For some scenarios like delete gp_relation_code tuple case it adds extra tuple
deform call which was not done earlier but doesn't seem heavy enough to
be performed on ddl operation.

In cdbcat.h, include only the header files that are actually needed for
the single function prototype in that file. And don't include cdbcat.h
unnecessarily. A couple of .c files were including cdbcat.h to get
GpPolicy, but that's actually defined in catalog/gp_policy.h, so #include
that directly instead where needed.

Moving the installation of gp_toolkit.sql into initdb, in commit f8910c3c,
broke all the functions that are supposed to execute on all nodes, like
gp_toolkit.__gp_localid. After that change, gp_toolkit.sql was executed
in utility mode, and the gp_distribution_policy entries for those functions
were not created as a result.

To fix, change the code so that gp_distribution_policy entries are never
never created, or consulted, for EXECUTE-type external tables. They have
more fine-grained information in pg_exttable.location field anyway, so rely
on that instead. With this change, there is no difference in whether an
EXECUTE-type external table is created in utility mode or not. We would
still have similar problems if gp_toolkit contained other kinds of
external tables, but it doesn't.

This removes the isMasterOnly() function and changes all its callers to
call GpPolicyFetch() directly instead. Some places used GpPolicyFetch()
directly to check if a table is distributed, so this just makes that the
canonical way to do it. The check for system schemas that used to be in
isMasterOnly() are no longer performed, but they should've unnecessary in
the first place. System tables don't have gp_distribution_policy entries,
so they'll be treated as master-only even without that check.

The current plan is to use something like pg_upgrade for future in-place
upgrades. The gpupgrade mechanism will not scale to the kind of drastic
catalog and other data directory layout changes that are coming as we
merge with later PostgreSQL releases.

Kept gpupgrademirror for now. Need to check if there's some logic that's
worth saving, for a possible pg_upgrade based solution later.

created it.

Simon Riggs

per-column options for btree indexes.  The planner's support for this is still
pretty rudimentary; it does not yet know how to plan mergejoins with
nondefault ordering options.  The documentation is pretty rudimentary, too.
I'll work on improving that stuff later.

Note incompatible change from prior behavior: ORDER BY ... USING will now be
rejected if the operator is not a less-than or greater-than member of some
btree opclass.  This prevents less-than-sane behavior if an operator that
doesn't actually define a proper sort ordering is selected.

back-stamped for this.

pg_opclass during LookupOpclassInfo(), I'd turned pg_opclass_oid_index
into a critical system index.  However the problem could only manifest
during a backend's first attempt to load opclass data, and then only
if it had successfully loaded pg_internal.init and subsequently received
a relcache flush; which made it impossible to reproduce in sequential
tests and darn hard even in parallel tests.  Memo to self: when
exercising cache flush scenarios, must disable LookupOpclassInfo's
internal cache too.

cases.  Operator classes now exist within "operator families".  While most
families are equivalent to a single class, related classes can be grouped
into one family to represent the fact that they are semantically compatible.
Cross-type operators are now naturally adjunct parts of a family, without
having to wedge them into a particular opclass as we had done originally.

This commit restructures the catalogs and cleans up enough of the fallout so
that everything still works at least as well as before, but most of the work
needed to actually improve the planner's behavior will come later.  Also,
there are not yet CREATE/DROP/ALTER OPERATOR FAMILY commands; the only way
to create a new family right now is to allow CREATE OPERATOR CLASS to make
one by default.  I owe some more documentation work, too.  But that can all
be done in smaller pieces once this infrastructure is in place.

stale relcache init files (pg_internal.init), and there is no mechanism for
updating them during WAL replay.  Easiest solution is just to delete the init
files at conclusion of startup, and let the first backend started in each
database take care of rebuilding the init file.  Simon Riggs and Tom Lane.

Back-patched to 8.1.  Arguably this should be fixed in 8.0 too, but it would
require significantly more code since 8.0 has no handy startup-time scan of
pg_database to piggyback on.  Manual solution of the problem is possible
in 8.0 (just delete the pg_internal.init files before starting WAL replay),
so that may be a sufficient answer.

can create or modify rules for the table.  Do setRuleCheckAsUser() while
loading rules into the relcache, rather than when defining a rule.  This
ensures that permission checks for tables referenced in a rule are done with
respect to the current owner of the rule's table, whereas formerly ALTER TABLE
OWNER would fail to update the permission checking for associated rules.
Removal of separate RULE privilege is needed to prevent various scenarios
in which a grantee of RULE privilege could effectively have any privilege
of the table owner.  For backwards compatibility, GRANT/REVOKE RULE is still
accepted, but it doesn't do anything.  Per discussion here:
http://archives.postgresql.org/pgsql-hackers/2006-04/msg01138.php

(table or index) before trying to open its relcache entry.  This fixes
race conditions in which someone else commits a change to the relation's
catalog entries while we are in process of doing relcache load.  Problems
of that ilk have been reported sporadically for years, but it was not
really practical to fix until recently --- for instance, the recent
addition of WAL-log support for in-place updates helped.

Along the way, remove pg_am.amconcurrent: all AMs are now expected to support
concurrent update.

Strip unused include files out unused include files, and add needed
includes to C files.

The next step is to remove unused include files in C files.

discussion (including making def_arg allow reserved words), add missed
opt_definition for UNIQUE case.  Put the reloptions support code in a less
random place (I chose to make a new file access/common/reloptions.c).
Eliminate header inclusion creep.  Make the index options functions safely
user-callable (seems like client apps might like to be able to test validity
of options before trying to make an index).  Reduce overhead for normal case
with no options by allowing rd_options to be NULL.  Fix some unmaintainably
klugy code, including getting rid of Natts_pg_class_fixed at long last.
Some stylistic cleanup too, and pay attention to keeping comments in sync
with code.

Documentation still needs work, though I did fix the omissions in
catalogs.sgml and indexam.sgml.

ITAGAKI Takahiro

by creating a reference-count mechanism, similar to what we did a long time
ago for catcache entries.  The back branches have an ugly solution involving
lots of extra copies, but this way is more efficient.  Reference counting is
only applied to tupdescs that are actually in caches --- there seems no need
to use it for tupdescs that are generated in the executor, since they'll go
away during plan shutdown by virtue of being in the per-query memory context.
Neil Conway and Tom Lane

needNewCacheFile flag anymore, it can just be local in RelationCacheInitializePhase2.

it's not necessary to have three separate calls anymore.  This patch also
fixes things so we don't try to read pg_internal.init until after we've
obtained lock on the target database; which was fairly harmless, but it's
certainly cleaner this way.

thereby saving a visit to the metapage in most index searches/updates.
This wouldn't actually save any I/O (since in the old regime the metapage
generally stayed in cache anyway), but it does provide a useful decrease
in bufmgr traffic in high-contention scenarios.  Per my recent proposal.

index's support-function cache (in index_getprocinfo).  Since none of that
data can change for an index that's in active use, it seems sufficient to
treat all open indexes the same way we were treating "nailed" system indexes
--- that is, just re-read the pg_class row and leave the rest of the relcache
entry strictly alone.  The pg_class re-read might not be strictly necessary
either, but since the reltablespace and relfilenode can change in normal
operation it seems safest to do it.  (We don't support changing any of the
other info about an index at all, at the moment.)

Back-patch as far as 8.0.  It might be possible to adapt the patch to 7.4,
but it would take more work than I care to expend for such a low-probability
problem.  7.3 is out of luck for sure.

This is utterly insignificant in normal operation, but it becomes a
problem during cache inval stress testing.  The original coding in fact
had no leak --- the 8.0 List rewrite created the issue.  I wonder whether
list_concat should pfree the discarded header?

and nail a couple more system indexes into cache.  This doesn't make
any difference in normal system operation, but when forcing constant
cache resets it's difficult to get through the rules regression test
without these changes.

http://archives.postgresql.org/pgsql-hackers/2006-01/msg00151.php for the
complete plan.

the data defining the semantics of a lock method (ie, conflict resolution
table and ancillary data, which is all constant) and the hash tables
storing the current state.  The only thing we give up by this is the
ability to use separate hashtables for different lock methods, but there
is no need for that anyway.  Put some extra fields into the LockMethod
definition structs to clean up some other uglinesses, like hard-wired
tests for DEFAULT_LOCKMETHOD and USER_LOCKMETHOD.  This commit doesn't
do anything about the performance issues we were discussing, but it clears
away some of the underbrush that's in the way of fixing that.

comment line where output as too long, and update typedefs for /lib
directory.  Also fix case where identifiers were used as variable names
in the backend, but as typedefs in ecpg (favor the backend for
indenting).

Backpatch to 8.1.X.

the convenience of tuptoaster.c and is no longer needed, so may as well
get rid of some small amount of overhead.

the parent table, even if the command that creates them is executed by
someone else (such as a superuser or a member of the owning role).
Per gripe from Michael Fuhr.

whenever we generate a new OID.  This prevents occasional duplicate-OID
errors that can otherwise occur once the OID counter has wrapped around.
Duplicate relfilenode values are also checked for when creating new
physical files.  Per my recent proposal.

ResourceOwner mechanism already released all reference counts for the
cache entries; therefore, we do not need to scan the catcache or relcache
at transaction end, unless we want to do it as a debugging crosscheck.
Do the crosscheck only in Assert mode.  This is the same logic we had
previously installed in AtEOXact_Buffers to avoid overhead with large
numbers of shared buffers.  I thought it'd be a good idea to do it here
too, in view of Kari Lavikka's recent report showing a real-world case
where AtEOXact_CatCache is taking a significant fraction of runtime.

spotted by Qingqing Zhou.  The HASH_ENTER action now automatically
fails with elog(ERROR) on out-of-memory --- which incidentally lets
us eliminate duplicate error checks in quite a bunch of places.  If
you really need the old return-NULL-on-out-of-memory behavior, you
can ask for HASH_ENTER_NULL.  But there is now an Assert in that path
checking that you aren't hoping to get that behavior in a palloc-based
hash table.
Along the way, remove the old HASH_FIND_SAVE/HASH_REMOVE_SAVED actions,
which were not being used anywhere anymore, and were surely too ugly
and unsafe to want to see revived again.

routines in the index's relcache entry, instead of doing a fresh fmgr_info
on every index access.  We were already doing this for the index's opclass
support functions; not sure why we didn't think to do it for the AM
functions too.  This supersedes the former method of caching (only)
amgettuple in indexscan scan descriptors; it's an improvement because the
function lookup can be amortized across multiple statements instead of
being repeated for each statement.  Even though lookup for builtin
functions is pretty cheap, this seems to drop a percent or two off some
simple benchmarks.

memset() or MemSet() to a char *. For one, memset()'s first argument is
a void *, and further void * can be implicitly coerced to/from any other
pointer type.