This change applies only to column oriented tables (CO).  It reads
block directory entries for only those columns that appear in project
list (e.g. select clause).

Closes #369

This commit represents an overhaul of the test suite relating to Veritas NetBackup
integration with the backup and restore utilities. It has the same goals and makes
similar changes to this suite that were made to the base backup and restore suite
in commit e8889fc5, and also focuses on removing
any scenarios that test functionality that isn't directly related to NetBackup and
is therefore sufficiently tested in the other suite, to improve the usability of
the suite and dramatically decrease total runtime (down from ~15 hours to ~5 hours
on Pivotal test machines).

This commit represents a major overhaul of the backup and restore test suite,
with the following main goals:
1) Cut down the time required to run the tests (previous runtime was ~3.5 hours
on Pivotal test machines, now it's down to around 1 hour 15 minutes).
2) Make each test scenario independent, so that the database state at the start
of a scenario does not depend on which scenarios were run before it and failures
in one scenario will not cause later scenarios to fail as a result.
3) Group commonly used steps, remove redundant scenarios and unnecessary steps
within scenarios, standardize common database and table names, and perform
similar cleanup, both to cut down on runtime and to improve readability and
comprehensibility of the suite.

This set of patches brings in RI operator selection fixes for generic types;
OR case handling in find_nonnullable_rels and some fixes to the recent
to_char() patch and some missing catalog entries for XML functions.

 Conflicts:
	doc/TODO
	doc/src/FAQ/TODO.html
	doc/src/sgml/config.sgml
	doc/src/sgml/docguide.sgml
	doc/src/sgml/ecpg.sgml
	doc/src/sgml/func.sgml
	doc/src/sgml/geqo.sgml
	doc/src/sgml/gin.sgml
	doc/src/sgml/installation.sgml
	doc/src/sgml/libpq.sgml
	doc/src/sgml/mvcc.sgml
	doc/src/sgml/plperl.sgml
	doc/src/sgml/runtime.sgml
	doc/src/sgml/sql.sgml
	src/backend/commands/tablecmds.c
	src/backend/executor/nodeResult.c
	src/backend/libpq/be-secure.c
	src/backend/optimizer/plan/initsplan.c
	src/backend/optimizer/plan/setrefs.c
	src/backend/postmaster/postmaster.c
	src/backend/utils/adt/formatting.c
	src/backend/utils/adt/numeric.c
	src/backend/utils/adt/xml.c
	src/backend/utils/misc/guc.c
	src/backend/utils/misc/postgresql.conf.sample
	src/backend/utils/misc/ps_status.c
	src/include/catalog/catversion.h
	src/include/catalog/pg_aggregate.h
	src/include/catalog/pg_proc.h
	src/include/postmaster/postmaster.h
	src/include/utils/builtins.h
	src/include/utils/xml.h
	src/interfaces/libpq/fe-secure.c
	src/test/regress/expected/create_aggregate.out
	src/test/regress/expected/timestamptz.out
	src/test/regress/sql/create_aggregate.sql

If an error is encountered while reading data from external tables,
FILEAM_HANDLE_ERROR sets pstate->cdbsreh->errmsg to edata->errmsg. It then
invokes FreeErrorData(). pstate->cdbsreh->errmsg is used to print out the error
message if segment reject limit is reached. This is incorrect since the memory
region it points to is no longer valid.

This patch fixes the issue by using pstrdup to copy the error message instead
of assigning the pointer.

This commit replaces a bad debug message with the real pexpect.pxssh
Exception messages.

Authors: Jimmy Yih and Phil Goodwin

We shouldn't error out if plpythonu already exists, it's required for
the test to run but it's not the thing we're testing here.

This merge contains a large update to the autovacuum code breaking it up
into a worker based process model. Since parts of the code had been cherry-
picked from later commits already, pulled in a set of upstream patches by
cherry-picking the complete commit since the code in them was partly in and
partly required to make the diff cleaner. Full commitmessages are copied in
below. There are no changes made to the autovacuum integration in GPDB in
this commit though, it remains turned off and marked as defunct.

The timestamp formatting changes were already merged since before, this
commit brings in lots of new testcases for the code though.

 Conflicts:
	doc/src/sgml/config.sgml
	doc/src/sgml/func.sgml
	doc/src/sgml/libpq.sgml
	doc/src/sgml/runtime.sgml
	src/backend/access/transam/varsup.c
	src/backend/bootstrap/bootstrap.c
	src/backend/commands/vacuum.c
	src/backend/libpq/be-secure.c
	src/backend/optimizer/README
	src/backend/optimizer/geqo/geqo_eval.c
	src/backend/optimizer/path/joinrels.c
	src/backend/optimizer/util/joininfo.c
	src/backend/postmaster/autovacuum.c
	src/backend/postmaster/pgstat.c
	src/backend/postmaster/postmaster.c
	src/backend/storage/ipc/ipci.c
	src/backend/storage/lmgr/proc.c
	src/backend/tcop/postgres.c
	src/backend/utils/adt/datetime.c
	src/backend/utils/adt/formatting.c
	src/backend/utils/adt/timestamp.c
	src/backend/utils/init/miscinit.c
	src/backend/utils/init/postinit.c
	src/backend/utils/misc/postgresql.conf.sample
	src/bin/initdb/initdb.c
	src/include/miscadmin.h
	src/include/postmaster/autovacuum.h
	src/include/postmaster/postmaster.h
	src/include/storage/lwlock.h
	src/include/storage/pmsignal.h
	src/include/storage/proc.h
	src/include/utils/datetime.h
	src/include/utils/timestamp.h
	src/interfaces/ecpg/pgtypeslib/dt.h
	src/interfaces/libpq/fe-secure.c
	src/test/regress/expected/timestamptz.out

  commit 8aaecaf8
  Author: Alvaro Herrera <alvherre@alvh.no-ip.org>
  Date:   Fri Mar 23 20:56:39 2007 +0000

    We no longer need to palloc the VacuumStmt node; keeping it on the stack is
    simpler.

  commit 6287eb7a
  Author: Alvaro Herrera <alvherre@alvh.no-ip.org>
  Date:   Fri Mar 23 21:23:13 2007 +0000

    Separate fetch of pg_autovacuum tuple into its own function.

  commit a0abe87f
  Author: Alvaro Herrera <alvherre@alvh.no-ip.org>
  Date:   Fri Mar 23 21:45:17 2007 +0000

    Separate the code to start a new worker into its own function.  The code is
    exactly the same, modulo whitespace.

  commit e019bbc9
  Author: Alvaro Herrera <alvherre@alvh.no-ip.org>
  Date:   Fri Mar 23 21:57:10 2007 +0000

    Set the node properly, per Tom.

  commit f1a596bd
  Author: Alvaro Herrera <alvherre@alvh.no-ip.org>
  Date:   Tue Mar 27 20:36:03 2007 +0000

    Cosmetic changes: rename some struct fields, and move the fetching of pgstat
    table entries to a separate routine.  Don't pass the pgstat database entry to
    do_autovacuum; rather, have it fetch it by itself.

  commit 7d4c9a57
  Author: Alvaro Herrera <alvherre@alvh.no-ip.org>
  Date:   Wed Mar 28 22:17:12 2007 +0000

    Add the "recheck" logic to autovacuum worker code.  The worker first builds
    its table list and then rechecks pgstat before vacuuming each table to
    verify that no one has vacuumed the table in the meantime.

    In the current autovacuum world this only means that a worker will not
    vacuum a table that a user has vacuumed manually after the worker started.
    When support for multiple autovacuum workers is introduced, this will reduce
    the probability of simultaneous workers on the same database doing redundant
    work.

  commit e2a186b0
  Author: Alvaro Herrera <alvherre@alvh.no-ip.org>
  Date:   Mon Apr 16 18:30:04 2007 +0000

    Add a multi-worker capability to autovacuum.  This allows multiple worker
    processes to be running simultaneously.  Also, now autovacuum processes do not
    count towards the max_connections limit; they are counted separately from
    regular processes, and are limited by the new GUC variable
    autovacuum_max_workers.

    The launcher now has intelligence to launch workers on each database every
    autovacuum_naptime seconds, limited only on the max amount of worker slots
    available.

    Also, the global worker I/O utilization is limited by the vacuum cost-based
    delay feature.  Workers are "balanced" so that the total I/O consumption does
    not exceed the established limit.  This part of the patch was contributed by
    ITAGAKI Takahiro.

    Per discussion.

  commit a115bfe3
  Author: Alvaro Herrera <alvherre@alvh.no-ip.org>
  Date:   Wed May 2 15:47:14 2007 +0000

    Fix failure to check for INVALID worker entry in the new autovacuum code, which
    could happen when a worker took to long to start and was thus "aborted" by the
    launcher.  Noticed by lionfish buildfarm member.

This reverts commit a19e0e7e. I had misread
the code, thinking that GP_HOSTADDRESS is a constant throughout the script.
It's not; it's set by the SET_VAR function, so the lookups were all looking
up different addresses. I did not notice that while testing on a single
host.

I still think the "sleep 2" is unnecessary, but I'm now afraid of touching
this because I don't have a multi-host setup at hand to test with. So revert
the whole commit .

Spotted by Jimmy Yih.

TimeOffset is typdefinitioned in the relevant included header file
and should not be redefined locally. Also comment out the char
buffers which aren't used yet (due to GPDB not yet having merged the
functionality the code using them depend on). The TimeOffset typedef
broke the buildfarm.

strlcpy() isn't used in xlogdump and including it in the build cause
failures on platforms which supply strlcpy() in base. Remove source
and disconnect from Makefile.

This increases the diff in our version from upstream, however since
upstream is abandoned in favor of pg_xlogdump which was added to base
this is an Ok tradeoff to not have dead code in the tree.

xlogdump was temporarily disconnected during the merging of the
improved WAL handling during page splits. Reconnect again since
the code has stabilized.

We have a list of window values, one per level that the window functions of each level processes. We extract these window values from tuple store. To do the extraction, we use a temporary buffer, called serial_array, to serialize and deserialize the tuples. During this process, we obtain winvalues (the list of input values for a level). If the data type of the winvalues for a level is byref, then it ends up holding a pointer to the serial_array. However, we have only one serial_array for the entire window operator. Therefore, if we have more than one level with byref data types, we may end up overwriting the serial_array when we process another level, corrupting the earlier level’s byref datum pointers. To fix this, we now have one serial_array per level.
Signed-off-by: NFoyzur Rahman <foyzur@gmail.com>

It doesn't exist in the upstream.

These should reduce unnecessary conflicts while merging upcoming 8.3 patches.

Remove unnecessary #includes, add #includes that are actually needed by
some headers.

We always passed CurrentMemoryContext for them, so might as well remove
the parameter, making the code more readable, and always allocate the
return values in CurrentMemoryContext.

These are always compiled directly with C compiler. There's no prospect
of including these in a C++ file.

The correct way is that:

* Every .c file does '#include "postgres.h"', before anything else. After
  that, you can include any system header files (e.g. unistd.h). And after
  that, any other PostgreSQL/GPDB header files.

* Header files should not contain '#include "postgres.h"', as that's done
  in the .c files.

In cdbcat.h, include only the header files that are actually needed for
the single function prototype in that file. And don't include cdbcat.h
unnecessarily. A couple of .c files were including cdbcat.h to get
GpPolicy, but that's actually defined in catalog/gp_policy.h, so #include
that directly instead where needed.

"uint" is not a standard C type, so it might not be available on all
platforms. Indeed, we had a typedef for WIN32 for that. But there's no reason
to use "uint", might as well just use the C standard "unsigned int", or the
PostgreSQL-specific uint32. Makes the intention more clear too, IMHO.

None of the functions declared in cdblogsync exist, nor are used for
anything. cdbanalyze.h didn't contain any prototypes, it just #included
two other headers, so better to #include those directly where needed.

Closes #299.

All patches in this set were either doc commits or already merged except
the moving of the fsync macros from xlog.c to xlogdefs.h. GPDB had already
moved parts to xlog.h so revert these changes and sync with how the macros
are defined upstream.

 Conflicts:
	doc/TODO
	doc/src/FAQ/TODO.html
	doc/src/sgml/func.sgml
	doc/src/sgml/indices.sgml
	src/backend/access/transam/xlog.c
	src/backend/executor/execAmi.c
	src/backend/executor/nodeResult.c
	src/backend/utils/adt/formatting.c
	src/include/executor/nodeResult.h
	src/test/regress/expected/xml.out
	src/test/regress/expected/xml_1.out
	src/test/regress/sql/xml.sql
	src/tools/fsync/test_fsync.c
	src/tools/msvc/Solution.pm
	src/tools/msvc/config.pl

Finding the tle is a case for breaking the loop early but not finding
isn't a failure case. This elog() breaks the following query in the
gp_dqa testcase when found is properly initiated to false before the
loop.

select count(distinct c), count(distinct dt), i from dqa_t1 group by i;

When not initiated at all the variable was likely to not be false
from randomness and the test didnt fail.

Without altering functionality, fix a set of compiler warnings in
gpopt: Properly return in non-void function, remove non-function
invocation of variable and use the right formatter for ULLONG when
printing.

This commit merges a large rewrite of the foreign-key mechanism to properly
enforce the equality check. Instead of selecting an operator named "=" the
equality operator will now be selected from the opfamily of the unique
index that the foreign key depends on. Given how GPDB handles foreign keys
and referential integrity this isn't introducing changes but rather brings
in a good scaffolding for future work.

Other patches are minor spelling fixes on GUCs and some doc changes.

 Conflicts:
	doc/TODO
	doc/src/FAQ/TODO.html
	doc/src/sgml/catalogs.sgml
	doc/src/sgml/trigger.sgml
	src/backend/catalog/aclchk.c
	src/backend/catalog/dependency.c
	src/backend/catalog/index.c
	src/backend/catalog/namespace.c
	src/backend/catalog/pg_constraint.c
	src/backend/catalog/pg_conversion.c
	src/backend/commands/conversioncmds.c
	src/backend/commands/tablecmds.c
	src/backend/commands/trigger.c
	src/backend/commands/typecmds.c
	src/backend/tcop/utility.c
	src/backend/utils/adt/ri_triggers.c
	src/backend/utils/adt/ruleutils.c
	src/backend/utils/cache/syscache.c
	src/bin/pg_dump/pg_dump.c
	src/bin/psql/describe.c
	src/include/catalog/catversion.h
	src/include/catalog/pg_constraint.h
	src/include/utils/rel.h
	src/include/utils/syscache.h
	src/test/regress/expected/alter_table.out
	src/test/regress/expected/foreign_key.out
	src/test/regress/sql/alter_table.sql

A typo caused the planner to not always recognize that the distribution
keys of an outer join match. That would not lead to incorrect results, but
it did lead to unnecessary Motion nodes in some cases.

Fixes issue #340. Thanks @liruto!

For AO/CO tables due to functional nature, new EOF value to be updated
must always be greater than current value. Currently, this check is
performed on persistent table update but it happens as part of Commit-Prepared.
So its super too late to take any action, even if we detect it can only PANIC
the system and already resulted in data corruption for reads as aoseg has been
updated and transaction commited. Hence moving the check while updating the
aoseg table, when such case is detected now transaction can be aborted without
impacting the system or already loaded data in table.

* Remove an unnecessary sleep 2
* Perform the lookup of master hostname only once