Due to a option index error, exclude-tests, ignore-plans, prehook and
print-failure-diffs are not parsed from command line correctly.

If "./pg_regress --init-file init_file uao_ddl/compresstype_row" is
executed, it fails. As extra argument `--ao-dir=uao` needs to be
specified to pg_regress, to convey, to convert these .source files to
row and column .sql/.out files and not use regular standard logic on
them. This approach has following shortfalls:

- additional developer overhead to remember to add that option
- without that option .sql/.out files are still generated but are not
  usable / incorrect
- row and column test files directories will always need same prefix
- the check for ao-dir prefix was checked for each and every file in
  input/output directories, which is unnecessary

To improve the situation, modifying the logic in pg_regress. No more
need extra argument to pg_regress, instead, presence of special file
named "GENERATE_ROW_AND_COLUMN_FILES" conveys to pg_regress to apply
special conversion rule on that directory. This way always correct
conversion rule is applied and developer don't need to specify any
extra option.

In GPDB, gpstringsubs.pl is used to replace tokens in .source test
files when creating .sql/.out files for pg_regress. pg_regress itself
also performs the substitutions. The order is pg_regress performs
substitutions and after that remaining ones are performed by
gpstringsubs.pl if required.

The logic to check if there are still tokens remaining to be replaced
after pg_regress has performed replacement was based on checking if
'@' character is present in file or not. If '@' is present in file
then gpstringsubs.pl is invoked for that file. gpstringsubs.pl is
super slow and takes long time. In many .source files keywords exist
like `@Description` or `@db_name` (specially for UAO related tests),
which don't need any substitution. But due to presence of these '@'
charater keywords, gpstringsubs.pl was invoked for such files
unnecessarily, causing the slow down.

So, modifying the check to invoke gpstringsubs.pl only on presence of
"@gp" keyword instead. So, now any token replacements handled by
gpstringsubs.pl should start with "@gp" in the name. Hence, renamed
`@syslocale@` to "@gp_syslocale@".

Also, moved the logic to perform replacement for `@hostname@` and
`@gpcurusername@` to pg_regress itself, as it's much faster and easier
to do the same in C.

Given the whole testsuite takes very long time in GPD. Need to iterate
on single test routinely. These substitutions are invoked on every run
of pg_regress (mainly isolation2 also), hence helps to reduce the
iteration time for running single test in test suite during
development, along with helping cut down pg_regress file conversion
time in CI. For regress directory cuts down time from 8 secs and
isolation2 from 25 secs to few msecs.

In long run we should complete move all the logic in gpstringsubs.pl
to pg_regress and kill gpstringsubs.pl.

This function was modified in GPDB just for pg_regress. pg_regress
needs to locate gpdiff.pl and gpstringsubs.pl scripts where it uses
find_other_exec(). Since not correct version was passed to it, this
modification was required to find_other_exec(). These scripts have
code to correctly print the version information on "-V". So, regular
unmodified version of find_other_exec() works if the version is passed
correctly from pg_regress.
Reviewed-by: NAsim R P <apraveen@pivotal.io>
Reviewed-by: NDaniel Gustafsson <dgustafsson@pivotal.io>
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

This allows a user to specify regress options to consistently make
isolation2 tests for pg_basebackup_with_tablespaces pass.

Currently, the tests fail if the user's source directory creates
a tablespace location directory path that is longer than the 100
character limit for pg_basebackup to add a tablespace location
directory to the backup.

When given --print-failure-diffs, pg_regress prints the contents of
regression.diffs after a failure.

Special code was written for resgroup directory in input and
output. Instead convert it to generic code to recurse if any directory
encountered in input/output directory. This hurdle was faced when
attempting to add test using template framework for walreplication,
where we wished add test in "input/segwalrep" directory. Change in
this commit now enables to add directories easily to input and output.

Since no need exist now for special resgroup directory argument,
removing the same. Given the number of files in resgroup directory is
very less and not expected to grow super large, its okay to generate
the .sql and .out files always along with generating other files.

GCC 8.2 started giving this warning:

pg_regress.c: In function ‘generate_uao_sourcefiles’:
pg_regress.c:609:34: warning: argument to ‘sizeof’ in ‘strncpy’ call is the same expression as the source; did you mean to use the size of the destination? [-Wsizeof-pointer-memaccess]
    strncpy(line_row, line, sizeof(line));
                                  ^

In this case, 'line' and 'line_row' actually have the same size, 1024
bytes, so this is just pedantry.

While we're at it, switch to using strlcpy(), because this would not work
correctly if the buffer isn't NULL-terminated. The buffer is supposed to
"large enough", so that that never happens anyway, but might as well be
safe here.

This is a preparation of an upcoming random ICG pipeline job.  In that
pipeline job we will have tables created on different segments, so far
the only way to hack this behavior is by using the gp_debug_numsegments
extension.  We want to load and execute that extension for all the tests
without modifying them directly, so a hook mechanism is needed.

Added a pg_regress argument --prehook to set a hook.  A hook script
should be put under src/test/regress/{sql,input}/hooks/ directory,
depending on whether it need pg_regress to substitute the @@ tokens.  At
most one hook can be specied.  It can be set via make command like this:

    # suppose there is src/test/regress/sql/hooks/hookname.sql
    make installcheck EXTRA_REGRESS_OPTS=--prehook=hookname

This is a preparation of an upcoming random ICG pipeline job.  In that
pipeline job we will have tables created on different segments, so the
plans might be different expectation.  In atmsort.pm there is already an
argument -gpd_ignore_plans to ignore the plan diffs, but there was no
equivalence in pg_regress.

Added a pg_regress argument --ignore-plans to ignore plan diffs.  It can
be enabled via make command like this:

    make installcheck EXTRA_REGRESS_OPTS=--ignore-plans

In GPDB, platform_expectfile is used for determining
ORCA/planner/resgroup expect files, wheras in upstream that is not the
case and it is based on the underlying platform. Thus, it is unnecessary
and confusing to compare against default answer file even when
platform_expect file exists. It gets confusing because the below block
chooses the best expect file based on the number of lines in diff file.
Co-authored-by: NEkta Khanna <ekhanna@pivotal.io>

Fix a small number of places that were testing the result of snprintf()
but doing so incorrectly.  The right test for buffer overrun, per C99,
is "result >= bufsize" not "result > bufsize".  Some places were also
checking for failure with "result == -1", but the standard only says
that a negative value is delivered on failure.

(Note that this only makes these places correct if snprintf() delivers
C99-compliant results.  But at least now these places are consistent
with all the other places where we assume that.)

Also, make psql_start_test() and isolation_start_test() check for
buffer overrun while constructing their shell commands.  There seems
like a higher risk of overrun, with more severe consequences, here
than there is for the individual file paths that are made elsewhere
in the same functions, so this seemed like a worthwhile change.

Also fix guc.c's do_serialize() to initialize errno = 0 before
calling vsnprintf.  In principle, this should be unnecessary because
vsnprintf should have set errno if it returns a failure indication ...
but the other two places this coding pattern is cribbed from don't
assume that, so let's be consistent.

These errors are all very old, so back-patch as appropriate.  I think
that only the shell command overrun cases are even theoretically
reachable in practice, but there's not much point in erroneous error
checks.

Discussion: https://postgr.es/m/17245.1534289329@sss.pgh.pa.us

This reverts commit 4750e1b6.

This is the final batch of commits from PostgreSQL 9.2 development,
up to the point where the REL9_2_STABLE branch was created, and 9.3
development started on the PostgreSQL master branch.

Notable upstream changes:

* Index-only scan was included in the batch of upstream commits. It
  allows queries to retrieve data only from indexes, avoiding heap access.

* Group commit was added to work effectively under heavy load. Previously,
  batching of commits became ineffective as the write workload increased,
  because of internal lock contention.

* A new fast-path lock mechanism was added to reduce the overhead of
  taking and releasing certain types of locks which are taken and released
  very frequently but rarely conflict.

* The new "parameterized path" mechanism was added. It allows inner index
  scans to use values from relations that are more than one join level up
  from the scan. This can greatly improve performance in situations where
  semantic restrictions (such as outer joins) limit the allowed join orderings.

* SP-GiST (Space-Partitioned GiST) index access method was added to support
  unbalanced partitioned search structures. For suitable problems, SP-GiST can
  be faster than GiST in both index build time and search time.

* Checkpoints now are performed by a dedicated background process. Formerly
  the background writer did both dirty-page writing and checkpointing. Separating
  this into two processes allows each goal to be accomplished more predictably.

* Custom plan was supported for specific parameter values even when using
  prepared statements.

* API for FDW was improved to provide multiple access "paths" for their tables,
  allowing more flexibility in join planning.

* Security_barrier option was added for views to prevents optimizations that
  might allow view-protected data to be exposed to users.

* Range data type was added to store a lower and upper bound belonging to its
  base data type.

* CTAS (CREATE TABLE AS/SELECT INTO) is now treated as utility statement. The
  SELECT query is planned during the execution of the utility. To conform to
  this change, GPDB executes the utility statement only on QD and dispatches
  the plan of the SELECT query to QEs.
Co-authored-by: NAdam Lee <ali@pivotal.io>
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>
Co-authored-by: NAshwin Agrawal <aagrawal@pivotal.io>
Co-authored-by: NAsim R P <apraveen@pivotal.io>
Co-authored-by: NDaniel Gustafsson <dgustafsson@pivotal.io>
Co-authored-by: NGang Xiong <gxiong@pivotal.io>
Co-authored-by: NHaozhou Wang <hawang@pivotal.io>
Co-authored-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Co-authored-by: NJesse Zhang <sbjesse@gmail.com>
Co-authored-by: NJinbao Chen <jinchen@pivotal.io>
Co-authored-by: NJoao Pereira <jdealmeidapereira@pivotal.io>
Co-authored-by: NMelanie Plageman <mplageman@pivotal.io>
Co-authored-by: NPaul Guo <paulguo@gmail.com>
Co-authored-by: NRichard Guo <guofenglinux@gmail.com>
Co-authored-by: NShujie Zhang <shzhang@pivotal.io>
Co-authored-by: NTaylor Vesely <tvesely@pivotal.io>
Co-authored-by: NZhenghua Lyu <zlv@pivotal.io>

While looking at a recent buildfarm failure in the ecpg tests, I wondered
why the pg_regress output claimed the stderr part of the test failed, when
the regression diffs were clearly for the stdout part.  Looking into it,
the reason is that pg_regress.c's logic for iterating over three parallel
lists is wrong, and has been wrong since it was written: it advances the
"tag" pointer at a different place in the loop than the other two pointers.
Fix that.

If we avoid calling pg_regress with an empty --multibyte we
can skip GPDB specific handling of the encoding parameter in
pg_regress and instead follow upstream. While the local diffs
to pg_regress weren't wrong per se, avoiding merge conflicts
is of more interest.

Also revert the commented out MULTIBYTE in the makefile to
match the upstream value. A follow-up commit will fix this
as well.

This patch removes codegen wholesale from Greenplum.

In addition to reverting the commits involving codegen, we also removed
miscellaneous references to the feature and GUC.

The following commits from 5.0.0 were reverted (topologically ordered):

f38c9064 Support avg aggregate function in codegen
87dcae4c Capture and print error messages from llvm::verifyFunction
65137540 libgpcodegen assert() overrides use GPDB Assert()
81d378b4 GenerateExecVariableList calls regular slot_getattr when the code generation of the latter fails
05a28211 Update Google Test source path used by codegen
22a35fcc Call ereport when code generating float operators
79517271 Support overflow checks for doubles.
b5373a1e Fix codegen unittest link problem
7a1a98c9 Print filename and lineno in codegened CreateElog
58eda293 Fix wrong virtual tuple check in codegened slot_getattr
bc6faa08 Set llvm_isNull_ptr to false when the result of codegened expression evaluation is not null
8bbbd63f Enhance codegened advance_aggregates with support for null attributes
e1fd6072 Abort code generation of expression evaluation trees with unsupported ExprState types
509460ee Support null attributes in codegen expression evaluation framework
739d978d Move enrollment of codegened ExecVariableList to ExecInitNode
c05528d1 Fix CHECK_SYMBOL_DEFINED macro in CMakeLists.txt
12cfd7bd Support offset calculation when there is null in the tuple and codegen is enabled
40a2631e Use slot_getattr wrapper function for regular version.(#1194)
613e9fbb Revert "Fix codegen issue by moving slot_getattr to heaptuple.c similar to"
ee03f799 Fix cpplint error on advance aggregate
2a65b0aa Fix slot function nullptr issue in expr eval.
3107fc0e Fix for !(expr1 & expr2) and hasnull logic in slot_getatrr codegen.
c940c7f6 Fix codegen issue by moving slot_getattr to heaptuple.c similar to Postgres.
c8125736 Introduce guc to enable/disable all generator.
a4f39507 Ensure that codegen unittests pass with GCC 6.2.0 (#1177)
682d0b28 Allow overriding assert() functionality in libgpcodegen in DEBUG mode
3209258a Organize codegen source files and unit tests
d2ba88a9 Fix codegen unittests using DatumCastGenerator
020b71a5 Generate code for COUNT aggregate function
0eeec886 Fixing codegen related bugs in InitializeSupportedFunction
41055352 Rewrite PGGenericFuncGenerator to support variable number of types
e5318b6b Add AdvanceAggregatesCodegenEnroll mock function
87521715 Codegen advance_aggregates for SUM transition function
2043d95b Use string messages in codegened slot_getattr fallback block
f160fa5a Add new GUC to control Codegen Optimization level.
697ffc1a Fix cpplint errors
c5e3aed4 Support Virtual Tuples and Memtuples in SlotGetAttrCodegen
5996aaa7 Keep a reference to the CodegenManager in code generators
6833b3c2 Remove unused header and just include what you use in codegen
ab1eda87 Allow setting codegen guc to ON only if code generation is supported by the build
dcd40712 Use PGFuncGeneratorInfo to codegen pg functions
83869d1c Replace dynamic_cast with dyn_cast for llvm objects
23007017 Decide what to generate for ExecEvalExpr based on PlanState
387d8ce8 Add EXPLAIN CODEGEN to print generated IR to the client
c4a9bd27 Introduce Datum to cpp cast, cpp type to Datum cast  and normal cast.(#944)
66158dfd Record external function names for useful debugging
adab9120 Support variable length attributes in SlotGetAttrCodegen.
335e09aa Proclaim that the codegen'ed version of generated functions are called in debug build
50fd9140 Fix cpplint errors
88a04412 Use ExprTreeGeneratorInfo for expression tree generation.
3b4af3bb Split code generation of ExecVariableList from slot_getattr
8cd9ed9f Support <= operator for date/timestamp data types and some minor refactors.
e4dccf46 Implement InlineFunction to force inline at call site
71170942 Mock postmaster.o for codegen_framework_unittest.t
09f00581 Codegen expressions that contain plus, minus and mul operators on float8 data types
d7fb2f6d Fix codegen unittests on Linux and various compiler warnings while building codegen.
45f2aa96 Fix test and update coding style This closes #874
1b26fbfc Enrolled targetlist for scan and aggregate
ebd1d014 Enhance codegen framework to support arbitrary expr tree
ec626ce6 Generate and enroll naive ExecEvalExpr in Scan's quals
1186001e Revert "Create naive code-generated version of ExecQual"
6f928a65 Replace RegisterExternalFunction with GetOrRegisterExternalFunction using an unordered_map in codegen_utils
6ae0085b Move ElogWrapper to GpCodegenUtils.
d3f80b45 Add verifyfunction for generated llvm function.
7bcf094a Fix codegen compiler error with 8.3 merge
aae0ad3d Create naive code-generated version of ExecQual
dce266ad Minor code quality fixes in src/backend/codegen
d281f340 Support null attributes in code generated ExecVariableList
82fd418e Address a number of cpplint errors in codegen_utils_unittest.cc
887aa48d Add check for CodegenUtils::GetType for bool arrays
bb9b92c6 Enhance Base Codegen to do clean up when generation fails
b9ef5e3f Fix build error for Annotated types
a5cfefd9 Add support for array types in codegen_utils.
2b883384 Fix static_assert call
7b75d9ea This commit  generates code for code path: ExecVariableList > slot_getattr > _slot_getsomeattrs > slot_deform_tuple. This code path is executed during scan in simple select queries that do not have a where clause (e.g., select bar from foo;).
6d0a06e8 Fix CodeGen typos and CodeGeneratorManagerCreate function signature in gpcodegen_mock.c
4916a606 Add support for registering vararg external functions with codegen utils.
ae4a7754 Integrate codegen framework and make simple external call to slot deform tuple. This closes #649
ee5fb851 Renaming code_generator to codegen_utils and CodeGenerator to CodegenUtils. This closes #648
88e9baba Adding GPDB code generation utils
Signed-off-by: NJesse Zhang <sbjesse@gmail.com>
Signed-off-by: NMelanie Plageman <mplageman@pivotal.io>
Signed-off-by: NSambitesh Dash <sdash@pivotal.io>

When running pg_regress, sometimes you may want to exclude certain
tests from your schedule file from running. Before, you would have to
modify the schedule file to comment/ignore out the unwanted test. Now
we can do it from command line with --exclude-tests option that is
space and comma delimited.

Example:
./pg_regress --exclude-tests="test1 test2,test3 ... testN"

Authors: Abhijit Subramanya and Jimmy Yih

* resgroup: increase max slots for isolation tests.
* ICW: ignore resgroup related warnings.
* ICW: try to load resgroup variant of answers when resgroup enabled.
* ICW: provide resgroup variant of answers.
* ICW: check whether resqueue is enabled in UDF.
* ICR: substitude usrname in gpconfig output.
* ICR: explicitly set max_connections.
* isolation2: increase resgroup concurrency for max_concurrency tests.

resource group tests are messing up isolation2 directory, this commits
move them all to the same directory to be tidy and organized, it also
handle the sub directory in input/output directory for resgroup.

1. Detect cgroup mount point in test code.
2. Fix bug when buflen is 0.
3. Check cgroup status on master in gpconfig.
4. Fix coverity warnings.

The gpmapreduce application is an optional install included via
the --enable-mapreduce configure option. The tests were however
still in src/test/regress and unconditionally included in the
ICW schedule, thus causing test failures when mapreduce wasn't
configured.

Move all gpmapreduce tests to co-locate them with the mapreduce
code and only test when configured.

Also, add a dependency on Perl for gpmapreduce in autoconf since
it's a required component.

`outputdir` here is where to place the converted files, not the
directory named `output`. PostgreSQL places them into `outputdir`
rightly.

    commit 64cdbbc4
    Author: Peter Eisentraut <peter_e@gmx.net>
    Date:   Sat Feb 14 21:33:41 2015 -0500

        pg_regress: Write processed input/*.source into output dir

        Before, it was writing the processed files into the input directory,
        which is incorrect in a vpath build.

This is useful if a test wants to use gp_toolkit.__gp_{aoseg|aocsseg}*
functions.

These are the same tests queries for column-oriented append-only tables,
as those moved by commit 11a5a807, for row-oriented append-only tables.
There were two additional tests that were never executed for row-oriented
tables though: phantom_reads_update_serializable and
phantom_reads_delete_serializable. I believe that was an oversight in the
original test suite; they are now also executed for row-oriented tables.

We use the UAO templating mechanism, to run the same test files against
row- and column-oriented tables. To make that work, fix a bug in the
templating mechanism pg_regress.c: if the --ao-dir argument was shorter
than 7 characters, the uao directory was not detected correctly.

This allows writing only one .source file for a UAO test.  Create the
.source file in "input/uao*/" directory.  Place the answer file, also
named as .source, into corresponding "output/uao*/" directory.  The
.source files must contain the following header:

   create schema <filename_prefix>@orientation@;
   set search_path="$user",<filename_prefix>@orientation@,public;
   SET gp_default_storage_options='orientation=@orientation@';

Replace "<filename_prefix>" with the filename excluding the ".source"
extension.

Generated files are named as <filename_prefix>_row.sql and
<filename_prefix>_column.sql.  Add the generated filenames to schedule
files and run pg_regress as usual.

A new option "--ao-dir" is added to pg_regress.  To enable row/column test
generation, set it to the directory name containing generic UAO .source tests.
The directory should be created under src/test/regress/input.

Commit 96f22dac expanded the feature check to handle more than just ORCA
but when moving to using passed in string variables for printing on/off
it generates a clang compiler warning on -Wformat-security:

  warning: format string is not a string literal (potentially insecure)

Move to using format strings rather than just passing the string variable
to fix the warning and also avoid any theoretical issues with the passed
string.

This is a collection of small fixes to check_feature_status() that
fell out when fixing a compiler warning in the code. The fixes are
(listed in no specific order):

  * Print tuples only in SHOW <feature>; command since we know there
    is little point in reading the header rows when we're just hunting
	for the on/off
  * Move to const char arguments since this function has no business
    altering the input
  * Explicitly clean up temporary files from feature checking and
    remove them from .gitignore files to avoid littering the tree
  * Check for overflow in command generation since we are appending
    a string with a theoretical limit of MAXPGPATH onto a string
	with MAXPGPATH size combined with additional content thus
	risking a truncation. Also fix a minor related style issue to
	match the rest of the file

Signed-off-by: NNikos Armenatzoglou <nikos.armenatzoglou@gmail.com>

Removes unused CVS $Header$ tags and moves comments closer to where
they make sense as well as updates a few comments to match reality.

We resolve the path for gpstringsubs.pl with find_other_exec() so
use the outcome of that rather than hardcoding it at invocation.

In commit 2ffa8696 we made pg_ctl recognize an environment variable
PGCTLTIMEOUT to set the default timeout for starting and stopping the
postmaster.  However, pg_regress uses pg_ctl only for the "stop" end of
that; it has bespoke code for starting the postmaster, and that code has
historically had a hard-wired 60-second timeout.  Further buildfarm
experience says it'd be a good idea if that timeout were also controlled
by PGCTLTIMEOUT, so let's make it so.  Like the previous patch, back-patch
to all active branches.

Discussion: <13969.1461191936@sss.pgh.pa.us>

Ensure that the comment match what can happen, YAML files with .yml
suffix can also be created from mapred.

This patch replaces all occurrences of pgsql-bugs@postgresql.org with
bugs@greenplum.org. Pointers to the pgsql-bugs mailinglist in the code
remain as they re. Regression and BugBuster tests are also patched.

pg_regress uses a tool called gpdiff.pl in order to compare diffs between
expected and actual result files in installcheck-good. pg_regress was hard
coded to always invoke gpdiff.pl to ignore NOTICE statements. Instead it would
be better to selectively ignore some NOTICE statements which are harmless and
which show up very often and compare the rest. This patch causes pg_regress to
invoke gpdiff.pl with an init_file which contains patterns that are to be
ignored.