We need some tests to see if binaries can be swapped around. This
commit provides a simple framework and basic tests to check if
binaries can be upgraded/downgraded by just swapping the Greenplum
binary install paths. Running the tests requires the user to have
another Greenplum compiled which will be provided as an argument to
the test script. It also requires the regression database to be built
from running tests in src/test/regress.

Signed-off-by: NAbhijit Subramanya <asubramanya@pivotal.io>

Define a xlog format for AO operations. The xlog is generated when an AO block
is written.

A test is added to receive the AO log from the WAL Sender process after an
INSERT operation and verify that the the received xlog has the AO xlog record in
it.
Signed-off-by: NAbhijit Subramanya <asubramanya@pivotal.io>

Commit 5a2d9a5a pulled in free_readfile() from upstream but one
invocation was erroneously placed too early in the logic. Move down
to after all uses of optlines are done. With that, also allocate
memory for data pulled out of the config file, partially backporting
dd917bb7 from upstream which does the same thing.

At some point in the past it seems the keyword list was merged from
upstream 8.4 without merging the actual code, and instead keywords
which weren't yet supported where commented out. The XML whitespace
keywords for STRIP WHITESPACE and PRESERVE WHITESPACE were however
not uncommented when XML support was merged, resulting in the below
error when trying to restore the XML views in ICW back after a dump:

db=# SELECT XMLPARSE(CONTENT '<abc>x</abc>'::text PRESERVE WHITESPACE) AS "xmlparse";
ERROR:  syntax error at or near "WHITESPACE"
LINE 1: ...CT XMLPARSE(CONTENT '<abc>x</abc>'::text PRESERVE WHITESPACE...
                                                             ^
Put the keywords back, and also remove commented out keywords which
we dont support but will get when merging 8.4 to reduce the diff with
upstream. Also add a testcase for XML whitespace syntax parsing.

The aosegments table is in fact using the oid of the relation and
not the relfilenode. Also add a short note about the mapping in
pg_appendonly and reflow the section with the new content.

Record memory usage for resource group.

1. Update total memory usage for a resource group when
   a session belonging to this group allocates/frees memory.

2. Update total memory usage for related resource groups
   when a session enters into or leaves from a resource group.

3. Dispatch current resource group ID from QD to QEs to
   keep track of current resource group.

4. Show total memory usage of a resource group.

5. Add test case for memory usage recording of resource group.
Signed-off-by: Nxiong-gang <gxiong@pivotal.io>
Signed-off-by: NKenan Yao <kyao@pivotal.io>
Signed-off-by: NNing Yu <nyu@pivotal.io>

Because the g++ doesn't meet gpcloud's requirements.

This is a quick fix, we better upgrade GCC to building env's version.

This commit does things below to make developers' life, who are still
using old version C++ compilers, easier.

Add a flag to optionally disable gpcloud compilation.
Add a configure check for C++11 if gpcloud is enabled.
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>

Static variables when used inside function are initialized only once and stored
on static storage area. The original code without static initializes these
variables every time the function is called and these variables are stored in
stack. Since we don't change the array value, they can be static const.

pg_dump will append an extra semicolon to function definitions prior to
printing SET statements for function GUCs. This results in the SET statements
actually being printed outside the CREATE FUNCTION statement, so the GUCs are
set on a session level rather than being set for the scope of the function.

In addition to renaming the files, a new test is added to cover the
scenario when readers need to use subtransaction xid cache in writer's
PGPROC.

NOTE: Please use `git show -M10` to see the rename instead of separate remove
and add.
Signed-off-by: NAsim R P <apraveen@pivotal.io>

Now that subtransaction information is no longer maintained in shared
snapshot between readers and a writer, the tests are obsolete.  Faults
used by the tests are removed as well.

This commit contains a gpdb_master pipeline change - to remove
"subtransaction" job that used to run the tests being removed by this
commit.
Signed-off-by: NAsim R P <apraveen@pivotal.io>

Originally, the reader kept copies of subtransaction information in
two places.  First, it copied SharedLocalSnapshotSlot to share between
writer and reader.  Second, reader kept another copy in subxbuf for
better performance.  Due to lazy xid, subtransaction information can
change in the writer asynchronously with respect to the reader.  This
caused reader's subtransaction information out of date.

This fix removes those copies of subtransaction information in the
reader and adds a reference to the writer's PGPROC to
SharedLocalSnapshotSlot.  Reader should refer to subtransaction
information through writer's PGPROC and pg_subtrans.

Also added is a lwlock per shared snapshot slot.  The lock protects
shared snapshot information between a writer and readers belonging to
the same session.

Fixes github issues #2269 and #2284.
Signed-off-by: NAsim R P <apraveen@pivotal.io>

They are not used, so remove them.

appendonlywriter.c: CID 129803 Identical code for different branches. The
condition is redundant in AtEOXact_AppendOnly_StateTransition: The same code is
executed regardless of the condition.

cdbfilerep.c: CID 129285 Buffer not null terminated. In
FileRep_GetUnknownIdentifier: The string buffer may not have a null terminator
if the source string's length is equal to the buffer size. Replaced the call
with strlcpy.

cdbmirroredflatfile.c: CID 130180 Uninitialized pointer read. In MirrorFlatFile:
Reads an uninitialized pointer or its target for mirroredOpen. Hence
initializing it.

postmaster.c: CID 129834 Argument cannot be negative. In checkIODataDirectory:
Negative value used as argument `fd` to a function `write` expecting a positive
value. Hence fixing it to be called in else only when fd not negative.

faultinject.c: CID 149132 Uninitialized scalar variable. In gp_fault_inject_impl: Use of an
uninitialized variable hdr.is_segv_msg just initializing the same.

persistentutil.c: CID 129278 Buffer not null terminated. In
gp_persistent_relation_node_check: The string buffer `fdata->databaseDirName`
may not have a null terminator if the source string's length is equal to the
buffer size. Use safer alternative strlcpy.

pg_resetxlog.c: CID 130064 Resource leak. In WriteEmptyXLOG: Leak of memory or
pointers to system resources `fp`. Hence call fclose on it.

Refactor addDistributedBy() to use less clever coding and instead
write it with readable code. Also remove variable initializations
to allow the compiler to help us catch errors, and clean up some
whitespace issues.

Before parallelization on nodes in cdbparallelize if there are
any subplan nodes in the plan which refer to the same plan_id,
parallelization step breaks as a node must be processed only
once by it. This patch fixes the issue by generating a new
subplan node in glob subplans, and updating the plan_id of the
subplan to refer to the newly created node.

The condition containing subplans will be duplicated
as the partition selection key in the PartitionSelector node.
It is not OK to duplicate the expression, if it contains
SubPlans, because the code that adds motion nodes to a
subplan gets confused if there are multiple SubPlans
referring the same subplan ID.

Skip duplicating subplans clauses as multiple subplan node
referring to same plan_id of subplans breaks cdbparallize.
cdbparallize expects to apply motion only to a node once,
so if two subplan nodes refers to the same plan_id the assertion breaks.
Signed-off-by: NJemish Patel <jpatel@pivotal.io>

Before this commit, snapshot stored information of distributed in-progress
transactions populated during snapshot creation and its corresponding localXids
found during tuple visibility check later (used as cache) by reverse mapping
using single tightly coupled data structure DistributedSnapshotMapEntry. Storing
the information this way possed couple of problems:

1] Only one localXid can be cached for a distributedXid. For sub-transactions
same distribXid can be associated with multiple localXid, but since can cache
only one, for other local xids associated with distributedXid need to consult
the distributed_log.

2] While performing tuple visibility check, code must loop over full size of
distributed in-progress array always first to check if cached localXid can be
utilized to avoid reverse mapping.

Now, decoupled the distributed in-progress with local xids cache separately. So,
this allows us to store multiple xids per distributedXid. Also, allows to
optimize scanning localXid only if tuple xid is relevant to it and also scanning
size only equivalent to number of elements cached instead of size of distributed
in-progress always even if nothing was cached.

Along the way, refactored relevant code a bit as well to simplify further.

Few minor updates in error messages and comments.
Signed-off-by: NDhanashree Kashid <dkashid@pivotal.io>

This commit updates the `rangefuncs_cdb` tests to use `plpgsql`
instead of `plperlu`; since `plperlu` is not supported by SLES.

We do not need a separate answer file for ORCA because the results
produced by planner and ORCA are the same. Hence this commit also
removes `rangefuncs_cdb_optimizer.out` file.
Signed-off-by: NEkta Khanna <ekhanna@pivotal.io>

* Move ddboost cleanup step to be last to ensure all files in directory are removed from data domain server

* Update documentation to reflect new test target:
"make installcheck-good" -> "make installcheck-world"

Provide alternative answer for gp_metadata test when GPORCA is not compiled in

Also, add retry to check for walsender is gone after walrcv_disconnect() as it
can take sometime to detect the connection drop.

For subselect_gp test, we were removing the distribution policy of a
table to see if it would do a gather motion or not. Since it's
technically a corrupted table, we should delete it after we're done
with it. We also remove a quicklz reference that should not have been
there.

For gppc test, they were using the regression database. This made our
gpcheckcat call at the end of ICW relatively useless since all our
data would have been deleted due to gppc tests recreating the
regression database.

For gpload test, some generated files were previously commited. We
should be actively cautious of this and remove them when we see them.

PL/Perl is an optional component, and the main ICW should not use
it as it may not be present. Move the tests that seem useful to
the plperl test suite instead and remove the ones which we have
ample coverage for elsewhere.

The gpmapreduce application is an optional install included via
the --enable-mapreduce configure option. The tests were however
still in src/test/regress and unconditionally included in the
ICW schedule, thus causing test failures when mapreduce wasn't
configured.

Move all gpmapreduce tests to co-locate them with the mapreduce
code and only test when configured.

Also, add a dependency on Perl for gpmapreduce in autoconf since
it's a required component.

- Before building Index object (IMDIndex), we build LogicalIndexes
  via calling `gpdb::Plgidx(oidRel)` in which a partition tables is
  traversed and index  information (such as logicalIndexOid,
  nColumns, indexKeys, indPred, indExprs, indIsUnique, partCons,
  defaultLevels) is captured.
- For Indexes which are available on all the partitions partCons
  and defaultLevels are NULL/empty.
- Later in `CTranslatorRelcacheToDXL::PmdindexPartTable` to build
  Index object, we use the derived LogicalIndexes information and
  populates the array  holding the levels on which default
  partitions exists. But since defaultLevels is NIL in this case,
  pdrgpulDefaultLevels is set to empty i,e `default partitions on levels: {}`
- This causes an issue while trying to build the propagation expression,
  as because of wrong number of default partitions on level we mark the
  scan as partial and tries to construct a test propagation expression
  instead of a const propagation expression.
- This patch fixes the issue by marking the default partitions on
  levels for index equal to the default partitions on levels for the
  part relation if the index exists on all the parts.
Signed-off-by: NJemish Patel <jpatel@pivotal.io>

This patch is to fix below defects reported by Coverity Scan.

```
New defect(s) Reported-by: Coverity Scan
Showing 3 of 3 defect(s)

** CID 169611:  Security best practices violations  (TOCTOU)
/tmp/build/0e1b53a0/gpdb_src/src/backend/utils/resgroup/resgroup-ops-cgroup.c:
272 in createDir()

________________________________________________________________________________________________________
*** CID 169611:  Security best practices violations  (TOCTOU)
/tmp/build/0e1b53a0/gpdb_src/src/backend/utils/resgroup/resgroup-ops-cgroup.c:
272 in createDir()
266
267             buildPath(group, comp, "", path, pathsize);
268
269             if (access(path, F_OK))
270             {
271                     /* the dir is not created yet, create it */
>>>     CID 169611:  Security best practices violations  (TOCTOU)
>>>     Calling function "mkdir" that uses "path" after a check
>>>     function. This can cause a time-of-check, time-of-use race
>>>     condition.
272                     if (mkdir(path, 0755) && errno != EEXIST)
273                             return false;
274             }
275
276             return true;
277     }

** CID 169610:  Control flow issues  (NO_EFFECT)
/tmp/build/0e1b53a0/gpdb_src/src/backend/utils/resgroup/resgroup-ops-cgroup.c:
412 in writeData()

________________________________________________________________________________________________________
*** CID 169610:  Control flow issues  (NO_EFFECT)
/tmp/build/0e1b53a0/gpdb_src/src/backend/utils/resgroup/resgroup-ops-cgroup.c:
412 in writeData()
406             size_t ret = write(fd, data, datasize);
407
408             /* save errno before close */
409             int err = errno;
410             close(fd);
411
>>>     CID 169610:  Control flow issues  (NO_EFFECT)
>>>     This less-than-zero comparison of an unsigned value is never
>>>     true. "ret < 0UL".
412             if (ret < 0)
413                     CGROUP_ERROR("can't write data to file '%s':
%s", path, strerror(err));
414             if (ret != datasize)
415                     CGROUP_ERROR("can't write all data to file
'%s'", path);
416     }
417

** CID 169609:  Control flow issues  (NO_EFFECT)
/tmp/build/0e1b53a0/gpdb_src/src/backend/utils/resgroup/resgroup-ops-cgroup.c:
385 in readData()

________________________________________________________________________________________________________
*** CID 169609:  Control flow issues  (NO_EFFECT)
/tmp/build/0e1b53a0/gpdb_src/src/backend/utils/resgroup/resgroup-ops-cgroup.c:
385 in readData()
379             size_t ret = read(fd, data, datasize);
380
381             /* save errno before close() */
382             int err = errno;
383             close(fd);
384
>>>     CID 169609:  Control flow issues  (NO_EFFECT)
>>>     This less-than-zero comparison of an unsigned value is never
>>>     true. "ret < 0UL".
385             if (ret < 0)
386                     CGROUP_ERROR("can't read data from file '%s':
%s", path, strerror(err));
387
388             return ret;
389     }
390
```

`outputdir` here is where to place the converted files, not the
directory named `output`. PostgreSQL places them into `outputdir`
rightly.

    commit 64cdbbc4
    Author: Peter Eisentraut <peter_e@gmx.net>
    Date:   Sat Feb 14 21:33:41 2015 -0500

        pg_regress: Write processed input/*.source into output dir

        Before, it was writing the processed files into the input directory,
        which is incorrect in a vpath build.

This way we are consistent now with the upstream Postgres.

ON MASTER feature is not fully supported by built-in protocols other
than s3 yet, disable it with those protocols for now.