- Created the needed control files
- Renamed and adapted the installation script
- Remove redundant tests in regress that were just checking installation
- Add tests to ensure installation is successfull
- Updated the Makefile to support the extension and add information
about regression tests
- Add contrib/gp_internal_tools tests to ICW
- Remove session_level_memory_consumption from regress test suite

Replace function `cdbpath_rows(root, path)` with path->rows, this is more in
line with upstream 9.2, thus removes a GPDB_92_MERGE_FIXME

Co-authored-by: Alexandra Wang<lewang@pivotal.io>
Co-authored-by: Gang Xiong<gxiong@pivotal.io>

Variable numFreeProcs is added by GPDB to accelerate HaveNFreeProcs(),
but considering the argument size of calls to HaveNFreeProcs(), this
optimization may be not worth it.
In addition, the current codes calculate numFreeProcs wrong.

So remove numFreeProcs to keep the same as PostgreSQL.

Relevant thread from gpdb-dev: https://groups.google.com/a/greenplum.org/d/msg/gpdb-dev/NG8t5XXYQB0/7BDVoQQLBAAJ

We had backported the tuplesort_skiptuples() function, as part of commit
fd6212ce which backported upstream support for ordered-set aggregates.
Since we backported the feature, we also need to keep backporting all
bugfixes that follow, until we catch up to that in the merge.

In GPDB, the same fix needs to also be applied to the "mk sort" variant.

Fixes github issue #5334.

Upstream commit:

commit 1def747d
Author: Tom Lane <tgl@sss.pgh.pa.us>
Date:   Tue Dec 24 17:13:02 2013 -0500

    Fix inadequately-tested code path in tuplesort_skiptuples().

    Per report from Jeff Davis.

A parse object has been copyed to subroot, and We do not free
root->simple_rel_array and root->simple_rte_array. So We do not
need to copy group clause.

The show_gp_connections_per_thread function is not needed, because the
default "show" functionality for an integer GUC does the same.

* docs - add missing gp-specific options to pg_dump

* qualify the options as unsupported

* use a note

Signed-off-by: NAbhijit Subramanya <asubramanya@pivotal.io>

Prior to this commit, there was no support for GiST indexes in GPORCA.
For queries involving GiST indexes, ORCA was selecting Table Scan paths
as the optimal plan. These plans could take up to 300+ times longer than
Planner, which generated a index scan plan using the GiST index.

Example:
```
CREATE TABLE gist_tbl (a int, p polygon);
CREATE TABLE gist_tbl2 (b int, p polygon);
CREATE INDEX poly_index ON gist_tbl USING gist(p);

INSERT INTO gist_tbl SELECT i, polygon(box(point(i, i+2),point(i+4,
i+6))) FROM generate_series(1,50000)i;
INSERT INTO gist_tbl2 SELECT i, polygon(box(point(i+1, i+3),point(i+5,
i+7))) FROM generate_series(1,50000)i;

ANALYZE;
```
With the query `SELECT count(*) FROM gist_tbl, gist_tbl2 WHERE
gist_tbl.p <@ gist_tbl2.p;`, we see a performance increase with the
support of GiST.

Before:
```
EXPLAIN SELECT count(*) FROM gist_tbl, gist_tbl2 WHERE gist_tbl.p <@ gist_tbl2.p;
                                                     QUERY PLAN
---------------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=0.00..171401912.12 rows=1 width=8)
   ->  Gather Motion 3:1  (slice2; segments: 3)  (cost=0.00..171401912.12 rows=1 width=8)
         ->  Aggregate  (cost=0.00..171401912.12 rows=1 width=8)
               ->  Nested Loop  (cost=0.00..171401912.12 rows=335499869 width=1)
                     Join Filter: gist_tbl.p <@ gist_tbl2.p
                     ->  Table Scan on gist_tbl2  (cost=0.00..432.25 rows=16776 width=101)
                     ->  Materialize  (cost=0.00..530.81 rows=49997 width=101)
                           ->  Broadcast Motion 3:3  (slice1; segments: 3)  (cost=0.00..525.76 rows=49997 width=101)
                                 ->  Table Scan on gist_tbl  (cost=0.00..432.24 rows=16666 width=101)
 Optimizer status: PQO version 2.65.1
(10 rows)

Time: 170.172 ms
SELECT count(*) FROM gist_tbl, gist_tbl2 WHERE gist_tbl.p <@ gist_tbl2.p;
 count
-------
 49999
(1 row)

Time: 546028.227 ms
```

After:
```
EXPLAIN SELECT count(*) FROM gist_tbl, gist_tbl2 WHERE gist_tbl.p <@ gist_tbl2.p;
                                                  QUERY PLAN
---------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=0.00..21749053.24 rows=1 width=8)
   ->  Gather Motion 3:1  (slice2; segments: 3)  (cost=0.00..21749053.24 rows=1 width=8)
         ->  Aggregate  (cost=0.00..21749053.24 rows=1 width=8)
               ->  Nested Loop  (cost=0.00..21749053.24 rows=335499869 width=1)
                     Join Filter: true
                     ->  Broadcast Motion 3:3  (slice1; segments: 3)  (cost=0.00..526.39 rows=50328 width=101)
                           ->  Table Scan on gist_tbl2  (cost=0.00..432.25 rows=16776 width=101)
                     ->  Bitmap Table Scan on gist_tbl  (cost=0.00..21746725.48 rows=6667 width=1)
                           Recheck Cond: gist_tbl.p <@ gist_tbl2.p
                           ->  Bitmap Index Scan on poly_index  (cost=0.00..0.00 rows=0 width=0)
                                 Index Cond: gist_tbl.p <@ gist_tbl2.p
 Optimizer status: PQO version 2.65.1
(12 rows)

Time: 617.489 ms

SELECT count(*) FROM gist_tbl, gist_tbl2 WHERE gist_tbl.p <@ gist_tbl2.p;
 count
-------
 49999
(1 row)

Time: 7779.198 ms
```

GiST support was implemented by sending over GiST index information to
GPORCA in the metadata using a new index enum specifically for GiST.
Signed-off-by: NBhuvnesh Chaudhary <bchaudhary@pivotal.io>
Signed-off-by: NEkta Khanna <ekhanna@pivotal.io>

I don't think anyone's tried to run this on Greenplum, until @imshubhangi
reported in github issue #5241, that it doesn't work. Most of the failures
are just a matter of updating the expected output for extra NOTICES, and
using gpdiff.pl to mask out differences in row order. That's what this
patch does.

However, there is one remaining failure, in the COPY TO STDOUT at the
end of the 'sjis' test, which looks like a genuine bug. That needs to be
investigated and fixed, but let's do that as a separate commit.

converted insist_log to ereport to match upstream
Changed the log level on lock.c from LOG to ERROR so that ereport will
halt the execution without using Insist

Remove the usages of Insist and replace them with Assert instead

Changed only the files regress_gp

If GPOS_ASSERT() evaluates to a no-op, the 'pmdagg' variable was unused,
and generated a compiler warning.

Commit unsued 030ec314 commented out the function with #if 0, do the
same for its prototype.

Remove the the unused pre-7.3 priority table; it has been out of sync
for a while now and we don't want to accidentally make use of it in the
future. It'll disappear in the 10.x merge with upstream.
Co-authored-by: NNadeem Ghani <nghani@pivotal.io>
Co-authored-by: NJim Doty <jdoty@pivotal.io>

This was caused by a misplaced parenthesis which caused the check
to always return false.

If we don't find any AO tables we exit early, but we failed to close
the PQExpBuffer we had for the query. Fix by destroying the buffer
explicitly.

Move the freeing of numeric_rels to be unconditionally since pg_free
can cope with a NULL pointer.

Save the quote_identifier() returned strings in a char * rather than
passing them to fprint() so we can pg_free() them on the way out.

This is a partial backport of the below commit from upstream, with
one hunk removed as it touches code yet in the future of this fork
and another hunk massaged to account for path changes. Since we
want the pg_upgrade hunk from this commit to silenve Coverity, we
may as well grab all the hunks that are relevant and close those
leaks ahead time in the fork.

  commit f712289f
  Author: Heikki Linnakangas <heikki.linnakangas@iki.fi>
  Date:   Thu Jul 2 20:58:51 2015 +0300

    Plug some trivial memory leaks in pg_dump and pg_upgrade.

    There's no point in trying to free every small allocation in these
    programs that are used in a one-shot fashion, but these ones seems like
    an improvement on readability grounds.

    Michael Paquier, per Coverity report.

For a PL/pgSQL function like the following:

set optimizer_trace_fallback to on;
CREATE OR REPLACE FUNCTION boom()
RETURNS bool AS $$
DECLARE
	mel bool;
	sesh int[];
BEGIN
	sesh := '{42,1}'::int[]; -- query 1
	select c = ANY (sesh) INTO mel FROM (values (42), (0)) nums(c); -- query 2
	return mel;
END
$$
LANGUAGE plpgsql VOLATILE;

SELECT boom();

With Orca enabled, the database crashes.  Starting in 9.2, PL/pgSQL
supplies bound param values in more statement types to enable planner to
fold constants in more cases. This is in contrast to leaving the param
intact and waiting until execution to substitute it with its values.
Previously, only dynamic execution ("EXECUTE 'SELECT $1' USING sesh")
gets this treatment.  This revealed the bug because Orca would not have
been able to plan queries whose query trees included params that were
not in subplans (external params) and would just fall back.

When query 1 is planned, it is translated into select '{42,1}'::int[];
For uninteresting reasons, the planner-produced plan for query 1 is
considered "simple", and the ORCA-produced plan is considered regular
(not simple). PL/pgSQL has a fast-path for "simple" plans, minimally
starting the executor via `ExecEvalExpr`. Regular plans are executed
through SPI. During execution, SPI will pack (as part of
`heap_form_tuple`) the 4-byte header datum into a 1-byte header datum.

While planning query 2, we will attempt to substitute the param "sesh"
with the actual const value during pre-processing.  Since Orca doesn't
recognize const arrays as arrays, the translator will take the
additional step of translating the const into an array expression.  When
accessing the array-typed const, we need to "unpack"
(`DatumGetArrayTypeP`) the datum.  This commit does that.
Co-authored-by: NMelanie Plageman <mplageman@pivotal.io>

There are a lot of blank lines in the 'gpinitsystem' log. These are
read directly from the configuration file using commands 'cat'. The
comments are turned into blank lines, but they are not removed:

  gpinitsystem:node:gp6-[INFO]:-Start Main
  gpinitsystem:node:gp6-[INFO]:-Command line options passed to utility = -c ../gpinitsystem_config
  gpinitsystem:node:gp6-[INFO]:-Start Function CHK_GPDB_ID
  ...
  gpinitsystem:node:gp6-[INFO]:-End Function CHK_FILE
  gpinitsystem:node:gp6-[INFO]:-Dumping gpinitsystem_config to logfile for reference

  ARRAY_NAME="EMC Greenplum DW"

  SEG_PREFIX=gpseg

  PORT_BASE=40300

This extends the exclusion regex used when appending to the logfile
to remove blank lines completely and to handle comments that doesn't
start on column zero.
Co-authored-by: NDaniel Gustafsson <dgustafsson@pivotal.io>
Reviewed-by: NDaniel Gustafsson <dgustafsson@pivotal.io>

Commit 8e60838c22735fcacabc125170fc1d removed PG_PRINTF_ATTRIBUTE
from pg_config_manual.h, which exposed the fact that gpmapreduce
was erroneously including that header instead of the correct one.
Instead include postgres_fe.h for now, as gpmapreduce is client
side tool and not a separated extension, to fix compilation.
Reviewed-by: NAsim R P <apraveen@pivotal.io>

The GUCs controlling filerep were removed when filerep was replaced
by walrep, but the documentation hasn't caught up.  Remove mentions
of filerep GUCs as they no longer exist.
Reviewed-by: NAsim R P <apraveen@pivotal.io>
Reviewed-by: NMel Kiyama <mkiyama@pivotal.io>

1)Refactor function externalgettup_custom
  scan->raw_buf_done means there is no new data in the formatter->fmt_databuf( a block data) for the formatter to process. Maybe there is some data left in the fmt_databuf, but it is not a complete tuple and formatter can not  process it .pstate->fe_eof  means has no data left in the external file. When scan->raw_buf_done and pstate->fe_eof are both true, it means there is no new data for formatter to process.
  If there is no new data and still there is some data in the formatter->fmt_databuf, it means the external file is not complete.
2) ereports WARNING instead of ERROR if the external file is not complete. If reports error, the transaction will rollback. To ignore the incomplete data at the end of file is better.

Nowait could set false, when we get a object address

The bug would only show up if the C sockaddr structure contained
zero in the first byte for a valid address; otherwise it would
fail to fail, which is probably why it went unnoticed for so long.

Patch submitted by Joel Jacobson after seeing an article by Andrey
Karpov in which he reports finding this through static code
analysis using PVS-Studio.  While I was at it I moved a definition
of a local variable referenced in the buggy code to a more local
context.

Backpatch to all supported branches.

Remove GPDB_92_MERGE_FIXME in cdbpath.c and change create_new_ph from
'true' to false.

When the function 'add_vars_to_targetlist' is called. The param 'create_new_ph'
should be true before deconstruct_jointree, and false after that. The funtion
'cdbpath_dedup_fixup_baserel' is always called after deconstruct_jointree. So
it should be false.

These were detected by compiler option -Wimplicit-fallthrough (gcc 7.x).
Some of these issues are introduced by git merge during postgres merge.

We've seen similar cases (missing break) several times so we probably
want this compiler option as the default in the future after we modify
code to remove all or most false alarms. Before that, we will need to
run with this option locally during postgres merge.

GPDB includes newer PG code that detects PG_PRINTF_ATTRIBUTE automatically,
thus there is no need to define it in pg_config_manual.h.

The commit a3f7f4d7 introduced a fall back if the query contained ONLY
in the FROM clause for relations. However, it should exclude external
tables. For external tables RangeTblEntry::inh is set to false always,
so the commit causes ORCA to fall back for all external table queries.

This commit fixes the issue by excluding external tables from the check
of ONLY clause in CTranslatorQueryToDXL::PdxlnFromRelation(), and adds
relevant test cases.

FindOperatorGroupForNode() has been commented out for six years
almost to the day, with no reason in the original commit as to
why it was commented out in the first place. Remove since it
clearly is neither used nor needed.
Reviewed-by: NJoao Pedro De Almeida Pereira <jdealmeidapereira@pivotal.io>

This commit removes the FIXME along with the code that was disabled as
part of the FIXME, since it is no more relevant.

Prior to the merge, this code was introduced to handle the following
cases:
1. In case of UPDATE, DELETE and INSERT, we would set all plans to
`DISPATCH_PARALLEL` if the relation is either POLICYTYPE_REPLICATED or
POLICYTYPE_PARTITIONED
2. If UPDATE, DELETE and INSERT have returning as part of the sql and
the relation is either `POLICYTYPE_REPLICATED` or `POLICYTYPE_PARTITIONED`,
it should error out as `RETURNING` was not supported.

Post merge,
1. The dispatch for UPDATE, DELETE and INSERT plan is set to
`DISPATCH_PARALLEL` in the function `prescan()`, called later in the
code, based on `FLOW`
Earlier, the checks were explicitly required before the `prescan()`
as this function sets the `DISPATCH_PARALLEL` based on `FLOW` and the
necessary `FLOW` information was not passed on for all cases.

Now, for UPDATE, DELETE and INSERT plans, planner creates a
`ModifyTable` plan node and the appropriate FLOW information is set for
it in `adjust_modifytable_flow()`, including the case where the relation
is either POLICYTYPE_PARTITIONED or POLICYTYPE_REPLICATED.
Additionally, for case of no-op queries(as shown below), where we do not
create a `ModifyTable` plan node, we do not need to set the plan dispatch
as `DISPATCH_PARALLEL`
```
CREATE TABLE target(a int, b int, c int)
 PARTITION BY RANGE(c) (START(0) END(6) EVERY(3));

UPDATE target SET b=10 where c =10;
```

2. As part of the merge, commit 8a736a5f enabled `RETURNING` support in
GPDB, leaving case 2 above irrelevant.

To avoid format poisoning and possible exploitation (albetit unlileky),
always pass a format string to gfile_printf_then_putc_newline(). This
avoids compiler warnings like the following:

	warning: format string is not a string literal (potentially insecure)
	                gfile_printf_then_putc_newline(*response_string);
                                               ^~~~~~~~~~~~~~~~
Reviewed-by: NVenkatesh Raghavan <vraghavan@pivotal.io>