Co-authored-by: NChris Hajas <chajas@pivotal.io>
Co-authored-by: NShreedhar Hardikar <shardikar@pivotal.io>

Adds tests for ORCA commit 'Fix wrong results from LOJ reordering'

gphdfs is being deprecated for GPDB 6. This commit removes the
following:
- Remove gphdfs from pipelines
- Remove gphdfs from gpcontrib
- Remove gphdfs from Makefile
- Remove tests using gphdfs
- Remove GUCs, code, and comments used for gphdfs
- Remove gphdfs from GPDB docs
- Remove avro and parquet formats

Authored-by: NKaren Huddleston <khuddleston@pivotal.io>

We backported 128-bit integer support to speed up aggregates (commits
8122e143 and 959277a4) from upstream 9.6 into Greenplum (in
commits 9b164486 and 325e6fcd). However, we forgot to also port a
follow-up fix postgres/postgres@7518049980b, mostly because it's nuanced
and hard to reproduce.

There are two ways to tell the brokenness:

1. On a lucky day, tests would fail on my workstation, but not my laptop (or
   vice versa).

1. If you stare at the generated code for `int8_avg_combine` (and friends),
   you'll notice the compiler uses "aligned" instructions like `movaps` and
   `movdqa` (on AMD64).

Today's my lucky day.

Original commit message from postgres/postgres@7518049980b (by Tom Lane):

> Our initial work with int128 neglected alignment considerations, an
> oversight that came back to bite us in bug #14897 from Vincent Lachenal.
> It is unsurprising that int128 might have a 16-byte alignment requirement;
> what's slightly more surprising is that even notoriously lax Intel chips
> sometimes enforce that.

> Raising MAXALIGN seems out of the question: the costs in wasted disk and
> memory space would be significant, and there would also be an on-disk
> compatibility break.  Nor does it seem very practical to try to allow some
> data structures to have more-than-MAXALIGN alignment requirement, as we'd
> have to push knowledge of that throughout various code that copies data
> structures around.

> The only way out of the box is to make type int128 conform to the system's
> alignment assumptions.  Fortunately, gcc supports that via its
> __attribute__(aligned()) pragma; and since we don't currently support
> int128 on non-gcc-workalike compilers, we shouldn't be losing any platform
> support this way.

> Although we could have just done pg_attribute_aligned(MAXIMUM_ALIGNOF) and
> called it a day, I did a little bit of extra work to make the code more
> portable than that: it will also support int128 on compilers without
> __attribute__(aligned()), if the native alignment of their 128-bit-int
> type is no more than that of int64.

> Add a regression test case that exercises the one known instance of the
> problem, in parallel aggregation over a bigint column.

> This will need to be back-patched, along with the preparatory commit
> 91aec93e.  But let's see what the buildfarm makes of it first.

> Discussion: https://postgr.es/m/20171110185747.31519.28038@wrigleys.postgresql.org

(cherry picked from commit 75180499)

- Added with-quicklz configure flag
- Added quicklz gpcontrib directory with C wrapper functions and SQL installation file
- Added simple quicklz functional tests
- Added #undef HAVE_LIBQUICKLZ to pg_config.h.win32.
  This is to parallel the recent change in pg_config.h.in that adds
  quicklz. pg_config.h.win32 should be autogenerated, but isn't in
  practice.
Co-authored-by: NJimmy Yih <jyih@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>
Co-authored-by: NDavid Sharp <dsharp@pivotal.io>

Co-authored-by: NChris Hajas <chajas@pivotal.io>

This reverts commit dbece3da.

Performance regression observed for tpcds queries due to cardinality
misestimation. Impacted TPCDS queries 174, 111 and 104.
Signed-off-by: NSambitesh Dash <sdash@pivotal.io>

- set config flags to disable zstd in platforms that don't have the
  library baked in
  - currently only centos6 and centos7 have zstd provided in the build
    images
- delete the suse10 config flags, because it is no longer a supported
  platform
- add EXTRA_CONFIG_FLAGS to deb_create_package.bash
Co-authored-by: NTaylor Vesely <tvesely@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>
Co-authored-by: NKaren Huddleston <khuddleston@pivotal.io>

ORCA version to v3.17.0

This patch introduces a guc `optimizer_enable_agg_skew_avoidance` which
can be used to avoid plans which generate a lot of skew when aggregates
are used in the query. The default value of this guc is true.
Co-authored-by: NAbhijit Subramanya <asubramanya@pivotal.io>
Co-authored-by: NHans Zeller <hzeller@pivotal.io>

Also bump ORCA's version to 3.16.0
Co-authored-by: NHans Zeller <hzeller@pivotal.io>

The support for sending alerts via Email or SNMP was quite a kludge,
and there are much better external tools for managing alerts than
what we can supply in core anyways so this retires the capability.

All references to alert sending in the docs are removed, but there
needs to be section written about how to migrate off this feature
in the release notes or a similar location.

Discussion: https://github.com/greenplum-db/gpdb/pull/6384

Co-authored-by: NChris Hajas <chajas@pivotal.io>
Co-authored-by: NSambitesh Dash <sdash@pivotal.io>

When ZSTD compression is used for AO CO tables, insertion of data may cause an
error "Destination buffer is too small". This happens when compressed data is
larger than uncompressed input data.

This commit adds handling of this situation: do not change output buffer and
return size used equal to source size. The caller (e.g.,
'cdbappendonlystoragewrite.c') is able to handle such output; in this case, it
copies data from input to output itself.

Co-authored-by: NSambitesh Dash <sdash@pivotal.io>

The solution arrived at in commit e74dd00f presumes that the compiler
has a suitable default -isysroot setting ... but further experience
shows that in many combinations of macOS version, XCode version, Xcode
command line tools version, and phase of the moon, Apple's compiler
will *not* supply a default -isysroot value.

We could potentially go back to the approach used in commit 68fc227d,
but I don't have a lot of faith in the reliability or life expectancy of
that either.  Let's just revert to the approach already shipped in 11.0,
namely specifying an -isysroot switch globally.  As a partial response to
the concerns raised by Jakob Egger, adjust the contents of Makefile.global
to look like

CPPFLAGS = -isysroot $(PG_SYSROOT) ...
PG_SYSROOT = /path/to/sysroot

This allows overriding the sysroot path at build time in a relatively
painless way.

Add documentation to installation.sgml about how to use the PG_SYSROOT
option.  I also took the opportunity to document how to work around
macOS's "System Integrity Protection" feature.

As before, back-patch to all supported versions.

Discussion: https://postgr.es/m/20840.1537850987@sss.pgh.pa.us

The --enable-orafce and --enable-gphdfs were both described as "do
not build X", probably from copy/pasting another PGAC_OPTION_BOOL
which defaults to true and thus becomes --disable-X.
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

Co-authored-by: NTaylor Vesely <tvesely@pivotal.io>

Update the tests and bump orca version corresponding to the below fix in
ORCA.

If the Scalar Agg node has child which has volatile function and has
universal distribution for relational child, instead of deriving the
spec as that of the child i.e Universal Spec, we should derive the
distribution spec as singleton.

Deriving singleton distribution spec will ensure that the tree is
executed on one node and the results will be distributed to the
remaining segments.

Consider the below setup:

create table f (a text);
explain insert into f select sum(random()) from (select 1);

Physical plan:
+--CPhysicalDML (Insert, "foo"), Source Columns: ["f1" (2)], Action: ("ColRef_0004" (4))   rows:1   width:1  rebinds:1   cost:0.016658   origin: [Grp:12, GrpExpr:2]
   +--CPhysicalComputeScalar   rows:1   width:1  rebinds:1   cost:0.001033   origin: [Grp:23, GrpExpr:3]
      |--CPhysicalMotionHashDistribute HASHED: [ +--CScalarIdent "f1" (2)
 , nulls colocated, duplicate sensitive ]   rows:1   width:1  rebinds:1   cost:0.001029   origin: [Grp:11, GrpExpr:2]
      |  +--CPhysicalComputeScalar   rows:1   width:1  rebinds:1   cost:0.001008   origin: [Grp:11, GrpExpr:1]
      |     |--CPhysicalScalarAgg( Global ) Grp Cols: [], Minimal Grp Cols:[], Generates Duplicates :[ 0 ]    rows:1   width:8  rebinds:1   cost:0.001000   origin: [Grp:6, GrpExpr:3]
      |     |  |--CPhysicalTVF   rows:1000   width:1  rebinds:1   cost:0.001000   origin: [Grp:1, GrpExpr:1]
      |     |  |  |--CScalarConst (1)   origin: [Grp:0, GrpExpr:0]
      |     |  |  +--CScalarConst (1)   origin: [Grp:0, GrpExpr:0]
      |     |  +--CScalarProjectList   origin: [Grp:5, GrpExpr:0]
      |     |     +--CScalarProjectElement "sum" (1)   origin: [Grp:4, GrpExpr:0]
      |     |        +--CScalarAggFunc (sum , Distinct: false , Aggregate Stage: Global)   origin: [Grp:3, GrpExpr:0]
      |     |           +--CScalarFunc (random)   origin: [Grp:2, GrpExpr:0]
      |     +--CScalarProjectList   origin: [Grp:10, GrpExpr:0]
      |        +--CScalarProjectElement "f1" (2)   origin: [Grp:9, GrpExpr:0]
      |           +--CScalarCoerceViaIO   origin: [Grp:8, GrpExpr:0]
      |              +--CScalarIdent "sum" (1)   origin: [Grp:7, GrpExpr:0]
      +--CScalarProjectList   origin: [Grp:22, GrpExpr:0]
         +--CScalarProjectElement "ColRef_0004" (4)   origin: [Grp:21, GrpExpr:0]
            +--CScalarConst (1)   origin: [Grp:0, GrpExpr:0]
",
                                             QUERY PLAN
----------------------------------------------------------------------------------------------------
 Insert  (cost=0.00..0.02 rows=1 width=8)
   ->  Result  (cost=0.00..0.00 rows=1 width=12)
         ->  Result  (cost=0.00..0.00 rows=1 width=8)
               ->  Result  (cost=0.00..0.00 rows=1 width=8)
                     ->  Aggregate  (cost=0.00..0.00 rows=1 width=8)
                           ->  Function Scan on generate_series  (cost=0.00..0.00 rows=334 width=1)

Above CPhysicalMotionHashDistribute has a universal child (i.e
CPhysicalTVF), and translator will convert it to a Result Node with Hash
Filter. The hash filter is build on the projected column in this query
i.e sum(random()).. Random being a volatile function, will cause the
resulting sum to be non-deterministic i.e volatile in this case. So the
hash filter may not appropriately filter out the segment and the insert
may result in inserting non-deterministic number of rows in the table
(with the maximum being equal to the number of segments).  So, we should
derive singleton distribution if the relation child of the scalar agg
node delivers universal spec and the project list has volatile
functions.

Resulting plan will be:

pivotal=# explain insert into foo select sum(random()) from (select 1) f;
                                   QUERY PLAN
---------------------------------------------------------------------------------
 Insert  (cost=0.00..0.02 rows=1 width=8)
   ->  Result  (cost=0.00..0.00 rows=1 width=12)
         ->  Redistribute Motion 1:3  (slice1)  (cost=0.00..0.00 rows=1 width=8)
               Hash Key: (((sum(random())))::text)
               ->  Result  (cost=0.00..0.00 rows=1 width=8)
                     ->  Aggregate  (cost=0.00..0.00 rows=1 width=8)
                           ->  Result  (cost=0.00..0.00 rows=1 width=1)

To avoid the sorts of problems complained of by Jakob Egger, it'd be
best if configure didn't emit any references to the sysroot path at all.
In the case of PL/Tcl, we can do that just by keeping our hands off the
TCL_INCLUDE_SPEC string altogether.  In the case of PL/Perl, we need to
substitute -iwithsysroot for -I in the compile commands, which is easily
handled if we change to using a configure output variable that includes
the switch not only the directory name.  Since PL/Tcl and PL/Python
already do it like that, this seems like good consistency cleanup anyway.

Hence, this replaces the advice given to Perl-related extensions in commit
5e221713; instead of writing "-I$(perl_archlibexp)/CORE", they should
just write "$(perl_includespec)".  (The old way continues to work, but not
on recent macOS.)

It's still the case that configure needs to be aware of the sysroot
path internally, but that's cleaner than what we had before.

As before, back-patch to all supported versions.

Discussion: https://postgr.es/m/20840.1537850987@sss.pgh.pa.us

Also bump ORCA version to 3.4.0

Rethink the solution applied in commit 5e221713 to get PL/Tcl to
build on macOS Mojave.  I feared that adding -isysroot globally might
have undesirable consequences, and sure enough Jakob Egger reported
one: it complicates building extensions with a different Xcode version
than was used for the core server.  (I find that a risky proposition
in general, but apparently it works most of the time, so we shouldn't
break it if we don't have to.)

We'd already adopted the solution for PL/Perl of inserting the sysroot
path directly into the -I switches used to find Perl's headers, and we
can do the same thing for PL/Tcl by changing the -iwithsysroot switch
that Apple's tclConfig.sh reports.  This restricts the risks to PL/Perl
and PL/Tcl themselves and directly-dependent extensions, which is a lot
more pleasing in general than a global -isysroot switch.

Along the way, tighten the test to see if we need to inject the sysroot
path into $perl_includedir, as I'd speculated about upthread but not
gotten round to doing.

As before, back-patch to all supported versions.

Discussion: https://postgr.es/m/20840.1537850987@sss.pgh.pa.us

Also bump ORCA version to v3.2.0.

We noticed the following error during configure:

    checking Python configuration directory... ./configure: line 9650: 2: command not found

The brackets for the `if` were being swallowed by m4.

Remove the "major version = 2" case, since it seems reasonable that no
platform was executing that path anyway, and move this block of code
closer to upstream Postgres.
Co-authored-by: NDavid Kimura <dkimura@pivotal.io>
Co-authored-by: NJim Doty <jdoty@pivotal.io>