ORCA can do some optimizations - partition pruning at least - if it can
"see" into the elements of an array in a ScalarArrayOpExpr. For example, if
you have a qual like "column IN (1, 2, 3)", and the table is partitioned on
column, it can eliminate partitions that don't hold those values. The
IN-clause is converted into an ScalarArrayOpExpr, so that is really
equivalent to "column = ANY <array>"

However, ORCA doesn't know how to extract elements from an array-typed
Const, so it can only do that if the array in the ScalarArrayOpExpr is
an ArrayExpr. Normally, eval_const_expressions() simplifies any ArrayExprs
into Const, if all the elements are constants, but we had disabled that
when ORCA was used, to keep the ArrayExprs visible to it.

There are a couple of reasons why that was not a very good solution. First,
while we refrain from converting an ArrayExpr to an array Const, it doesn't
help if the argument was an array Const to begin with. The "x IN (1,2,3)"
construct is converted to an ArrayExpr by the parser, but we would miss the
opportunity if it's written as "x = ANY ('{1,2,3}'::int[])" instead.
Secondly, by not simplifying the ArrayExpr, we miss the opportunity to
simplify the expression further. For example, if you have a qual like
"1 IN (1,2)", we can evaluate that completely at plan time to 'true', but
we would not do that with ORCA because the ArrayExpr was not simplified.

To be able to also optimize those cases, and to slighty reduce our diff
vs upstream in clauses.c, always simplify ArrayExprs to Consts, when
possible. To compensate, so that ORCA still sees ArrayExprs rather than
array Consts (in those cases where it matters), when a ScalarArrayOpExpr
is handed over to ORCA, we check if the argument array is a Const, and
convert it (back) to an ArrayExpr if it is.
Signed-off-by: NJemish Patel <jpatel@pivotal.io>

Signed-off-by: NTushar Dadlani <tdadlani@pivotal.io>

Similar to the optimizer, in planner with this
change`BitmapIndexScanState` is responsible for freeing the bitmaps
during `ExecEnd` or `ExecRescan`. The parent node doesn't need to worry
about freeing the bitmaps.

This also fix the following error for planner plan(bitmapscan_ao.sql).
```
select count(1) from bmcrash b1, bmcrash b2 where b1.bitmap_col =
b2.bitmap_col or b1.bitmap_col = '999' and b1.btree_col1 = 'abcdefg999';
ERROR:  non hash bitmap (nbtree.c:572)  (seg1 slice2 127.0.0.1:25433
pid=83873)
```

`HashBitMap` is owned by `BitmapIndexScanState`(`node->bitmap`).
`BitmapIndexScanState` will free the `HashBitMap` when `ExecEnd` is
called. So it is not the responsibility of `HashStreamOpaque` to free it
which just has the reference to it.
Also, it should never try to access
the `HashBitMap` when `HashStreamOpaque` gets freed. `HashBitMap` might
have been freed even before the `HashStreamOpaque` gets freed.

This reverts commit b5d7b30f.

`HashStreamOpaque` will never own the `HashBitmap` and hence there is no
need of variable `free` in `HashStreamOpaque` to denote if it owns the
`HashBitmap` or not. Hence we don't need `tbm_create_stream_node_ref`
API at all. Given that `HashStreamOpaque` is not owning the `HashBitMap`
it should never try to access the `HashBitMap` during the free of
`HashStreamOpaque`. There is no guarantee that `HashBitmap` is valid at
that time. `ExecEndBitmapIndexScan` might have freed the `HashBitmap`
that is referenced in `HashStreamOpaque`.

* GPDB DOCS - Fix issues found when validating files in gpdb-webhelp.ditamap.

* fix for GUC gp_connections_per_thread update to GPDB 5 description.

Avoid inclusion of PTHREAD_CFLAGS where it's not needed, and skip
appending -pthread where it is needed since PTHREAD_CFLAGS already
contains -pthread.

Let gpcloud generate the config file but not provide one. Before this,
s3conf is a base64'd string, not intelligible or easily updated.

This will simplify the dependencies of running gpcloud regression tests.

This should make sreh test pass in ORCA CI.

In assure_collocation_and_order, we check if the lower plan is already
distributed on the partitioning key, and insert a Motion node if not.
That check was broken. When a query with multiple windows is planned, we
create a chain of Group+Sort stages, to compute each aggregate. To do that,
we create a synthetic subquery for each stage, and each subquery will have
its own minimal range table and target list. The sort and distribution
path keys should use expressions that refer to the original plan and
PlannerInfo that we started with, but in assure_collocation_and_order(), we
were incorrectly building path keys based on the target lists in the
intermediate stages. Those are not comparable with path keys referring to
the original target list, and as a result.

Fixes github issue #2236.

Signed-off-by: NMelanie Plageman <mplageman@pivotal.io>

This was disabled a long time ago, but even after digging into the git
history, I could not figure out why. Nothing seems to break with it, and
if there's a problem with handling composite types in ORCA, disabling
this transformation would just be plastering over it anyway, because there
are many other ways that you could end up with composite type constants
in a query.

Currently the windows clients and loaders are not packaging correctly.
This code will be used when the team responsible for the CLs is ready
to fix the issues, and integrate the changes into the full CI for gpdb.
Signed-off-by: NDavid Sharp <dsharp@pivotal.io>

Signed-off-by: NJim Doty <jdoty@pivotal.io>

- Update license paths to fix client and loader build

  The license header from gpAux/client/install/src/ was moved to the gpaddon
  repository as it was introducing a non-commercial EULA into the open source repository.
  The makefile path for the clients and loaders build has been updated to fix the build.

- Make the host used for running WiX a variable, and set it to match the
  infrastructure.

  We set up a private hosted zone in the the gpdb5-pipeline concourse
  network, and gave the wix-packaging vm the hostname:
  `wix-packaging.gpdb5-pipeline-vpc.pivotal.io`
Signed-off-by: NJingyi Mei <jmei@pivotal.io>
Signed-off-by: NDavid Sharp <dsharp@pivotal.io>

Fixes Windows clients and loaders build, which uses the default system python.

PYTHONHOME was being set to the empty string when BLD_ARCH was win32 (or
another arch without an override for PYTHONHOME).
Setting PYTHONHOME to the empty string causes python not to find its
home, resulting in an error like "distutils module not found".
Signed-off-by: NJim Doty <jdoty@pivotal.io>

This commit restores the old packaging code.  A separate commit will
update the code so that it runs on the the current infrastructure.

[#139296853](https://www.pivotaltracker.com/story/show/139296853)
Signed-off-by: NDavid Sharp <dsharp@pivotal.io>

Upstream commit fc1adce4 (and follow-up commit 9e6dc137) tightened up
pg_get_expr() so that you cannot pass an arbitrary string as the argument.
It must come from one of the few system catalogs that actually store
serialized expressions.

GPDB has one more catalog table that stores expression, pg_partition_rule,
so list the appropriate pg_partition_rule columns as allowed exceptions,
too.

Fixes github issue #2000, reported by @jonasbu11

Only normal xids can have distributed xids, so no point trying to check
distributed log and lookout for its corresponding distributed xid. Not sure how
much performance this impacts but its more important to not go chasing down
non-existent file, corresponding to frozen xids. Though currently for
distributed log it treats that as local transaction but just lot of extra work
to get that answer.

When specifying exclusion items to pg_basebackup we need to quote
them properly to avoid the risk of breaking out of the SCONST rule
in the lexer with ./pg_basebackup -E "foo'NOWAIT EXCLUDE'bar" or
something equally silly.

1.Change the format of 'total_queue_duration' in 'gp_toolkit.gp_resgroup_status' to keep
it consistent with 'rsgqueueduration' in 'pg_stat_activity'.
2.Show resource group information for running queries in 'pg_stat_activity'.

Signed-off-by: Richard Guo<riguo@pivotal.io>

The 'sreh' test was disabled in the regression suite, looks useful,
re-enable it.

GPDB 5 removed the error table feature, rewrite these cases to use
gp_read_error_log().
Signed-off-by: NPeifeng Qiu <pqiu@pivotal.io>

It's good, but triggers an assertion with ORCA enabled, safe to ignore.

> FATAL:  Unexpected internal error (transform.c:54)
> DETAIL:  FailedAssertion("!(equal(qcopy, query) && "Preprocessing should not modify original query object")", File: "transform.c", Line: 54)
> ...
> connection to server was lost

Objective of the test is only to check if DDL/DML on partition tables lock only
the root table on QD.  Child partitions are locked on QE as needed.  The
intermittent failures are due to more locks acquired than expected.  In all
such cases, the additional locks were AccessShare locks on some catalog table.
This commit filters out locks on catalog tables from expected output.

The intermittent failures merit investigation but that can happen regardless of
this test fix.

The coverity build was failing on the gpperfmon `make distclean` step

Fixed on both sides:
- Don't call `make distclean` in the coverity build script -- not
  necessary
- Don't clean gpperfmon from `gpAux/Makefile`: this Makefile's clean
  targets are for enterprise-build related artifacts. General developers
  should be running `make clean distclean` from the top level directory of
  this repository.

Additional fix -- doesn't address the coverity red but a similar
cleanup:
- Add cleanup of gpperfmon to the top level makefile's `distclean`
  target. Not sure why it wasn't there; probably just an accidental
  omission.
- Add note explaining why the top level makefile has to circumvent
  `gpAux/Makefile`
Signed-off-by: NMelanie Plageman <mplageman@pivotal.io>

This fixes a bug introduced in e378d84b
where the partition selector may go in an infinite loop because it never
got a chance to actually _select_ the partitions.

Commit 7b6dc9bf was well-intentioned,
but wrong: in cases where `outerPlan` is NULL, inputSlot should be
`NULL`, but the execution of the partition selector should *NOT*
prematurely end there, instead, a static partition selection should
happen (once) before this operator was exhausted (returning NULL).

This is a pure revert commit. The `NULL` initialization will follow.

This reverts commit 7b6dc9bf.

My OCD prevented me from glossing over this one..

FileRepPrimary_IsResyncManagerOrWorker() is implicitly declared via
the MIRROREDLOCK_BUFMGR_LOCK macro unless cdb/cdbfilerepprimary.h is
included. Add to avoid warning.

This codepath triggered a misleading-indentation warning in GCC and
I can agree with that. Re-indent to make reading easier.