functions

In copyFuncs(), the fallthrough for SpecialJoinInfo was unintentionally
done as a part of subselect merge.  This commit fixes that along with
adding support for new fields in copyFuncs(), outFunc() and equalFunc()

[Ref #151491231]
Signed-off-by: NDhanashree Kashid <dkashid@pivotal.io>

Previously this test's result and plan was wrong. Now, Orca is fixed to
produce a valid plan along with the correct result.

Bump Orca version to 2.46.3

The glob syntax used to validate filenames for segment dump filenames
in the "old" backup format is only correct for single-digit dbids, as
it assumes the use of regex syntax that glob doesn't actually support.

This commit modifies get_filename_for_content to glob for all dump
files present and then explicitly match the expected filenames with a
regular expression.

It also modifies generate_filename to account for files created on a
standby master, which was not correctly handled previously.

Some environment variables passed through the pipeline YAML were not
being passed to the test run (e.g. WITH_MIRRORS=false). Removing the
full login flag of su fixes the issue.

This should speed up the tests a bit, since they are currently spending a significant amount of time in createdb statements.

* Enable PXF by default for GPDB builds

* Disable pxf wherever libcurl is disabled

- instead of trying to keep GPDB readme in sync with GPORCA readme, just refer to it

- fixes cd path
- removes macOS warning that is not currently an issue
- treats comment like other readme comments

After 7e268107 started seeing warnings like
------------------
cdbgroup.c:1478:68: warning: expression which evaluates to zero treated as a null pointer constant of type 'List *' (aka 'struct List *') [-Wnon-literal-null-conversion]
                result_plan = (Plan*)make_motion_gather_to_QE(root, result_plan, false);
                                                                                 ^~~~~
------------------

After commit 1822c826 started seeing bunch of
warnings like
-------------------------------------------
memaccounting.c:247:1: warning: no previous prototype for function 'MemoryAccounting_DeclareDone' [-Wmissing-prototypes]
MemoryAccounting_DeclareDone()
^
../../../../src/include/utils/memaccounting.h:238:1: note: this declaration is not a prototype; add 'void' to make it a prototype for a zero-parameter function
MemoryAccounting_DeclareDone();
^
                             void
memaccounting.c:263:1: warning: no previous prototype for function 'MemoryAccounting_RequestQuotaIncrease' [-Wmissing-prototypes]
MemoryAccounting_RequestQuotaIncrease()
^
../../../../src/include/utils/memaccounting.h:241:1: note: this declaration is not a prototype; add 'void' to make it a prototype for a zero-parameter function
MemoryAccounting_RequestQuotaIncrease();
^
                                      void
2 warnings generated.
ext_alloc.c:35:1: warning: no previous prototype for function 'Ext_OptimizerAlloc' [-Wmissing-prototypes]
Ext_OptimizerAlloc(size_t size)
^
ext_alloc.c:50:17: warning: unused variable 'account' [-Wunused-variable]
        MemoryAccount *account = MemoryAccounting_ConvertIdToAccount(ActiveMemoryAccountId);
                       ^
ext_alloc.c:48:1: warning: no previous prototype for function 'Ext_OptimizerFree' [-Wmissing-prototypes]
Ext_OptimizerFree(void *ptr)
^
ext_alloc.c:59:1: warning: no previous prototype for function 'GetOptimizerOutstandingMemoryBalance' [-Wmissing-prototypes]
GetOptimizerOutstandingMemoryBalance()
^
-------------------------------------------

As part of commit efed2fcc new walrep state was
added 'n' (not-in-sync). The toolkit function gp_pgdatabase__() needs to be
modified as well to check for that state.

Original code not using #defines but direct characters like 's', 'c' makes it
very tricky find stuff in code. Hopefully, future chnages would be easy to spot
and make.

QE detach resource group slot at the end of transaction, the
last QE of the slot release the slot, and release the overused
memory if resource group config has been changed.

commit 07b9936a
Author: Tom Lane <tgl@sss.pgh.pa.us>
Date:   Fri Feb 27 23:30:29 2009 +0000

    Temporarily (I hope) disable flattening of IN/EXISTS sublinks that are within
    the ON clause of an outer join.  Doing so is semantically correct but results
    in de-optimizing queries that were structured to take advantage of the sublink
    style of execution, as seen in recent complaint from Kevin Grittner.  Since
    the user can get the other behavior by reorganizing his query, having the
    flattening happen automatically is just a convenience, and that doesn't
    justify breaking existing applications.  Eventually it would be nice to
    re-enable this, but that seems to require a significantly different approach
    to outer joins in the executor.

Added relevant test case.
Signed-off-by: NDhanashree Kashid <dkashid@pivotal.io>

Author: Tom Lane <tgl@sss.pgh.pa.us>
Date:   Tue Jul 8 14:03:32 2014 -0400

    While the x output of "select x from t group by x" can be presumed unique,
    this does not hold for "select x, generate_series(1,10) from t group by x",
    because we may expand the set-returning function after the grouping step.
    (Perhaps that should be re-thought; but considering all the other oddities
    involved with SRFs in targetlists, it seems unlikely we'll change it.)
    Put a check in query_is_distinct_for() so it's not fooled by such cases.

    Back-patch to all supported branches.

    David Rowley

(cherry picked from commit 2e7469dc8b3bac4fe0f9bd042aaf802132efde85)

Signed-off-by: NJemish Patel <jpatel@pivotal.io>

Author: Tom Lane <tgl@sss.pgh.pa.us>
Date:   Tue Dec 10 16:10:36 2013 -0500

An expression such as WHERE (... x IN (SELECT ...) ...) IN (SELECT ...)
could produce an invalid plan that results in a crash at execution time,
if the planner attempts to flatten the outer IN into a semi-join.
This happens because convert_testexpr() was not expecting any nested
SubLinks and would wrongly replace any PARAM_SUBLINK Params belonging
to the inner SubLink.  (I think the comment denying that this case could
happen was wrong when written; it's certainly been wrong for quite a long
time, since very early versions of the semijoin flattening logic.)

Per report from Teodor Sigaev.  Back-patch to all supported branches.

(cherry picked from commit 884c6384a2db34f6a65573e6bfd4b71dfba0de90)

commit 0a0ca1cb18a34e92ab549df171e174dcce7bf7a3
Author: Tom Lane <tgl@sss.pgh.pa.us>
Date:   Sat Mar 24 16:22:00 2012 -0400

    Fix planner's handling of outer PlaceHolderVars within subqueries.

    For some reason, in the original coding of the PlaceHolderVar mechanism
    I had supposed that PlaceHolderVars couldn't propagate into subqueries.
    That is of course entirely possible.  When it happens, we need to treat
    an outer-level PlaceHolderVar much like an outer Var or Aggref, that is
    SS_replace_correlation_vars() needs to replace the PlaceHolderVar with
    a Param, and then when building the finished SubPlan we have to provide
    the PlaceHolderVar expression as an actual parameter for the SubPlan.
    The handling of the contained expression is a bit delicate but it can be
    treated exactly like an Aggref's expression.

    In addition to the missing logic in subselect.c, prepjointree.c was failing
    to search subqueries for PlaceHolderVars that need their relids adjusted
    during subquery pullup.  It looks like everyplace else that touches
    PlaceHolderVars got it right, though.

    Per report from Mark Murawski.  In 9.1 and HEAD, queries affected by this
    oversight would fail with "ERROR: Upper-level PlaceHolderVar found where
    not expected".  But in 9.0 and 8.4, you'd silently get possibly-wrong
    answers, since the value transmitted into the subquery wouldn't go to null
    when it should.

GPDB handles a lot of the cases that are restricted by
is_simple_subquery; and the restrictions not handled, are checked for
separately in convert_EXISTS_sublink_to_join().

Resulting from cascading ICG failures, we also fixed the following:

- initialize all the members of IncrementVarSublevelsUp_context
  properly.
- remove incorrect assertions brought in from upstream. In GPDB, these
  cases are handled.
- improve plans for NOT EXISTS sub-queries containing an aggregation
  without limits by creating a "false" one-time filter.
Signed-off-by: NDhanashree Kashid <dkashid@pivotal.io>

This was used to keep information about the subquery join tree for
pulled-up sublinks for use later in deconstruct_recurse().  With the
upstream subselect merge, a JoinExpr constructed at the pull-up time
itself, so this is no longer needed since the subquery join tree
information is available in the constructed JoinExpr.

Also with the merge, deconstruct_recurse() handles JOIN_SEMI JoinExprs.
However, since GPDB differs from upstream by treating SEMI joins as
INNER join for internal join planning, this commit also updates
inner_join_rels correctly for SEMI joins (see regression test).

Also remove unused function declaration for not_null_inner_vars().
Signed-off-by: NDhanashree Kashid <dkashid@pivotal.io>

This issue was discovered during the subselect merge. wherin planner
incorrectly commutes anti joins.
`cdb_add_subquery_join_paths()` creates join paths for (rel1, rel2) and
(rel2, rel1) for all join types including JOIN_ANTI and JOIN_LASJ_NOTIN.
This produces wrong results since these joins are order-sensitive w.r.t
inner and outer relations (see new regression tests). So, do not add
(rel2, rel1) for JOIN_ANTI and JOIN_LASJ_NOTIN.

This commit also refactors cdb_add_subquery_join_paths() and
make_join_rel() to make it easier to control the commuting.
Signed-off-by: NDhanashree Kashid <dkashid@pivotal.io>

1. convert_IN_to_antijoin() should fail pull-up when left relids are not
   a subset of available_rels, otherwise we get wrong results. See
   regression tests in qp_correlated_query.sql.
2. convert_EXPR_to_join() is a GPDB-only function that already handles
   this case via ProcessSubqueryToJoin().
Signed-off-by: NDhanashree Kashid <dkashid@pivotal.io>

commit 0dec3226ee905f94d0b9d6e2f274e13bbcaf5370
Author: Tom Lane <tgl@sss.pgh.pa.us>
Date:   Mon Jun 20 14:33:20 2011 -0400

    Fix thinko in previous patch for optimizing EXISTS-within-EXISTS.

    When recursing after an optimization in pull_up_sublinks_qual_recurse, the
    available_rels value passed down must include only the relations that are
    in the righthand side of the new SEMI or ANTI join; it's incorrect to pull
    up a sub-select that refers to other relations, as seen in the added test
    case.  Per report from BangarRaju Vadapalli.

NOTE: The second part of the upstream commit is not pulled in because that
produces inferior plans in GPDB by not pulling nested sublinks below NOT
EXISTS. That part is reverted later upstream in 9.2 anyway.

Also update regression tests.
Signed-off-by: NDhanashree Kashid <dkashid@pivotal.io>

commit f3f0f37068e06d01e88abbf3ed596664b139f7e2
Author: Tom Lane <tgl@sss.pgh.pa.us>
Date:   Mon May 2 15:56:47 2011 -0400

    Fix pull_up_sublinks' failure to handle nested pull-up opportunities.

    After finding an EXISTS or ANY sub-select that can be converted to a
    semi-join or anti-join, we should recurse into the body of the sub-select.
    This allows cases such as EXISTS-within-EXISTS to be optimized properly.
    The original coding would leave the lower sub-select as a SubLink, which
    is no better and often worse than what we can do with a join.  Per example
    from Wayne Conrad.

    Back-patch to 8.4.  There is a related issue in older versions' handling
    of pull_up_IN_clauses, but they're lame enough anyway about the whole area
    that it seems not worth the extra work to try to fix.
Signed-off-by: NDhanashree Kashid <dkashid@pivotal.io>

commit c4ac2ff765d9b68a3ff2a3461804489721770d06
Author: Tom Lane <tgl@sss.pgh.pa.us>
Date:   Mon Jun 21 00:14:54 2010 +0000

    Fix mishandling of whole-row Vars referencing a view or sub-select.
    If such a Var appeared within a nested sub-select, we failed to translate it
    correctly during pullup of the view, because the recursive call to
    replace_rte_variables_mutator was looking for the wrong sublevels_up value.
    Bug was introduced during the addition of the PlaceHolderVar mechanism.
    Per bug #5514 from Marcos Castedo.

commit dcd647d7cf98e3393f919135f6e113e896781f60
Author: Tom Lane <tgl@sss.pgh.pa.us>
Date:   Mon Jan 18 18:17:52 2010 +0000

    Fix an oversight in convert_EXISTS_sublink_to_join: we can't convert an
    EXISTS that contains a WITH clause.  This would usually lead to a
    "could not find CTE" error later in planning, because the WITH wouldn't
    get processed at all.  Noted while playing with an example from Ken Marshall.

  commit e5536e77
  Author: Tom Lane <tgl@sss.pgh.pa.us>
  Date:   Mon Aug 25 22:42:34 2008 +0000

      Move exprType(), exprTypmod(), expression_tree_walker(), and related routines
      into nodes/nodeFuncs, so as to reduce wanton cross-subsystem #includes inside
      the backend.  There's probably more that should be done along this line,
      but this is a start anyway
Signed-off-by: NShreedhar Hardikar <shardikar@pivotal.io>

commit fbcce080
Author: Tom Lane <tgl@sss.pgh.pa.us>
Date:   Sun Apr 5 19:59:40 2009 +0000

    Change EXPLAIN output so that subplans and initplans (particularly CTEs)
    are individually labeled, rather than just grouped under an "InitPlan"
    or "SubPlan" heading.  This in turn makes it possible for decompilation of
    a subplan reference to usefully identify which subplan it's referencing.
    I also made InitPlans identify which parameter symbol(s) they compute,
    so that references to those parameters elsewhere in the plan tree can
    be connected to the initplan that will be executed.  Per a gripe from
    Robert Haas about EXPLAIN output of a WITH query being inadequate,
    plus some longstanding pet peeves of my own.

commit 173a6760
Author: Tom Lane <tgl@sss.pgh.pa.us>
Date:   Mon Dec 8 00:16:09 2008 +0000

    Don't try to optimize EXISTS subqueries with empty FROM-lists: we need to
    form a join and that case doesn't have anything to join to.  (We could
    probably make it work if we didn't pull up the subquery, but it seems to
    me that the case isn't worth extra code.)  Per report from Greg Stark.

We had a bunch of fixmes that we added as part of the subselect merge;
All of the fixmes are now marked as `GPDB_84_MERGE_FIXME` so that they can
be grepped easily.

After the introduction of `placeholder` mechanism,
we see some changes in the plan.
These are all good changes in that we remove the
redundant `NOTIN_subquery` subquery scan nodes:

Before
```
...
->  Broadcast Motion 3:3  (slice1; segments: 3)
	->  Subquery Scan "NotIn_SUBQUERY"
		->  Seq Scan on subselect_tbl
			Filter: f3 IS NOT NULL
```
After
```
...
->  Broadcast Motion 3:3  (slice1; segments: 3)
	 ->  Seq Scan on subselect_tbl
	 	 Filter: f3 IS NOT NULL
```

New plans have better cost since we will not be executing
subquery scan for each tuple coming from its OUTER.

We are now consistent with plans produced for same queries
with IN instead of NOT IN.
Signed-off-by: NEkta Khanna <ekhanna@pivotal.io>

commit dc9cc887b74bfa0d40829c4df66dead509fdd8f6
Author: Tom Lane <tgl@sss.pgh.pa.us>
Date:   Tue Sep 28 14:15:42 2010 -0400

    The point of a PlaceHolderVar is to allow a non-strict expression to be
    evaluated below an outer join, after which its value bubbles up like a Var
    and can be forced to NULL when the outer join's semantics require that.
    However, there was a serious design oversight in that, namely that we
    didn't ensure that there was actually a correct place in the plan tree
    to evaluate the placeholder :-(.  It may be necessary to delay evaluation
    of an outer join to ensure that a placeholder that should be evaluated
    below the join can be evaluated there.  Per recent bug report from Kirill
    Simonov.

    Back-patch to 8.4 where the PlaceHolderVar mechanism was introduced.
Signed-off-by: NEkta Khanna <ekhanna@pivotal.io>

commit 2bdd765f79df947b46a8b5a22e3b993b58cd6d32
Author: Tom Lane <tgl@sss.pgh.pa.us>
Date:   Wed Sep 2 17:52:33 2009 +0000

    Fix subquery pullup to wrap a PlaceHolderVar around the entire RowExpr
    that's generated for a whole-row Var referencing the subquery, when the
    subquery is in the nullable side of an outer join.  The previous coding
    instead put PlaceHolderVars around the elements of the RowExpr.  The effect
    was that when the outer join made the subquery outputs go to null, the
    whole-row Var produced ROW(NULL,NULL,...) rather than just NULL.  There
    are arguments afoot about whether those things ought to be semantically
    indistinguishable, but for the moment they are not entirely so, and the
    planner needs to take care that its machinations preserve the difference.
    Per bug #5025.

    Making this feasible required refactoring ResolveNew() to allow more caller
    control over what is substituted for a Var.  I chose to make ResolveNew()
    a wrapper around a new general-purpose function replace_rte_variables().
    I also fixed the ancient bogosity that ResolveNew might fail to set
    a query's hasSubLinks field after inserting a SubLink in it.  Although
    all current callers make sure that happens anyway, we've had bugs of that
    sort before, and it seemed like a good time to install a proper solution.

    Back-patch to 8.4.  The problem can be demonstrated clear back to 8.0,
    but the fix would be too invasive in earlier branches; not to mention
    that people may be depending on the subtly-incorrect behavior.  The
    8.4 series is new enough that fixing this probably won't cause complaints,
    but it might in older branches.  Also, 8.4 shows the incorrect behavior
    in more cases than older branches do, because it is able to flatten
    subqueries in more cases.
Signed-off-by: NEkta Khanna <ekhanna@pivotal.io>

commit c59d8dd4
Author: Tom Lane <tgl@sss.pgh.pa.us>
Date:   Tue Apr 28 21:31:16 2009 +0000

    Improve pull_up_subqueries logic so that it doesn't insert unnecessary
    PlaceHolderVar nodes in join quals appearing in or below the lowest
    outer join that could null the subquery being pulled up.  This improves
    the planner's ability to recognize constant join quals, and probably
    helps with detection of common sort keys (equivalence classes) as well.
Signed-off-by: NDhanashree Kashid <dkashid@pivotal.io>

commit 31468d05
Author: Tom Lane <tgl@sss.pgh.pa.us>
Date:   Wed Oct 22 20:17:52 2008 +0000

    Dept of better ideas: refrain from creating the planner's placeholder_list
    until vars are distributed to rels during query_planner() startup.  We don't
    really need it before that, and not building it early has some advantages.
    First, we don't need to put it through the various preprocessing steps, which
    saves some cycles and eliminates the need for a number of routines to support
    PlaceHolderInfo nodes at all.  Second, this means one less unused plan for any
    sub-SELECT appearing in a placeholder's expression, since we don't build
    placeholder_list until after sublink expansion is complete.
Signed-off-by: NDhanashree Kashid <dkashid@pivotal.io>

commit e6ae3b5d
Author: Tom Lane <tgl@sss.pgh.pa.us>
Date:   Tue Oct 21 20:42:53 2008 +0000

    Add a concept of "placeholder" variables to the planner.  These are variables
    that represent some expression that we desire to compute below the top level
    of the plan, and then let that value "bubble up" as though it were a plain
    Var (ie, a column value).

    The immediate application is to allow sub-selects to be flattened even when
    they are below an outer join and have non-nullable output expressions.
    Formerly we couldn't flatten because such an expression wouldn't properly
    go to NULL when evaluated above the outer join.  Now, we wrap it in a
    PlaceHolderVar and arrange for the actual evaluation to occur below the outer
    join.  When the resulting Var bubbles up through the join, it will be set to
    NULL if necessary, yielding the correct results.  This fixes a planner
    limitation that's existed since 7.1.

    In future we might want to use this mechanism to re-introduce some form of
    Hellerstein's "expensive functions" optimization, ie place the evaluation of
    an expensive function at the most suitable point in the plan tree.
Signed-off-by: NEkta Khanna <ekhanna@pivotal.io>