As we merge with upstream and by that keep refining the Postgres
planner, legacy planner is no longer a suitable name. This changes
all variations of the spelling (legacy planner, legacy optimizer,
legacy query optimizer etc) to say "Postgres" rather than "legacy".
Reviewed-by: NVenkatesh Raghavan <vraghavan@pivotal.io>
Reviewed-by: NDavid Yozie <dyozie@pivotal.io>
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>

This commit is part of Add Partitioned Indexes #7047.

After adding INTERNAL_AUTO, and dependencies between partitioned indexs,
many tests that assumed that we need to manually delete indexes added to
leaf partitions need updating.
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

Pull from upstream Postgres to make DefineIndex recursively create partitioned
indexes. Instead of creating an individual IndexStmt for every partition,
create indexes by recursing on the partition children.  This aligns index
creation with upstream in preparation for adding INTERNAL_AUTO relationships
between partition indexes.

 * The QD will now choose the same name for partition indexes as Postgres.
 * Update tests to reflect the partition index names changes.
 * The changes to DefineIndex are mostly cherry-picked from Postgres commit:
   8b08f7d4
 * transformIndexStmt and its callers have been aligned with Postgres
   REL9_4_STABLE
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

The Greenplum specific error handling via ereport()/elog() calls was
in need of a unification effort as some parts of the code was using a
different messaging style to others (and to upstream). This aims at
bringing many of the GPDB error calls in line with the upstream error
message writing guidelines and thus make the user experience of
Greenplum more consistent.

The main contributions of this patch are:

* errmsg() messages shall start with a lowercase letter, and not end
  with a period. errhint() and errdetail() shall be complete sentences
  starting with capital letter and ending with a period. This attempts
  to fix this on as many ereport() calls as possible, with too detailed
  errmsg() content broken up into details and hints where possible.

* Reindent ereport() calls to be more consistent with the common style
  used in upstream and most parts of Greenplum:

	ereport(ERROR,
			(errcode(<CODE>),
			 errmsg("short message describing error"),
			 errhint("Longer message as a complete sentence.")));

* Avoid breaking messages due to long lines since it makes grepping
  for error messages harder when debugging. This is also the de facto
  standard in upstream code.

* Convert a few internal error ereport() calls to elog(). There are
  no doubt more that can be converted, but the low hanging fruit has
  been dealt with. Also convert a few elog() calls which are user
  facing to ereport().

* Update the testfiles to match the new messages.

Spelling and wording is mostly left for a follow-up commit, as this was
getting big enough as it was. The most obvious cases have been handled
but there is work left to be done here.

Discussion: https://github.com/greenplum-db/gpdb/pull/6378Reviewed-by: NAshwin Agrawal <aagrawal@pivotal.io>
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

GPDB has a concept of "one-off" plans. In PostgreSQL, the planner doesn't
normally const-evaluate stable functions, but GPDB does that. Such plans
cannot be reused across invocations, so they're marked as "one-off".

Before the PostgreSQL 9.2 merge, the same "one-off" flag was used to mark
plans where query Params were aggressively const-evaluated, for similar
reasons. That mechanism was obsoleted in 9.2, however, when the concept
of "custom" and "generic" plans was introduced. The planner will now try
to create "custom" plans for the specific query parameters, without any
GPDB adde code.

The snippet in In eval_const_expressions that this removes was a no-op.
There used to be an extra condition, before the 9.2 merge, so that it
const-evaluated Params more aggressively. But that was removed as part
of the merge, and as a result, the removed if()'s condition was always
false. We don't need the more aggressive stance here, because the upstream
custom plans mechanism does the same. Move the comment explaining the
aggressive const-evaluation of stable functions to evaluate_function(),
that seems like a better place for it, anyway.

Remove the GPDB_92_MERGE_FIXME comment in BuildCachedPlan(). I'm not sure
why we had a check for boundParams there before. AFAICS, marking the plans
as one-off in eval_const_expressions_mutator (the code that was removed),
should've been enough, and the check for boundParams in plancache.c was
redundant (and sub-optimal, because some plans were marked not-for-reuse,
which could've been reused safely). In any case, there isn't anything
special about query parameters in this code anymore, it's all upstream
code, so we don't need an extra boundParams check either.

Add a test case for the const-evaluation of query Params, too. It was
passing before these changes as well, but seems good to have a test for it.

The portals_updatable test makes use of the public.bar table, which the
partition_pruning test occasionally dropped. To fix, don't fall back to
the public schema in the partition_pruning search path. Put the
temporary functions in the partition_pruning schema as well for good
measure.

Author: Asim R P <apraveen@pivotal.io>
Author: Jacob Champion <pchampion@pivotal.io>

A Concourse job just failed with a diff like this:

*** ./expected/partition_pruning.out	2017-11-16 08:38:51.463996042 +0000
--- ./results/partition_pruning.out	2017-11-16 08:38:51.691996042 +0000
***************
*** 5785,5790 ****
--- 5786,5793 ----

  -- More Equality and General Predicates ---
  drop table if exists foo;
+ ERROR:  "foo" is not a base table
+ HINT:  Use DROP EXTERNAL TABLE to remove an external table.
  create table foo(a int, b int)
  partition by list (b)
  (partition p1 values(1,3), partition p2 values(4,2), default partition other);

That's because both 'partition_pruning' and 'portals_updatable' use a test
table called 'foo', and they are run concurrently in the test scheudle. If
the above 'drop table' happens to run at just the right time, when the table
is created but not yet dropped in 'portals_updatable', you get that error.
The test table in 'partitions_pruning' is created in a different schema,
so creating two tables with the same name would actually be a problem, but
the 'drop table' sees the table created in the public schema, too.

To fix, rename the table in 'portals_updatable', and also remove the above
unnecessary 'drop table. Either one would be enough to fix the race
condition, but might as well do both.

The indexes on these tests were quite useless. In 'alter_tab_drop_col_2',
the DROP COLUMN also dropped the index, so we got no chance to use it in
any queries. In 'alter_tab_drop_col_3', ORCA doesn't know how to use the
index for the IS FALSE query. In the translation from TINC, I removed the
former case, but kept the latter one, and also added another query where
ORCA also know how to use the index.

This removes the last tests from the QP_optimizer-functional test suite,
so remove all traces of that from the Makefiles and the Concourse pipeline.

We had very similar tests in the "partition_pruning" test, even with the
same test table, so that's a natural home for this too.

I left out the "alter_tab_add_pkey" tests, because they were identical to
the "alter_tab_add_uniquekey" tests, except that the index was created for
a PRIMARY KEY rather than a UNIQUE constraint. I don't see the point of
that. Furthermore, the test queries in "alter_tab_add_pkey" didn't actually
make use of the index.

Brings in ORCA changes to support partition elimination with IDFs in the
partition predicate. Also update tests.

Don't transform large array Const into ArrayExpr for Orca (#3406)

If the number of elements in the array Const is greater than
optimizer_array_expansion_threshold, returns the original Const unmodified.
Otherwise, it will cause severe performance issue for Orca optimizer for array
with very large number of elements, e.g. 50K.

Fixes issue #3355
[#151340976]

If a prepared statement, or a cached plan for an SPI query e.g. from a
PL/pgSQL function, contains stable functions, the stable functions were
incorrectly evaluated only once at plan time, instead of on every execution
of the plan. This happened to not be a problem in queries that contain any
parameters, because in GPDB, they are re-planned on every invocation
anyway, but non-parameter queries were broken.

In the planner, before this commit, when simplifying expressions, we set
the transform_stable_funcs flag to true for every query, and evaluated all
stable functions at planning time. Change it to false, and also rename it
back to 'estimate', as it's called in the upstream. That flag was changed
back in 2010, in order to allow partition pruning to work with qual
containing stable functions, like TO_DATE. I think back then, we always
re-planned every query, so that was OK, but we do cache plans now.

To avoid regressing to worse plans, change eval_const_expressions() so that
it still does evaluate stable functions, even when the 'estimate' flag is
off. But when it does so, mark the plan as "one-off", meaning that it must
be re-planned on every execution. That gives the old, intended, behavior,
that such plans are indeed re-planned, but it still allows plans that don't
use stable functions to be cached.

This seems to fix github issue #2661. Looking at the direct dispatch code
in apply_motion(), I suspect there are more issues like this lurking there.
There's a call to planner_make_plan_constant(), modifying the target list
in place, and that happens during planning. But this at least fixes the
non-direct dispatch cases, and is a necessary step for fixing any remaining
issues.

For some reason, the query now gets planned *twice* for every invocation.
That's not ideal, but it was an existing issue for prepared statements with
parameters, already. So let's deal with that separately.

For now, ORCA don't support index with express or constraints, so it's OK
to remap logical index info with scan's relid by calling IndexScan_MapLogicalIndexInfo().
Maybe later when we should consider remap them with hard-coded 1
as in gp_get_physical_index_relid().

This reverts commit 8c62e892.

Partition pruning test in ICW is failing because of `max_stack_depth` is
too small for sles11 environment. This commit increases the limit from
`100kB` to `500kB`.

ORCA can do some optimizations - partition pruning at least - if it can
"see" into the elements of an array in a ScalarArrayOpExpr. For example, if
you have a qual like "column IN (1, 2, 3)", and the table is partitioned on
column, it can eliminate partitions that don't hold those values. The
IN-clause is converted into an ScalarArrayOpExpr, so that is really
equivalent to "column = ANY <array>"

However, ORCA doesn't know how to extract elements from an array-typed
Const, so it can only do that if the array in the ScalarArrayOpExpr is
an ArrayExpr. Normally, eval_const_expressions() simplifies any ArrayExprs
into Const, if all the elements are constants, but we had disabled that
when ORCA was used, to keep the ArrayExprs visible to it.

There are a couple of reasons why that was not a very good solution. First,
while we refrain from converting an ArrayExpr to an array Const, it doesn't
help if the argument was an array Const to begin with. The "x IN (1,2,3)"
construct is converted to an ArrayExpr by the parser, but we would miss the
opportunity if it's written as "x = ANY ('{1,2,3}'::int[])" instead.
Secondly, by not simplifying the ArrayExpr, we miss the opportunity to
simplify the expression further. For example, if you have a qual like
"1 IN (1,2)", we can evaluate that completely at plan time to 'true', but
we would not do that with ORCA because the ArrayExpr was not simplified.

To be able to also optimize those cases, and to slighty reduce our diff
vs upstream in clauses.c, always simplify ArrayExprs to Consts, when
possible. To compensate, so that ORCA still sees ArrayExprs rather than
array Consts (in those cases where it matters), when a ScalarArrayOpExpr
is handed over to ORCA, we check if the argument array is a Const, and
convert it (back) to an ArrayExpr if it is.
Signed-off-by: NJemish Patel <jpatel@pivotal.io>

GPDB supports range and list partitions. Range partitions are represented as a set of rules. Each rule defines the boundaries of a part. E.g., a rule might say that a part contains all values between (0, 5], where left bound is 0 exclusive, but the right bound is 5, inclusive. List partitions are defined by a list of values that the part will contain. 

ORCA uses the above rule definition to generate expressions that determine which partitions need to be scanned. These expressions are of the following types:

1. Equality predicate as in PartitionSelectorState->levelEqExpressions: If we have a simple equality on partitioning key (e.g., part_key = 1).

2. General predicate as in PartitionSelectorState->levelExpressions: If we need more complex composition, including non-equality such as part_key > 1.

Note:  We also have residual predicate, which the optimizer currently doesn't use. We are planning to remove this dead code soon.

Prior to  this PR, ORCA was treating both range and list partitions as range partitions. This meant that each list part will be converted to a set of list values and each of these values will become a single point range partition.

E.g., consider the DDL:

```sql
CREATE TABLE DATE_PARTS (id int, year int, month int, day int, region text)
DISTRIBUTED BY (id)
PARTITION BY RANGE (year)
    SUBPARTITION BY LIST (month)
       SUBPARTITION TEMPLATE (
        SUBPARTITION Q1 VALUES (1, 2, 3), 
        SUBPARTITION Q2 VALUES (4 ,5 ,6),
        SUBPARTITION Q3 VALUES (7, 8, 9),
        SUBPARTITION Q4 VALUES (10, 11, 12),
        DEFAULT SUBPARTITION other_months )
( START (2002) END (2012) EVERY (1), 
  DEFAULT PARTITION outlying_years );
```

Here we partition the months as list partition using quarters. So, each of the list part will contain three months. Now consider a query on this table:

```sql
select * from DATE_PARTS where month between 1 and 3;
```

Prior to this ORCA generated plan would consider each value of the Q1 as a separate range part with just one point range. I.e., we will have 3 virtual parts to evaluate for just one Q1: [1], [2], [3]. This approach is inefficient. The problem is further exacerbated when we have multi-level partitioning. Consider the list part of the above example. We have only 4 rules for 4 different quarters, but we will have 12 different virtual rule (aka constraints). For each such constraint, we will then evaluate the entire subtree of partitions.

After this PR, we no longer decompose rules into constraints for list parts and then derive single point virtual range partitions based on those constraints. Rather, the new ORCA changes will use ScalarArrayOp to express selectivity on a list of values. So, the expression for the above SQL will look like 1 <= ANY {month_part} AND 3 >= ANY {month_part}, where month_part will be substituted at runtime with different list of values for each of quarterly partitions. We will end up evaluating that expressions 4 times with the following list of values:

Q1: 1 <= ANY {1,2,3} AND 3 >= ANY {1,2,3}
Q2: 1 <= ANY {4,5,6} AND 3 >= ANY {4,5,6}
...

Compare this to the previous approach, where we will end up evaluating 12 different expressions, each time for a single point value:

First constraint of Q1: 1 <= 1 AND 3 >= 1
Second constraint of Q1: 1 <= 2 AND 3 >= 2
Third constraint of Q1: 1 <= 3 AND 3 >= 3
First constraint of Q2: 1 <= 4 AND 3 >= 4
...

The ScalarArrayOp depends on a new type of expression PartListRuleExpr that can convert a list rule to an array of values. ORCA specific changes can be found here: https://github.com/greenplum-db/gporca/pull/149

These were not that exciting tests, because ORCA falls back to the Postgres
planner for these queries. But it's a reasonable test case of partition
pruning in general, after adding a WHERE clauses to the query, so add one
such test case to the main test suite.

The original TINC tests where these were copied for tested specifically
that ORCA can use Dynamic Index Scans for these queries. To be closer to
that spirit, even when running without ORCA, try to use index scans when
possible. It exercises more codepaths in the planner anyway, and allows
us to notice more easily if the planner loses the ability to use an index
for some reason.

The old test script had some special logic in it. It ran the test scripts
as usual, but it also did EXPLAIN on each query, and checked that the
EXPLAIN output contained "Dynamic Index Scan" (except for a few queries
that were marked with the "@negtest True" tag). I didn't copy over that
approach; instead, I added added explicit EXPLAINs to each query, and let
gpdiff.pl to check that the plan matches that in the expected output.
That's slightly less fragile, the tests will now fail if there are some
other changes to the plans, even if they still use a Dynamic Index Scans.
I think that's quite reasonable, and shouldn't cause too many false
positives. These queries are fairly simple, so there aren't very many
possible plans for them anyway. Worst case is that we'll have to update
the expected output every now and then, when there are changes to the
planners.

The original tests were run with:
  SELECT disable_xform('CXformDynamicGet2DynamicTableScan')

but I left that out of the new tests. With Orca and
CXformDynamicGet2DynamicTableScan, some test queries fell back the Postgres
planner. That didn't seem very interesting. I'm guessing that the point
of CXformDynamicGet2DynamicTableScan was to force the queries to use an
(Dynamic) Index scan, rather than a Table Scan, but Orca seems to choose
Index Scans for all the quries even without it.

Some test queries were tagged with "@negtest True", but nevertheless used
Dynamic Index Scans, so they would've passed the original Dynamic Index
Scan checks too, if they hadn't been tagged. Perhaps Orca has improved
since the tests were written?

I moved them to the 'partition_pruning_with_fn' test, but because there is
no functions involved in these tests, I renamed the test to just
'partition_pruning'.

There is possibly some overlap with these new tests and the existing ones
in the same file, but will let future test archeologists to decide that.
These are cheap tests, in any case.

The int -> text cast (and many other casts to text) were downgraded
to assignment casts in the upstream, and we're about to merge that patch
soon. To prepare for that and to reduce the size of that huge merge
commit somewhat, adjust GPDB test cases to not rely on those casts that
are about to be removed.

When the casts were downgraded in the upstream, it revealed a lot of buggy
queries in applications (or put another way, it broke a lot of applications
:-) ). Many of the queries in our regression tests look bogus like that
as well, for example the "sale.dt < 10" comparison in the qp_olap_mdqa test
(sale.dt is a date). But I didn't try to change the meaning of the queries,
I just added the casts needed to make them do the same thing they used to.
I'm afraid that changing the substance of the queries would reduce our
code coverage, if the tests are reproducing some very specific scenario.