[ci skip]
Signed-off-by: NJemish Patel <jpatel@pivotal.io>

[ci skip]

Todd broke our CI in greenplum-db/gpdb@398534a9.

[ci skip]

* Make sure intent of the traceflags is clear
* Remove double negation where possible
* Update comments

Signed-off-by: NEkta Khanna <ekhanna@pivotal.io>

For a query using a correlated subselect as predicate, such as:

```
CREATE TABLE partitioned_table (a int, pk int) DISTRIBUTED BY (a)
	PARTITION BY range(pk) (end(5), end(10));
CREATE TABLE other_table (c int, d int) DISTRIBUTED BY (c);
INSERT INTO partitioned_table VALUES (1, 1), (2, 4), (3, 9);
ANALYZE ROOTPARTITION partitioned_table;

EXPLAIN SELECT pk from partitioned_table WHERE a > (SELECT d FROM other_table WHERE c = a) AND pk < 12;
```

ORCA will generate a Correlated Nested Loop Left Outer Join, which
should translate to a DXL Scan under a DXL SubPlan filter. However, the
translation happened in the following order:

0. Translate outer child (which has a filter) of correlated NLJ
0. Build a DXL SubPlan using the inner child, which is intended to serve as an
   "additional filter condition" on top of the outer child
0. Now based on the DXL plan for the outer child, we decide whether or not to
   use the additional condition (generated in #2 or #3) as a filter in the
   final result.
0. If the outer child is a Physical Sequence, we discarded the condition
   assuming that filter condition is already present in the partition
   selector.
0. This code to discard the subplan was added in
   e99325cc because, previously, we were always
   inserting additional filter as "the second child" based on the assumption
   that every DXL node has a filter child in the 2nd place. As it turned out,
   the DXL Sequence node is one counterexample: it has no filter, and it's
   second child is expected to be a partition selector.
0. We didn't catch this error in e99325cc because the test cases had a trivial
   additional filter condition of a constant true, so dropping it didn't really
   raise any eyebrows. However, the generally correct approach should be to
   retain this additional condition, either as an additional DXL Result node on
   top of the DXL for the outer child, or for appropriate types of nodes,
   inline the condition into existing filters.

This patch set fixes that.
Signed-off-by: NJemish Patel <jpatel@pivotal.io>
Signed-off-by: NEkta Khanna <ekhanna@pivotal.io>

We ensure all three cases of PdxlnCorrelatedNLJoin take ownership of the
DXLProperties object

0. There are cases where the additional scalar condition cannot be combined
   with the original condition in the DXL plan. Case in point: when the outer
   child gets translated into a DXL Sequence, we cannot put the "combined
   condition" into the sequence.
0. Deferring the combination of conditions also gives us an optimization
   opportunity to reduce double translations.

installcheck has over the last year gotten slowly bloated. This test
runs about 32 minutes when nothing else happens in Concourse. Given we
also run the planner ICG in parallel, we'd better err on the safe side
and extend this to an hour.

[ci skip]

Signed-off-by: NJemish Patel <jpatel@pivotal.io>

If an error occurs while serializing an exception's error context, don't
recurse into Serialize. Serializing the context is likely to just fail
again, leading to infinite recursion.

Also disable abort-signal while serializing error context, to also avoid
recursing.

If an error occurs, a worker is no longer executing the given task. This
was causing problems later, when the Task object had already been
destroyed, but m_ptsk was left dangling.

The hash table iterator holds a spinlock on the hash table, and there's
a GPOS_ASSERT_NO_SPINLOCK in PvMalloc. Writing to the dumper stream can
cause allocations, so avoid doing that while iterating.

This isn't strictly necessary, but considerably speeds up the dumping of
large queries. This brought down the time needed to process and dump a
600 MB minidump from about 90 s to 30 s on my laptop.

Instead of having a fixed-size buffer to serialize minidumps to, refactor
the serialization functions to write to a stream. That simplifies the
serialization functions, as they no longer need to reserve space in the
buffer ahead of time (i.e. the UlpRequiredSpace() functions are gone),
reduces memory usage when dumping small queries, and makes it possible to
minidump large queries without running out of memory.

This removes the arbitrary 16 MB limit on minidump size. I'm not sure it's
a good idea to create multi-gigabyte minidumps in practice, but that's a
case of "if it hurts, don't do it". At least it's now possible, if you need
to.

Signed-off-by: NJemish Patel <jpatel@pivotal.io>

Signed-off-by: NOmer Arap <oarap@pivotal.io>

This commit provides better implementation of `CHashSet`. The improvements
follows the same logic as `CHashMap` and provided a new iterator called
`CHashSetIter`. Even though `CHashSet` exists in the code base, there
was no real usage in the Orca codebase.

Signed-off-by: NDhanashree Kashid <dkashid@pivotal.io>

While constructing the propagation expression in case of queries
involving default list partitions which have indexes, we ended
up with NULLs crashing the database.

Currently, we fallback to planner. But we need to handle it in a
better way in future.
Signed-off-by: NDhanashree Kashid <dkashid@pivotal.io>

Signed-off-by: NBhunvesh Chaudhary <bchaudhary@pivotal.io>

With improvement in the expression de duplication algorithm, we now
also retain the order of expression due to which the test cases required
the order expression to be fixed.

We see a lot of plan diffs and plan size changes and the reason for that
is the we always preserve the order of deduped list.

Since we output different deduped list the plan changes are expected
specifically the way that we hash join and hash distribute.
Signed-off-by: NBhunvesh Chaudhary <bchaudhary@pivotal.io>

This commit introduces a performance improvement and stability for
deduplication process of passed expression list.

Earlier the running time complexity of this operation was O(n^2). This
commit reduces the running time to O(n) using a HashMap.

In addition, previously the order of expression is not preserved, as we
used to add the last entry for a expr in the output list, however now
we insert the first entry which keeps the order as is.

E.g. Let's assume everything in the below list is an expression:

A = `a,a,b,b,c,c`
If you run `dedup(A)` It will produce B=`a,b,c`.

Then if you add another `a` to the list B, it will B'=`a,b,c,a`
and when we run the `dedup(B')` it will generate `b,c,a` in the old
code.

With this change, it will always generate `a,b,c`.
Signed-off-by: NBhuvnesh Chaudhary <bchaudhary@pivotal.io>