This was removed in upstream in c970292a,
and is one step forward to make a number of counters 64 bit safe.

In GPDB the dispatcher dispatches the entire plan tree to each query executor (QX). Each QX deserializes the entire plan tree and starts execution from the root of the plan tree. This begins by calling InitPlan on the QueryDesc, which blindly calls ExecInitNode on the root of the plan.

Unfortunately, this is wasteful, in terms of memory and CPU. Each QX is in charge of a single slice. There can be many slices. Looking into plan nodes that belong to other slices, and initializing (e.g., creating PlanState for such nodes) is clearly wasteful. For large plans, particularly planner plans, in the presence of partitions, this can add up to a significant waste.

This PR proposes a fix to solve this problem. The idea is to find the local root for each slice and start ExecInitNode there.

There are few special cases:

SubPlans are special, as they appear as expression but the expression holds the root of the sub plan tree. All the subplans are bundled in the plannedstmt->subplans, but confusingly as Plan pointers (i.e., we save the root of the SubPlan expression's Plan tree). Therefore, to find the relevant sub plans, we need to first find the relevant expressions and extract their roots and then iterate the plannedstmt->subplans, but only ExecInitNode on the ones that we can reach from some expressions in current slice.

InitPlan are no better as they can appear anywhere in the Plan tree. Walking from a local motion is not sufficient to find these InitPlan. Therefore, we need to walk from the root of the plan tree and identify all the SubPlan. Note: unlike regular subplan, the initplan may not appear in the expression as subplan; rather it will appear as a parameter generator in some other parts of the tree. We need to find these InitPlan and obtain the SubPlan for each InitPlan. We can then use the SubPlan's setParam to copy precomputed parameter values from estate->es_param_list_info to estate->es_param_exec_vals

We also found that the origSliceIdInPlan is highly unreliable and cannot be used as an indicator of a plan node's slice information. Therefore, we precompute each plan node's slice information to correctly determine if a Plan node is alien or not. This makes alien node identification more accurate. In successive PRs, we plan to use the alien memory account balance as a test to see if we successfully eliminated all aliens. We will also use the alien account balance to determine memory savings.

Under former TCP interconnect, after declaring a cursor for an invalid
query like "declare c1 cursor for select c1/0 from foo", the following FETCH
command can still fetch an empty row instead of an error. This is
incorrect and does not consist with UDP interconnect.

The RCA is senders of TCP interconnect always send an EOF message to
their peers regardless of the errors on segments, so the receivers can
not tell the difference of EOF or an error.

The solution in this commit is do not send the EOF to the peers if senders is
encountering an error and let the QD to check the whole segments status when it
can not read data from interconnect for a long time.

In commit 3a795d25, I refactored things so that it was left unused, but I
failed to actually remove it then.

DynamicTableScanInfo is an extension of EState, so always allocate it in the
same memory context. The DynamicTableScanInfo.memoryContext field always
pointed to es_query_cxt, so that is what we in fact always did anyway, this
just removes the unnecessary abstraction, for simplicity.

Signed-off-by: NKenan Yao <kyao@pivotal.io>

I found these with "callcatcher", written by Caolán McNamara. Many thanks
for the tool! See https://www.skynet.ie/~caolan/Packages/callcatcher.html

refactor CdbDispatchUtilityStatement() to make it flexible for cdbCopyStart(),
dispatchVacuum() to call directly. Introduce flags like DF_NEED_TWO_SNAPSHOT,
DF_WITH_SNAPSHOT, DF_CANCEL_ON_ERROR to make function call much clearer

The functions GetPartitionTargetlist() and UpdateGenericExprState()
are not called from anywhere so remove.

Include a map from sliceIndex to gang_id in the dispatched string,
and remove the localSlice field, hence QE should get the localSlice
from the map now. By this way, we avoid duplicating and modifying
the dispatch text string slice by slice, and each QE of a sliced
dispatch would get same contents now.

The extra space cost is sizeof(int) * SliceNumber bytes, and the extra
computing cost is iterating the SliceNumber-size array. Compared with
memcpy of text string for each slice in previous implementation, this
way is much cheaper, because SliceNumber is much smaller than the size
of dispatch text string. Also, since SliceNumber is so small, we just
use an array for the map instead of a hash table.

Also, clean up some dead code in dispatcher, including:
(1) Remove primary_gang_id field of Slice struct and DispatchCommandDtxProtocolParms
struct, since dispatch agent is deprecated now;
(2) Remove redundant logic in cdbdisp_dispatchX;
(3) Clean up buildGpDtxProtocolCommand;

Backport this patch from PostgreSQL 9.0, which replaces the tuple table
array with a linked list of individually palloc'd slots. With that, we
don't need to know the size of the array beforehand, and don't need to
count the slots. The counting was especially funky for subplans in GPDB,
and it was about to change with the upcoming PostgreSQL 8.3 merge again.
This makes it a lot simpler.

I don't plan to backport the follow-up patch to remove the ExecCountSlots
infrastructure. We'll get that later, when we merge with PostgreSQL 9.0.

commit f92e8a4b
Author: Tom Lane <tgl@sss.pgh.pa.us>
Date:   Sun Sep 27 20:09:58 2009 +0000

    Replace the array-style TupleTable data structure with a simple List of
    TupleTableSlot nodes.  This eliminates the need to count in advance
    how many Slots will be needed, which seems more than worth the small
    increase in the amount of palloc traffic during executor startup.

    The ExecCountSlots infrastructure is now all dead code, but I'll remove it
    in a separate commit for clarity.

    Per a comment from Robert Haas.

1) add one new type of gang: singleton reader gang.
2) change interface of allocateGang.
3) handling exceptions during gang creation: segment down and segment reset.
4) cleanup some dead code.

dispatcher/ directory

This commit has no logic change, it just contains movement of code across
files, to make dispatcher code clearer, and easier for unit testing.

Signed-off-by: Kenan Yao

This commit  generates code for code path: ExecVariableList > slot_getattr > _slot_getsomeattrs > slot_deform_tuple. This code path is executed during scan in simple select queries that do not have a where clause (e.g., select bar from foo;).

For each attribute A in the target list, regular implementation of ExecvariableList retrieves the slot that A comes from and calls slot_getattr. slot_get_attr() will eventually call slot_deform_tuple (through _slot_getsomeattrs), which fetches all yet unread attributes of the slot until the given attribute.

This commit generates the code for the case that all the attributes in target list use the same slot (created during a scan i.e, ecxt_scantuple). Moreover, instead of looping over the target list one at a time, it uses slot_getattr only once, with the largest attribute index from the target list.

If during code generation time, the completion is not possible (e.g., attributes use different slots), then function returns false and codegen manager will be responsible to manage the clean up.

This implementation does not support:
* Null attributes
* Variable length attributes
* Fixed length attributes passed by reference (e.g., uuid)
If at execution time, we see any of the above types of attributes, we fall back to the regular function.

Moreover, this commit:
* renames existing "codegen" guc, which is used for initiating llvm libraries, to "init_codegen" (Note: we set this guc in gpconfig),
* adds guc "codegen", which enables code generation and compilation at query execution time,
* enhances the existing code generation utilities by adding a function that creates all IR instructions for falling back to regular functions, and
* removes the existing code that generates the code of a naive slot_deform_tuple, which simply falls back to regular slot_deform_tuple.
Signed-off-by: NNikos Armenatzoglou <nikos.armenatzoglou@gmail.com>

the cleanup of EState

The jump table was missing some entries, and hence the code wouldn't have
worked correctly. Fortunately it was all dead code, as sendInitGpmonPkts()
only called itself, there were no other callers.

This fixes issue #256. Thanks to Craig Harris for the report!

Moving the installation of gp_toolkit.sql into initdb, in commit f8910c3c,
broke all the functions that are supposed to execute on all nodes, like
gp_toolkit.__gp_localid. After that change, gp_toolkit.sql was executed
in utility mode, and the gp_distribution_policy entries for those functions
were not created as a result.

To fix, change the code so that gp_distribution_policy entries are never
never created, or consulted, for EXECUTE-type external tables. They have
more fine-grained information in pg_exttable.location field anyway, so rely
on that instead. With this change, there is no difference in whether an
EXECUTE-type external table is created in utility mode or not. We would
still have similar problems if gp_toolkit contained other kinds of
external tables, but it doesn't.

This removes the isMasterOnly() function and changes all its callers to
call GpPolicyFetch() directly instead. Some places used GpPolicyFetch()
directly to check if a table is distributed, so this just makes that the
canonical way to do it. The check for system schemas that used to be in
isMasterOnly() are no longer performed, but they should've unnecessary in
the first place. System tables don't have gp_distribution_policy entries,
so they'll be treated as master-only even without that check.

This patch changes CREATE INDEX CONCURRENTLY so that the pg_index
flag changes it makes without exclusive lock on the index are made via
heap_inplace_update() rather than a normal transactional update.  The
latter is not very safe because moving the pg_index tuple could result in
concurrent SnapshotNow scans finding it twice or not at all, thus possibly
resulting in index corruption.

In addition, fix various places in the code that ought to check to make
sure that the indexes they are manipulating are valid and/or ready as
appropriate.  These represent bugs that have existed since 8.2, since
a failed CREATE INDEX CONCURRENTLY could leave a corrupt or invalid
index behind, and we ought not try to do anything that might fail with
such an index.

Also fix RelationReloadIndexInfo to ensure it copies all the pg_index
columns that are allowed to change after initial creation.  Previously we
could have been left with stale values of some fields in an index relcache
entry.  It's not clear whether this actually had any user-visible
consequences, but it's at least a bug waiting to happen.

This is a subset of a patch already applied in 9.2 and HEAD.  Back-patch
into all earlier supported branches.

Tom Lane and Andres Freund

When a whole-row Var is reading the result of a subquery, we need it to
ignore any "resjunk" columns that the subquery might have evaluated for
GROUP BY or ORDER BY purposes.  We've hacked this area before, in commit
68e40998, but that fix only covered
whole-row Vars of named composite types, not those of RECORD type; and it
was mighty klugy anyway, since it just assumed without checking that any
extra columns in the result must be resjunk.  A proper fix requires getting
hold of the subquery's targetlist so we can actually see which columns are
resjunk (whereupon we can use a JunkFilter to get rid of them).  So bite
the bullet and add some infrastructure to make that possible.

Per report from Andrew Dunstan and additional testing by Merlin Moncure.
Back-patch to all supported branches.  In 8.3, also back-patch commit
292176a1, which for some reason I had
not done at the time, but it's a prerequisite for this change.

but no database changes have been made since the last CommandCounterIncrement.
This should result in a significant improvement in the number of "commands"
that can typically be performed within a transaction before hitting the 2^32
CommandId size limit.  In particular this buys back (and more) the possible
adverse consequences of my previous patch to fix plan caching behavior.

The implementation requires tracking whether the current CommandCounter
value has been "used" to mark any tuples.  CommandCounter values stored into
snapshots are presumed not to be used for this purpose.  This requires some
small executor changes, since the executor used to conflate the curcid of
the snapshot it was using with the command ID to mark output tuples with.
Separating these concepts allows some small simplifications in executor APIs.

Something for the TODO list: look into having CommandCounterIncrement not do
AcceptInvalidationMessages.  It seems fairly bogus to be doing it there,
but exactly where to do it instead isn't clear, and I'm disinclined to mess
with asynchronous behavior during late beta.

columns, and the new version can be stored on the same heap page, we no longer
generate extra index entries for the new version.  Instead, index searches
follow the HOT-chain links to ensure they find the correct tuple version.

In addition, this patch introduces the ability to "prune" dead tuples on a
per-page basis, without having to do a complete VACUUM pass to recover space.
VACUUM is still needed to clean up dead index entries, however.

Pavan Deolasee, with help from a bunch of other people.

are not one of the query's defined result relations, but nonetheless have
triggers fired against them while the query is active.  This was formerly
impossible but can now occur because of my recent patch to fix the firing
order for RI triggers.  Caching a ResultRelInfo avoids duplicating work by
repeatedly opening and closing the same relation, and also allows EXPLAIN
ANALYZE to "see" and report on these extra triggers.  Use the same mechanism
to cache open relations when firing deferred triggers at transaction shutdown;
this replaces the former one-element-cache strategy used in that case, and
should improve performance a bit when there are deferred triggers on a number
of relations.

simplify a bit further.

ExecRelationIsTargetRelation() to check if the relation is a target
rel, rather than scanning through the result relation array ourselves.

parent query's EState.  Now that there's a single flat rangetable for both
the main plan and subplans, there's no need anymore for a separate EState,
and removing it allows cleaning up some crufty code in nodeSubplan.c and
nodeSubqueryscan.c.  Should be a tad faster too, although any difference
will probably be hard to measure.  This is the last bit of subsidiary
mop-up work from changing to a flat rangetable.

useless substructure for its RangeTblEntry nodes.  (I chose to keep using the
same struct node type and just zero out the link fields for unneeded info,
rather than making a separate ExecRangeTblEntry type --- it seemed too
fragile to have two different rangetable representations.)

Along the way, put subplans into a list in the toplevel PlannedStmt node,
and have SubPlan nodes refer to them by list index instead of direct pointers.
Vadim wanted to do that years ago, but I never understood what he was on about
until now.  It makes things a *whole* lot more robust, because we can stop
worrying about duplicate processing of subplans during expression tree
traversals.  That's been a constant source of bugs, and it's finally gone.

There are some consequent simplifications yet to be made, like not using
a separate EState for subplans in the executor, but I'll tackle that later.

storing mostly-redundant Query trees in prepared statements, portals, etc.
To replace Query, a new node type called PlannedStmt is inserted by the
planner at the top of a completed plan tree; this carries just the fields of
Query that are still needed at runtime.  The statement lists kept in portals
etc. now consist of intermixed PlannedStmt and bare utility-statement nodes
--- no Query.  This incidentally allows us to remove some fields from Query
and Plan nodes that shouldn't have been there in the first place.

Still to do: simplify the execution-time range table; at the moment the
range table passed to the executor still contains Query trees for subqueries.

initdb forced due to change of stored rules.

out that ExecEvalVar and friends don't necessarily have access to a tuple
descriptor with correct typmod: it definitely can contain -1, and possibly
might contain other values that are different from the Var's value.
Arguably this should be cleaned up someday, but it's not a simple change,
and in any case typmod discrepancies don't pose a security hazard.
Per reports from numerous people :-(

I'm not entirely sure whether the failure can occur in 8.0 --- the simple
test cases reported so far don't trigger it there.  But back-patch the
change all the way anyway.

made query plan.  Use of ALTER COLUMN TYPE creates a hazard for cached
query plans: they could contain Vars that claim a column has a different
type than it now has.  Fix this by checking during plan startup that Vars
at relation scan level match the current relation tuple descriptor.  Since
at that point we already have at least AccessShareLock, we can be sure the
column type will not change underneath us later in the query.  However,
since a backend's locks do not conflict against itself, there is still a
hole for an attacker to exploit: he could try to execute ALTER COLUMN TYPE
while a query is in progress in the current backend.  Seal that hole by
rejecting ALTER TABLE whenever the target relation is already open in
the current backend.

This is a significant security hole: not only can one trivially crash the
backend, but with appropriate misuse of pass-by-reference datatypes it is
possible to read out arbitrary locations in the server process's memory,
which could allow retrieving database content the user should not be able
to see.  Our thanks to Jeff Trout for the initial report.

Security: CVE-2007-0556

back-stamped for this.

involving HashAggregate over SubqueryScan (this is the known case, there
may well be more).  The bug is only latent in releases before 8.2 since they
didn't try to access tupletable slots' descriptors during ExecDropTupleTable.
The least bogus fix seems to be to make subqueries share the parent query's
memory context, so that tupdescs they create will have the same lifespan as
those of the parent query.  There are comments in the code envisioning going
even further by not having a separate child EState at all, but that will
require rethinking executor access to range tables, which I don't want to
tackle right now.  Per bug report from Jean-Pierre Pelletier.

that's shorter-lived than the expression state being evaluated in it really
doesn't work :-( --- we end up with fn_extra caches getting deleted while
still in use.  Rather than abandon the notion of caching expression state
across domain_in calls altogether, I chose to make domain_in a bit cozier
with ExprContext.  All we really need for evaluating variable-free
expressions is an ExprContext, not an EState, so I invented the notion of a
"standalone" ExprContext.  domain_in can prevent resource leakages by doing
a ReScanExprContext on this rather than having to free it entirely; so we
can make the ExprContext have the same lifespan (and particularly the same
per_query memory context) as the expression state structs.

(table or index) before trying to open its relcache entry.  This fixes
race conditions in which someone else commits a change to the relation's
catalog entries while we are in process of doing relcache load.  Problems
of that ilk have been reported sporadically for years, but it was not
really practical to fix until recently --- for instance, the recent
addition of WAL-log support for in-place updates helped.

Along the way, remove pg_am.amconcurrent: all AMs are now expected to support
concurrent update.