1. 22 9月, 2018 4 次提交
    • T
      Move memoryAccountId out of PlannedStmt/Plan Nodes · 7c9cc053
      Taylor Vesely 提交于
      Functions using SQL and PL/pgSQL will plan and execute arbitrary SQL
      inside a running query. The first time we initialize a plan for an SQL
      block, the memory accounting system creates a new memory account for
      each Executor/Node.  In the case that we are executing a cached plan,
      (i.e.  plancache.c) the memory accounts will have already been assigned
      in a previous execution of the plan.
      
      As a result, when explain_memory_verbosity is set to 'detail', it is not
      clear what memory account corresponds to which executor. Instead, move
      the memoryAccountId into PlanState/QueryDesc, which will insure that
      every time we initialize an executor, it will be assigned a unique
      memoryAccountId.
      Co-authored-by: NMelanie Plageman <mplageman@pivotal.io>
      7c9cc053
    • H
      Remove FIXME in RemoveLocalLock, it's alright. · 9e57124b
      Heikki Linnakangas 提交于
      The FIXME was added to GPDB in commit f86622d9, which backported the
      local cache of resource owners attached to LOCALLOCK. I think the comment
      was added, because in the upstream commit that added the cache, the
      upstream didn't thave the check guarding the pfree() yet. It was added
      later in upstream, too, in commit 7e6e3bdd3c, and that had already been
      backported to GPDB. So it's alright, the guard on the pfree is a good thing
      to have, and there's nothing further to do here.
      9e57124b
    • H
      Change pretty-printing of expressions in EXPLAIN to match upstream. · 4c54c894
      Heikki Linnakangas 提交于
      We had changed this in GPDB, to print less parens. That's fine and dandy,
      but it hardly seems worth it to carry a diff vs upstream for this. Which
      format is better, is a matter of taste. The extra parens make some
      expressions more clear, but OTOH, it's unnecessarily verbose for simple
      expressions. Let's follow the upstream on this.
      
      These changes were made to GPDB back in 2006, as part of backporting
      to EXPLAIN-related patches from PostgreSQL 8.2. But I didn't see any
      explanation for this particular change in output in that commit message.
      
      It's nice to match upstream, to make merging easier. However, this won't
      make much difference to that: almost all EXPLAIN plans in regression
      tests are different from upstream anyway, because GPDB needs Motion nodes
      for most queries. But every little helps.
      4c54c894
    • H
      Remove commented-out block of macOS makefile stuff. · c5d875b5
      Heikki Linnakangas 提交于
      I don't understand what all this was about, but people have compiled GPDB
      successfully after the merge commit, where this was commented out, so
      apparently it's not needed.
      c5d875b5
  2. 21 9月, 2018 18 次提交
    • H
      Remove duplicated code to handle SeqScan, AppendOnlyScan and AOCSScan. · ff8161a2
      Heikki Linnakangas 提交于
      They were all treated the same, with the SeqScan code being duplicated
      for AppendOnlyScans and AOCSScans. That is a merge hazard: if some code
      is changed for SeqScans, we would have to remember to manually update
      the other copies. Small differences in the code had already crept up,
      although given that everything worked, I guess it had no effect. Or
      only had a small effect on the computed costs.
      
      To avoid the duplication, use SeqScan for all of them. Also get rid of
      TableScan as a separate node type, and have ORCA translator also create
      SeqScans.
      
      The executor for SeqScan node can handle heap, AO and AOCS tables, because
      we're not actually using the upstream SeqScan code for it. We're using the
      GPDB code in nodeTableScan.c, and a TableScanState, rather than
      SeqScanState, as the executor node. That's how it worked before this patch
      already, what this patch changes is that we now use SeqScan *before* the
      executor phase, instead of SeqScan/AppendOnlyScan/AOCSScan/TableScan.
      
      To avoid having to change all the expected outputs for tests that use
      EXPLAIN, add code to still print the SeqScan as "Seq Scan", "Table Scan",
      "Append-only Scan" or "Append-only Columnar Scan", depending on whether
      the plan was generated by ORCA, and what kind of a table it is.
      ff8161a2
    • H
      Move UnpackCheckPointRecord to xlogdesc.c, to avoid duplicating it. · 16343336
      Heikki Linnakangas 提交于
      As noted in the FIXME, having two copies of the function is bad. It's easy
      to avoid the duplication, if we just put it in xlogdesc.c, so that it's
      available to xlog_desc() in client programs, too.
      16343336
    • D
      Remove unused variable · 414531a6
      Daniel Gustafsson 提交于
      Fixes compiler warning on unused variable which was left over in the
      9.3 merge.
      Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
      414531a6
    • A
      Avoid inconsistent type declaration · e52f9c2d
      Alvaro Herrera 提交于
      Clang 3.3 correctly complains that a variable of type enum
      MultiXactStatus cannot hold a value of -1, which makes sense.  Change
      the declared type of the variable to int instead, and apply casting as
      necessary to avoid the warning.
      
      Per notice from Andres Freund
      e52f9c2d
    • H
      Merge with PostgreSQL 9.3 (up to almost 9.3beta2) · c7649f18
      Heikki Linnakangas 提交于
      Merge with PostgreSQL, up to the point where the REL9_3_STABLE branch was
      created, and 9.4 development started on the PostgreSQL master branch. That
      is almost up to 9.3beta2.
      
      Notable upstream changes, from a GPDB point of view:
      
      * LATERAL support. Mostly works in GPDB now, although performance might not
        be very good. LATERAL subqueries, except for degenerate cases that can be
        made non-LATERAL during optimization, typically use nested loop joins.
        Unless the data distribution is the same on both sides of the join, GPDB
        needs to add Motion nodes, and cannot push down the outer query parameter
        to the inner side through the motion. That is the same problem we have
        with SubPlans and nested loop joins in general, but it happens frequently
        with LATERAL. Also, there are a couple of cases, covered by the upstream
        regression tests, where the planner currently throws an error. They have
        been disabled and marked with GPDB_93_MERGE_FIXME comments, and will need
        to be investigated later. Also, no ORCA support for LATERAL yet.
      
      * Materialized views. They have not been made to work in GPDB yet. CREATE
        MATERIALIZED VIEW works, but REFRESH MATERIALIZED VIEW does not. The
        'matviews' test has been temporarily disabled, until that's fixed. There
        is a GPDB_93_MERGE_FIXME comment about this too.
      
      * Support for background worker processes. Nothing special was done about
        them in the merge, but we could now make use of them for all the various
        GPDB-specific background processes, like the FTS prober and gpmon
        processes.
      
      * Support for writable foreign tables was introduced. I believe foreign
        tables now have all the same functionality, at a high level, as external
        tables, so we could start merging the two concepts. But this merge commit
        doesn't do anything about that yet, external tables and foreign tables
        are still two entirely different beasts.
      
      * A lot of expected output churn, thanks to a few upstream changes. We no
        longer print a NOTICE on implicitly created indexes and sequences (commit
        d7c73484), and the rules on when table aliases are printed were changed
        (commit 11e13185).
      
      * Caught up to a bunch of features that we had already backported from 9.3:
        data page checksums, numeric datatype speedups, COPY FROM/TO PROGRAM, and
        pg_upgrade as whole.
      
      A couple of other noteworthy changes:
      
      * contrib/xlogdump utility is removed, in favor of the upstream
        contrib/pg_xlogdump utility.
      
      * Removed "idle session timeout" hook. The current implementation was badly
        broken by upstream refactoring of timeout handling (commit f34c68f0).
        We'll probably need to re-introduce it in some form, but it will look
        quite different, to make it fit more nicely with the new timeout APIs.
      Co-authored-by: NAshwin Agrawal <aagrawal@pivotal.io>
      Co-authored-by: NAsim R P <apraveen@pivotal.io>
      Co-authored-by: NDavid Kimura <dkimura@pivotal.io>
      Co-authored-by: NEkta Khanna <ekhanna@pivotal.io>
      Co-authored-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
      Co-authored-by: NJacob Champion <pchampion@pivotal.io>
      Co-authored-by: NJinbao Chen <jinchen@pivotal.io>
      Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>
      Co-authored-by: NPaul Guo <paulguo@gmail.com>
      Co-authored-by: NRichard Guo <guofenglinux@gmail.com>
      Co-authored-by: NShaoqi Bai <sbai@pivotal.io>
      c7649f18
    • A
      Fix travis build error, old apr tarball doesn't exist anymore · 27adcf92
      Adam Lee 提交于
      ```
      $ wget http://ftp.jaist.ac.jp/pub/apache/apr/${APR}.tar.gz
      --2018-09-21 07:16:24--  http://ftp.jaist.ac.jp/pub/apache/apr/apr-1.6.3.tar.gz
      Resolving ftp.jaist.ac.jp (ftp.jaist.ac.jp)... 150.65.7.130, 2001:df0:2ed:feed::feed
      Connecting to ftp.jaist.ac.jp (ftp.jaist.ac.jp)|150.65.7.130|:80... connected.
      HTTP request sent, awaiting response... 404 Not Found
      2018-09-21 07:16:25 ERROR 404: Not Found.
      ```
      27adcf92
    • A
      Fix COPY SEGV caused by uninitialized variables · 688a43f0
      Adam Lee 提交于
      It happens if the copy command errors out before assigning
      dispatcherState. Initialize the dispatcherState as NULL to fix it,
      palloc0() to avoid future new member issues.
      
      5X has no such problem.
      
      ```
      (gdb) c
      Continuing.
      Detaching after fork from child process 25843.
      
      Program received signal SIGSEGV, Segmentation fault.
      0x0000000000aa04dd in getCdbCopyPrimaryGang (c=0x23d4150) at cdbcopy.c:44
      44              return (Gang *)linitial(c->dispatcherState->allocatedGangs);
      (gdb) bt
      \#0  0x0000000000aa04dd in getCdbCopyPrimaryGang (c=0x23d4150) at cdbcopy.c:44
      \#1  0x0000000000aa12d8 in cdbCopyEndAndFetchRejectNum (c=0x23d4150, total_rows_completed=0x0, abort_msg=0xd0c8f8 "aborting COPY in QE due to error in QD") at cdbcopy.c:642
      \#...
      (gdb) p c->dispatcherState
      $1 = (struct CdbDispatcherState *) 0x100000000
      ```
      688a43f0
    • H
      Use psql's unaligned format in EXPLAIN tests, to make it less brittle. · 43ccd3d5
      Heikki Linnakangas 提交于
      In aligned format, there is an end-of-line marker at the end of each line,
      and its position depends on the longest line. If the width changes, all
      lines need to be adjusted for the moved end-of-line-marker.
      
      While testing this, we found out that 'atmsort' had been doing bad things
      to the YAML output before:
      
          -- Check Explain YAML output
          EXPLAIN (FORMAT YAML) SELECT * from boxes LEFT JOIN apples ON apples.id = boxes.apple_id LEFT JOIN box_locations ON box_locations.id = boxes.location_id;
          QUERY PLAN
          ___________
          {
            'id' => 1,
            'short' => '- Plan:                                                  +'
          }
          GP_IGNORE:(1 row)
      
      In other worse, we were not comparing the output at all, except for that one
      line that says "Plan:". The access plan for one of the queries had changed,
      from a Left Join to a Right Join, and we still had the old plan memorized
      in expected output, but the test was passing because atmsort hid the issue.
      This commit fixes the expected output for the new plan.
      43ccd3d5
    • H
      Remove check for NOT NULLable column from ORCA translation of INSERT values. · e89be84b
      Heikki Linnakangas 提交于
      When creating an ORCA plan for "INSERT ... (<col list>) VALUES (<values>)"
      statement, the ORCA translator performed NULL checks for any columns not
      listed in the column list. Nothing wrong with that per se, but we needed
      to keep the error messages in sync, or we'd get regression test failures
      caused by different messages. To simplify that, remove the check from
      ORCA translator, and rely on the execution time check.
      
      We bumped into this while working on the 9.3 merge, because 9.3 added
      DETAIL to the error message in executor:
      
      postgres=# create table notnulls (a text NOT NULL, b text NOT NULL);
      CREATE TABLE
      postgres=# insert into notnulls (a) values ('x');
      ERROR:  null value in column "b" violates not-null constraint
      postgres=# insert into notnulls (a,b) values ('x', NULL);
      ERROR:  null value in column "b" violates not-null constraint  (seg2 127.0.0.1:40002 pid=26547)
      DETAIL:  Failing row contains (x, null).
      
      Doing this now will avoid that inconsistency in the merge.
      
      One little difference with this is that EXPLAIN on an insert like above
      now works, and you only get the error when you try to execute it. Before,
      with ORCA, even EXPLAIN would throw the error.
      e89be84b
    • H
      Remove gpfdist --sslclean option (#5800) · d94bebb5
      Huiliang.liu 提交于
      gpfdist --sslclean option is a platform related patch for Solaris system.
      gpfdist delays cleaning ssl buffer for some seconds which is configured
      by sslclean option.
      GPDB6 doesn't support Solaris now. We don't think that solution has
      benefit for other platforms, so we remove --sslclean option.
      
      Have Verified this patch manually and the default test cases cover this change.
      d94bebb5
    • J
      Unify all error report to ereport in cdbutil.c:getCdbComponentInfo. (#5802) · 74270ccc
      Jialun 提交于
      * Unify all error report to ereport in cdbutil.c:getCdbComponentInfo.
      
      * Change the message to lower case which is the usual PostgreSQL
      style for error messages.
      74270ccc
    • B
      Remove a FIXME comment about the need to have EvalPlanQual recheck functions for AO table scans · 12f96ee7
      BaiShaoqi 提交于
      Reviewed by Paul and heikki, ideas brought by heikki
      12f96ee7
    • T
      Fix pg_stat_activity show wrong session id after session reset bug (#5757) · ac54faad
      Teng Zhang 提交于
      * Fix pg_stat_activity show wrong session id after session reset bug
      
      Currently if a session is reset because of some error such as OOM,
      after call CheckForResetSession, gp_session_id will bump to a new one,
      but sess_id in pg_stat_activity remains unchanged and show the wrong number.
      This commit changes sess_id in pg_stat_activity, once a session is reset.
      
      * Refactor test using gp_execute_on_server to trigger session reset
      ac54faad
    • D
      Revert "Bump ORCA version to 3.0.0" · fcadfe88
      Dhanashree Kashid 提交于
      Revert following commits related to ORCA version 3.0.0
      
      b1f543f3.
      
      b0359e69.
      
      a341621d.
      fcadfe88
    • S
      Bump ORCA version to 3.0.0 · b1f543f3
      Sambitesh Dash 提交于
      Signed-off-by: NDhanashree Kashid <dkashid@pivotal.io>
      b1f543f3
    • S
      Randomize output segment for non-master gather motion · b0359e69
      Sambitesh Dash 提交于
      Via https://github.com/greenplum-db/gporca/pull/400, ORCA will optimize
      DML queries by enforcing a gather on segment instead of master, whenever
      possible.
      
      Previous to this commit, ORCA always picked the first segment to gather
      on while translating the DXL-GatherMotion node to GPDB motion node.
      
      This commit uses GPDB's hash function to select the segment to gather
      on, in a round-robin fashion starting with a random segment index. This
      will ensure that concurrent DML queries issued via a same session, will
      be gathered on different segments to distribute the workload.
      Signed-off-by: NDhanashree Kashid <dkashid@pivotal.io>
      b0359e69
    • S
      Introduce optimizer_enable_gather_on_segment_for_DML GUC · a341621d
      Sambitesh Dash 提交于
      When ON, ORCA will optimize DML queries by enforcing a non-master gather
      whenever possible. When off, a gather on master will be enforced
      instead.
      
      Default value will be ON.
      
      Also add new tests to ensure sane behavior when this optimization is
      turned on and fix the existing tests.
      Signed-off-by: NSambitesh Dash <sdash@pivotal.io>
      Signed-off-by: NDhanashree Kashid <dkashid@pivotal.io>
      a341621d
    • A
      Improve PR pipeline feedback loop · 723fbcfb
      Adam Berlin 提交于
      Previously we were running all tests for all operating systems. This
      commit reduces the running tests down to *only* ICW for both planner and
      ORCA running *only* on Ubuntu. This reduces the chance that a flakey
      test causes a false-negative run of the PR pipeline. Less tests to run
      should also improve the speed of the pipeline, giving us faster
      feedback, while giving us most of the confidence we need.  Exhaustive
      testing will be done downstream after a PR has been merged.
      Co-authored-by: NTaylor Vesely <tvesely@pivotal.io>
      723fbcfb
  3. 20 9月, 2018 7 次提交
    • J
      Fix ao_upgrade tests and add them to the schedule · 2b96acd2
      Jacob Champion 提交于
      9.1 added a new, more compact "short" format to the numeric datatype.
      This format wasn't handled by the ao_upgrade test in isolation2, so it
      failed -- but the pipeline was still green because I forgot to add the
      new test to the schedule in 54895f54.
      
      To fix the issue, add a new helper which will force any Numeric back to
      the legacy long format, and call that from convertNumericToGPDB4() in
      the ao_upgrade test. And add the test to the schedule, so we don't have
      to do this again.
      Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>
      2b96acd2
    • H
      Fix and clean up db/relation/tablespace size functions. · f8a80aeb
      Heikki Linnakangas 提交于
      This fixes several small bugs:
      
      - Schema-qualify the functions in all queries.
      
      - Quote database and tablespace names correctly in the dispatched
        queries.
      
      - In the variants that take OID, also dispatch the OID rather than the
        resolved name. This avoids having to deal with quoting schema and table
        names in the query, and seems like the right thing to do anyway.
      
      - Dispatch pg_table_size() pg_indexes_size() variants. These were added
        in PostgreSQL 9.0, but we missed modifying them in the merge, the same
        way that we have modified all the other variants.
      
      Also, refactor the internal function used to dispatch the pg_*_size()
      calls to use CdbDispatchCommand directly, instead of using SPI and the
      gp_dist_random('gp_id') trick. Seems more straightforward, although I
      believe that trick should've worked, too.
      
      Add tests. We didn't have any bespoken tests for these functions, although
      we used some of the variants in other tests.
      Reviewed-by: NDaniel Gustafsson <dgustafsson@pivotal.io>
      f8a80aeb
    • H
      Remove the printing of overflowed value, when scale is exceeded. · 4b31e46f
      Heikki Linnakangas 提交于
      Printing the value was added in GPDB, back in 2007. The commit message of
      that change (in the historical pre-open-sourcing git repository) said:
      
          Merge forward from Release-3_0_0-branch. Update comment block.
          Tidy numeric_to_pos_int8_trunc.
      
      That wasn't not very helpful...
      
      Arguably, printing the value can be useful, but if so, we should submit
      this change to upstream. I don't think it's worth the trouble, though, so
      I suggest that we just revert this to the way it is in the upstream. The
      reason I'm doing this now is that this caused merge conflicts in the 9.3
      merge, that we're working on right now. We could probably fix the conflict
      in a way that keeps the extra message, but it's simpler to just drop it.
      4b31e46f
    • H
      Remove some leftover initGpmonPkt* functions. · 46b8293f
      Heikki Linnakangas 提交于
      Commit c1690010 removed most of these, but missed these few in GPDB-
      specific executor node. These are no longer needed, just like all the ones
      that were removed in commit c1690010.
      46b8293f
    • B
      Fix subquery with column alias `zero` produces wrong result (#5790) · b70b0086
      BaiShaoqi 提交于
      Reviewed and ideas brought by heikki
      b70b0086
    • H
      Enable 'gp_types' test. · a90e92d9
      Heikki Linnakangas 提交于
      The test was added in commit 3672f912, but I forgot to add it to the
      schedule. Oops.
      a90e92d9
    • L
      docs - correct pljava JAVA_HOME, LD_LIBRARY_PATH settings (#5760) · 4cd7c0c8
      Lisa Owen 提交于
      * docs - correct pljava JAVA_HOME, LD_LIBRARY_PATH settings
      
      * restoring typo fix after resolving conflict
      4cd7c0c8
  4. 19 9月, 2018 9 次提交
    • H
      Fix "could not find pathkey item to sort" error with MergeAppend plans. · 1722adb8
      Heikki Linnakangas 提交于
      When building a Sort node to represent the ordering that is preserved
      by a Motion node, in make_motion(), the call to make_sort_from_pathkeys()
      would sometimes fail with "could not find pathkey item to sort". This
      happened when the ordering was over a UNION ALL operation. When building
      Motion nodes for MergeAppend subpaths, the path keys that represented the
      ordering referred to the items in the append rel's target list, not the
      subpaths. In create_merge_append_plan(), where we do a similar thing for
      each subpath, we correctly passed the 'relids' argument to
      prepare_sort_from_pathkeys(), so that prepare_sort_from_pathkeys() can
      match the target list entries of the append relation with the entries of
      the subpaths. But when creating the Motion nodes for each subpath, we
      were passing NULL as 'relids' (via make_sort_from_pathkeys()).
      
      At a high level, the fix is straightforward: we need to pass the correct
      'relids' argument to prepare_sort_from_pathkeys(), in
      cdbpathtoplan_create_motion_plan(). However, the current code structure
      makes that not so straightforward, so this required some refactoring of
      the make_motion() and related functions:
      
      Previously, make_motion() and make_sorted_union_motion() would take a path
      key list as argument, to represent the ordering, and it called
      make_sort_from_pathkeys() to extract the sort columns, operators etc.
      After this patch, those functions take arrays of sort columns, operators,
      etc. directly as arguments, and the caller is expected to do the call to
      make_sort_from_pathkeys() to get them, or build them through some other
      means. In cdbpathtoplan_create_motion_plan(), call
      prepare_sort_from_pathkeys() directly, rather than the
      make_sort_from_pathkeys() wrapper, so that we can pass the 'relids'
      argument. Because prepare_sort_from_pathkeys() is marked as 'static', move
      cdbpathtoplan_create_motion_plan() from cdbpathtoplan.c to createplan.c,
      so that it can call it.
      
      Add test case. It's a slightly reduced version of a query that we already
      had in 'olap_group' test, but seems better to be explicit. Revert the
      change in expected output of 'olap_group', made in commit 28087f4e,
      which memorized the error in the expected output.
      
      Fixes https://github.com/greenplum-db/gpdb/issues/5695.
      Reviewed-by: NPengzhou Tang <ptang@pivotal.io>
      Reviewed-by: NMelanie Plageman <mplageman@pivotal.io>
      1722adb8
    • H
      Simplify 'partindex_test', by using OUT parameters and pg_get_expr(). · fcc7898b
      Heikki Linnakangas 提交于
      OUT parameters make calling the function much less awkward, as you don't
      need to specify the output columns in every invocation. Also, change the
      datatype of a few columns to pg_node_tree, so that you can use pg_get_expr()
      to pretty-print them.
      
      I'm doing this to hide the trivial differences in the internal string
      representation of expressions. This came up in the 9.3 merge, which added
      a new field to FuncExpr again, which would cause the test to fail. Using
      pg_get_expr() to print the fields in human-readable format, which is not
      sensitive to small changes like that, will avoid that problem.
      fcc7898b
    • D
      Fix Greenplum acronym in comment · f879cc9b
      Daniel Gustafsson 提交于
      s/GDPB/GPDB/
      f879cc9b
    • D
      Attempt to make olap_window_seq query deterministic · e6af34f0
      Daniel Gustafsson 提交于
      One of the queries in the olap_window_seq suite was non-deterministic as
      it was ordering around a column which had duplicate values. Switch to the
      _ord version of the table and use the sequence column which yields
      deterministic results.
      Reviewed-by: NVenkatesh Raghavan <vraghavan@pivotal.io>
      e6af34f0
    • B
      Update optimizer output test files and bump orca to v2.75.0 · 35697528
      Bhuvnesh Chaudhary 提交于
      This commit adds the tests corresponding to the below change introduced in PQO.
      In case of Insert on Randomly distributed relations, a random / redistribute
      motion must exists to ensure randomness of the data inserted, else there will
      be skew on 1 segment.
      
      Consider the below scenario:
      Scenario 1:
      ```
      CREATE TABLE t1_random (a int) DISTRIBUTED RANDOMLY;
      INSERT INTO t1_random VALUES (1), (2);
      SET optimizer_print_plan=on;
      
      EXPLAIN INSERT INTO t1_random VALUES (1), (2);
      Physical plan:
      +--CPhysicalDML (Insert, "t1_random"), Source Columns: ["column1" (0)], Action: ("ColRef_0001" (1))   rows:2   width:4  rebinds:1   cost:0.020884   origin: [Grp:1, GrpExpr:2]
         +--CPhysicalComputeScalar
            |--CPhysicalMotionRandom  ==> Random Motion Inserted (2)
            |  +--CPhysicalConstTableGet Columns: ["column1" (0)] Values: [(1); (2)] ==> Delivers Universal Spec (1)
            +--CScalarProjectList
               +--CScalarProjectElement "ColRef_0001" (1)
                  +--CScalarConst (1)
      ",
                                             QUERY PLAN
      ----------------------------------------------------------------------------------------
       Insert  (cost=0.00..0.02 rows=1 width=4)
         ->  Result  (cost=0.00..0.00 rows=1 width=8)
               ->  Result  (cost=0.00..0.00 rows=1 width=4) ==> Random Motion Converted to a Result Node with Hash Filter to avoid duplicates (4)
                     ->  Values Scan on "Values"  (cost=0.00..0.00 rows=1 width=4) ==> Delivers Universal Spec (3)
       Optimizer: PQO version 2.70.0
      ```
      
      When an Insert is requested on t1_random from a Universal Source, Optimization
      framework does add a Random Motion / CPhysicalMotionRandom (See (2) above) to
      redistribute the data. However, since CPhysicalConstTableGet / Values Scan
      delivers Universal Spec, it is converted to a Result Node with hash filter to
      avoid duplicates in DXL to Planned Statement (See (4) above). Now, since there
      is no redistribute motion to spray the data randomly on the segments, due to
      the result node with hash filters, data from only one segment is allowed to
      propagate further, which is inserted by the DML node. This results in all the
      data being inserted on to 1 segment only.
      
      Scenario 2:
      ```
      CREATE TABLE t1_random (a int) DISTRIBUTED RANDOMLY;
      CREATE TABLE t2_random (a int) DISTRIBUTED RANDOMLY;
      EXPLAIN INSERT INTO t1_random SELECT * FROM t1_random;
      Physical plan:
      +--CPhysicalDML (Insert, "t1_random"), Source Columns: ["a" (0)], Action: ("ColRef_0008" (8))   rows:1   width:34  rebinds:1   cost:431.010436   origin: [Grp:1, GrpExpr:2]
         +--CPhysicalComputeScalar   rows:1   width:1  rebinds:1   cost:431.000011   origin: [Grp:5, GrpExpr:1]
            |--CPhysicalTableScan "t2_random" ("t2_random")   rows:1   width:34  rebinds:1   cost:431.000006   origin: [Grp:0, GrpExpr:1]
            +--CScalarProjectList   origin: [Grp:4, GrpExpr:0]
               +--CScalarProjectElement "ColRef_0008" (8)   origin: [Grp:3, GrpExpr:0]
                  +--CScalarConst (1)   origin: [Grp:2, GrpExpr:0]
      ",
                                              QUERY PLAN
      ------------------------------------------------------------------------------------------
       Insert  (cost=0.00..431.01 rows=1 width=4)
         ->  Result  (cost=0.00..431.00 rows=1 width=8)
               ->  Table Scan on t1_random  (cost=0.00..431.00 rows=1 width=4)
       Optimizer: PQO version 2.70.0
      ```
      
      When an Insert is request on t1_random (randomly distributed table) from
      another randomly distributed table, optimization framework does not add a
      redistribute motion because CPhysicalDML requested Random Spec and the source
      t2_random delivers random distribution spec matching / satisfying the requested
      spec.
      
      So, in summary, there are 2 causes for skewness in the data inserted into randomly distributed table.
      Scenario 1: Where the Random Motion is converted into a Result Node to Hash Filters
      Scneario 2: Where the requested and derived spec matches.
      
      This patch fixes the issue by ensuring that if an insert is performed on a
      randomly distributed table, there must exists a random motion which has been
      enforced by a true motion.  This is achived in 2 steps:
      1. CPhysicalDML / Insert on Randomly Distributed table must request a Strict Random Spec
      2. Append Enforcer for Random Spec must track if the motion node enforced has
      a universal child, if it has, then update the bool flag to false else true
      
      Characteristic of Strict Random Spec:
      1. Strict Random Spec matches / satisfies Strict Random Spec Only
      2. Random Spec enforced by a true motion matches / satisfies Strict Random Spec.
      
      CPhysicalDML always has a CPhysicalComputeScalar node below it which projects
      the additional column indicating if it's an Insert. The request for Strict
      Motion is not propagated down the CPhysicalComputeScalar node,
      CPhysicalComputeScalar requests Random Spec from its child instead. This is to
      mandate the existence of a true random motion either by the childs of CPhysicalDML
      node or it should be inserted between CPhysicalDML and CPhysicalComputeScalar.
      If there is any motion in the same group holding CPhysicalComputeScalar,
      it should also request Random Spec from its childs.
      
      Plans after the fix
      Scenario 1:
      ```
      EXPLAIN INSERT INTO t1_random VALUES (1), (2);
      Physical plan:
      +--CPhysicalDML (Insert, "t1_random"), Source Columns: ["column1" (0)], Action: ("ColRef_0001" (1))   rows:2   width:4  rebinds:1   cost:0.020884   origin: [Grp:1, GrpExpr:2]
         +--CPhysicalMotionRandom   rows:2   width:1  rebinds:1   cost:0.000051   origin: [Grp:5, GrpExpr:2] ==> Strict Random Motion Enforcing true randomness
            +--CPhysicalComputeScalar   rows:2   width:1  rebinds:1   cost:0.000034   origin: [Grp:5, GrpExpr:1]
               |--CPhysicalMotionRandom   rows:2   width:4  rebinds:1   cost:0.000029   origin: [Grp:0, GrpExpr:2]
               |  +--CPhysicalConstTableGet Columns: ["column1" (0)] Values: [(1); (2)]   rows:2   width:4  rebinds:1   cost:0.000008   origin: [Grp:0, GrpExpr:1]
               +--CScalarProjectList   origin: [Grp:4, GrpExpr:0]
                  +--CScalarProjectElement "ColRef_0001" (1)   origin: [Grp:3, GrpExpr:0]
                     +--CScalarConst (1)   origin: [Grp:2, GrpExpr:0]
      ",
                                             QUERY PLAN
      ----------------------------------------------------------------------------------------
       Insert  (cost=0.00..0.02 rows=1 width=4)
         ->  Redistribute Motion 3:3  (slice1; segments: 3)  (cost=0.00..0.00 rows=1 width=8) ==> Strict Random Motion Enforcing true randomness
               ->  Result  (cost=0.00..0.00 rows=1 width=8)
                     ->  Result  (cost=0.00..0.00 rows=1 width=4)
                           ->  Values Scan on "Values"  (cost=0.00..0.00 rows=1 width=4)
       Optimizer: PQO version 2
      
      ```
      
      Scenario 2:
      ```
      EXPLAIN INSERT INTO t1_random SELECT * FROM t1_random;
      Physical plan:
      +--CPhysicalDML (Insert, "t1_random"), Source Columns: ["a" (0)], Action: ("ColRef_0008" (8))   rows:2   width:34  rebinds:1   cost:431.020873   origin: [Grp:1, GrpExpr:2]
         +--CPhysicalMotionRandom   rows:2   width:1  rebinds:1   cost:431.000039   origin: [Grp:5, GrpExpr:2] ==> Strict Random Motion Enforcing True Randomness
            +--CPhysicalComputeScalar   rows:2   width:1  rebinds:1   cost:431.000023   origin: [Grp:5, GrpExpr:1]
               |--CPhysicalTableScan "t1_random" ("t1_random")   rows:2   width:34  rebinds:1   cost:431.000012   origin: [Grp:0, GrpExpr:1]
               +--CScalarProjectList   origin: [Grp:4, GrpExpr:0]
                  +--CScalarProjectElement "ColRef_0008" (8)   origin: [Grp:3, GrpExpr:0]
                     +--CScalarConst (1)   origin: [Grp:2, GrpExpr:0]
      ",
                                              QUERY PLAN
      ------------------------------------------------------------------------------------------
       Insert  (cost=0.00..431.02 rows=1 width=4)
         ->  Redistribute Motion 3:3  (slice1; segments: 3)  (cost=0.00..431.00 rows=1 width=8) ==> Strict Random Motion Enforcing true randomness
               ->  Result  (cost=0.00..431.00 rows=1 width=8)
                     ->  Table Scan on t1_random  (cost=0.00..431.00 rows=1 width=4)
       Optimizer: PQO version 2.70.0
      (5 rows)
      ```
      
      Note: If Insert is performed on a Randomly Distributed Table from a Hash
      Distributed Table (childs), an additional redistribute motion is not enforced
      between CPhysicalDML and CPhysicalComputeScalar as there already exists a true
      random motion due to mismatch of random vs hash distributed spec.
      ```
      pivotal=# explain insert into t1_random select * from t1_hash;
      LOG:  statement: explain insert into t1_random select * from t1_hash;
      LOG:  2018-08-20 13:31:01:135438 PDT,THD000,TRACE,"
      Physical plan:
      +--CPhysicalDML (Insert, "t1_random"), Source Columns: ["a" (0)], Action: ("ColRef_0008" (8))   rows:100   width:34  rebinds:1   cost:432.043222   origin: [Grp:1, GrpExpr:2]
         +--CPhysicalComputeScalar   rows:100   width:1  rebinds:1   cost:431.001555   origin: [Grp:5, GrpExpr:1]
            |--CPhysicalMotionRandom   rows:100   width:34  rebinds:1   cost:431.001289   origin: [Grp:0, GrpExpr:2]
            |  +--CPhysicalTableScan "t1_hash" ("t1_hash")   rows:100   width:34  rebinds:1   cost:431.000623   origin: [Grp:0, GrpExpr:1]
            +--CScalarProjectList   origin: [Grp:4, GrpExpr:0]
               +--CScalarProjectElement "ColRef_0008" (8)   origin: [Grp:3, GrpExpr:0]
                  +--CScalarConst (1)   origin: [Grp:2, GrpExpr:0]
      ",
                                                 QUERY PLAN
      -------------------------------------------------------------------------------------------------
       Insert  (cost=0.00..432.04 rows=34 width=4)
         ->  Result  (cost=0.00..431.00 rows=34 width=8)
               ->  Redistribute Motion 3:3  (slice1; segments: 3)  (cost=0.00..431.00 rows=34 width=4)
                     ->  Table Scan on t1_hash  (cost=0.00..431.00 rows=34 width=4)
       Optimizer: PQO version 2.70.0
      (5 rows)
      
      Time: 69.309 ms
      ```
      35697528
    • H
      Rename GPDB added pgbench -q option. · cc797b96
      Heikki Linnakangas 提交于
      PostgreSQL 9.3 introduces a -q option, for "quiet" mode. But in GPDB, we
      have been using it to mean "create non-unique indexes instead of unique,
      primary key, indexes". To avoid the clash once we merge with 9.3, rename
      the GPDB option to --use-unique-keys.
      cc797b96
    • H
      Fix pgbench. · 073a7cfb
      Heikki Linnakangas 提交于
      It was left broken by the 9.2 merge. Need to use getopt_long() now, as in
      upstream, and we were passing wrong variable to snprintf(). Both of these
      were causing compiler warnings, and the latter was causing a segfault.
      
      We don't seem to have any tests for pgbench. I think we'll get some from
      upstream later on, and pgbench isn't super-critical anyway, so I didn't
      address that at this moment.
      073a7cfb
    • M
      docs - update when pg_class.relfrozenxid is 0 (zero) (#5767) · 891839df
      Mel Kiyama 提交于
      * docs - update when pg_class.relfrozenxid is 0 (zero)
      
      * docs - update relfrozenxid info. Update - XID values start at 3 (0, 1, 2 are special IDs).
      891839df
    • H
      Add error position to error on coercing partition boundaries. · a0e41dc7
      Heikki Linnakangas 提交于
      I think this is more clear. Now that we the error position is displayed
      with the error, we don't need to work so hard to deparse the value and the
      partition type. The error position makes that clear.
      
      This came up during the 9.3 merge. Passing InvalidOid to
      deparse_context_for() will no longer work in 9.3, so we'd need to change
      this somehow in the 9.3 merge, anyway.
      a0e41dc7
  5. 18 9月, 2018 2 次提交
    • H
      Remove mock test for cdbappendonlystorage.c · 33fc5f17
      Heikki Linnakangas 提交于
      Looking at the test, it seems pretty uninteresting. It basically tests that
      there is call to IsStandbyMode() in appendonly_redo(), and that the
      switch-case statement works. I don't think we need bespoken tests for those
      things.
      
      This came up while working on the 9.3 merge. 9.3 moves all the *_desc()
      functions to separate files, so after that, only the appendonly_redo()
      function would remain in cdbappendonlystorage.c. It would make sense to
      move it into cdbappendonlyxlog.c, but we couldn't do that, because this
      mock test got in the way. We could fix that in the merge, but I'm opening
      a separate PR of this removal, to give more visibility to the decision.
      33fc5f17
    • D
      Fix typos in comments · 8eca01c2
      Daniel Gustafsson 提交于
      Several instances of s/funciton/function/
      8eca01c2