[#134457777]

[#134457777]

The dev team has dedicated workers separate from the tagged workers for
gpdb team.

[#134457777]

[#134241435]

Adding a new step in concourse that generates a tarball for
the tinc.

Attach a suitable error code for many errors that were previously reported
as "internal errors". GPDB's elog.c prints a source file name and line
number for any internal errors, which is a bit ugly for errors that are
in fact unexpected internal errors, but user-facing errors that happen
as a result of e.g. an invalid query.

To make sure we don't accumulate more of these, adjust the regression tests
to not ignore the source file and line number in error messages. There are
a few exceptions, which are listed explicitly.

We had backported PL/python from PostgreSQL 9.1, but not this patch
that was applied later to the stable branch. I noticed when I started
looking at this, while cleaning up the source file/line number information
in error messages.

I did not backport the regression test changes, because we had apparently
not backported other regression test changes either, and I was not able
to make the tests work easily.

Original commit:

commit d2192a108c7b118d430f03cff0ff84861d5026e0
Author: Heikki Linnakangas <heikki.linnakangas@iki.fi>
Date:   Thu Nov 24 17:18:43 2011 +0200

    Preserve SQLSTATE when an SPI error is propagated through PL/python
    exception handler. This was a regression in 9.1, when the capability
    to catch specific SPI errors was added, so backpatch to 9.1.

    Mika Eloranta, with some editing by Jan Urbański.

It was somewhat broken. The order of evaluation of function arguments is
implementation-specific, so in a statement like:

ereport(ERROR,
        (errcode(ERRCODE_INTERNAL_ERROR),
	 errOmitLocation(true)));

We cannot assume that errOmitLocation() is evaluated after errcode(). If
errOmitLocation() is evaluated first, the errcode() call could overwrite
the omit_location field, seeing that the error code was "internal error".

Almost all of the errOmitLocation calls in the codebase were superfluous
anyway. The default logic is to omit the location for anything else than
ERRCODE_INTERNAL error, so it is not necessary to call errOmitLocation(true),
if errcode (other than ERRCODE_INTERNAL_ERROR) is given. Likewise
errOmitLocation(false) is not needed for internal errors.

Remove the whole errOmitLocation() function. It's not really needed. The
most notable callsite where it mattered was in cdbdisp.c, but that one
was broken by the order-of-evaluation issue. Use a different error code
there. What we really should do there is to pass the error code from the
segment back to the client, but I'll leave that for another day.

With commit 61972775, we use a proper SQLSTATE for the errors that
needed this before.

I figures out the reason this is needed. It was alluded to by the
"This is so the executor can use a faster path to evaluate the  given
node's targetlist" comment, but hopefully this new comment makes it
more clear what the "faster path" is.

Remove the GPDB_83_MERGE_FIXME comment. There's a new mechanism for
dealing with SRFs in targetlists in the works for PostgreSQL 10
as of this writing, but until then, I think we'll live with this.

pg_get_constraintdef() looks up information on the constraint
from the syscache, which uses SnapshotNow to fetch the information. But the
query itself uses an MVCC snapshot. If the query's MVCC snapshot sees a
constraint that is concurrently dropped so that it's not visible to SnapshotNow,
pg_get_constraintdef() will report error.

Thanks Heikki for above description.

Nothing actual things in this yet.
Signed-off-by: NNing Wu <nwu@pivotal.io>

s3ext fail to compile on CentOS 7,
/usr/include/libintl.h:39:14: error: expected unqualified-id before ‘const’
 extern char *gettext (const char *__msgid)

Root cause:
In greenplum-db-devel/include/postgresql/server/c.h
ENABLE_NLS macro controls gettext function definition.

#ifdef ENABLE_NLS
#ifndef __S3_COMMON_HEADERS_H__
#include <libintl.h>
#else
#include "s3utils.h"
#define gettext(x) (x)
#define dgettext(d,x) (x)
#define ngettext(s,p,n) ((n) == 1 ? (s) : (p))
#define dngettext(d,s,p,n) ((n) == 1 ? (s) : (p))
#endif

If c.h is included before libintl.h, and ENABLE_NLS is not defined,
gettext function definition in libintl.h be replaced.

libintl.h is also included by iomanip.h

Resolve:
Move iomanip.h to s3utils.cpp
Signed-off-by: NNing Wu <nwu@pivotal.io>

Append an EOL if files don't have it, otherwise the last line will be
combined with first line of next file.
Signed-off-by: NNing Wu <nwu@pivotal.io>
Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>

When one segment loads a set of file from S3 and each file has
a header line, we may meet data parsing error. We keep only
the header line of the 1st file and ignore others.
Signed-off-by: NPeifeng Qiu <pqiu@pivotal.io>

Fix gpcheckcat extraneous entry repair to use primary keys if oids not consistent across segments. (#1288)

gpcheckcat missing/extraneous repair assumed oids are consistent across 
segments for a given catalog entry.This is not always guaranteed for all catalog 
tables (which are defined in gpcheckcat). Instead, we need to generate the 
repair based on the primary keys (the unique identifier in this case).

Authors: Marbin, Larry, Chris, Chumki and Karen

Update README with instructions to duplicate pipelines for one-off
development pipelines. Add new pipeline to be a template for duplication
that has a dedicated S3 bucket to collect dev artifacts.

[#134454291]

When running backup with ddboost, we now use dump timestamp instead of current timestamp as the parent directory.  This fixes a bug where the dump fails if the report and cdatabase files are generated on different days (e.g. if the dump is kicked off just before midnight).

Authors: Stephen Wu and Jamie McAtamney

This file explains the contract that we are drawing out for the use and maintenance of the concourse directory. It describes the organization of the directory, and what should exist there.

[#134241385]

strndup is defined only when _GNU_SOURCE is defined, which won't
work in win32. This patch replace strndup with ANSI-C string
function strtok which will work in win32 env.

Without assertions, the only thing that was checked was that
numProcsPerSegment() returns 1, when the
gp_resqueue_priority_cpucores_per_segment is 1. That seems excessive, as
numProcsPerSegment() is a trivial function that just returns the value
of gp_resqueue_priority_cpucores_per_segment.

With assertions, it checked that an assertion is thrown if
gp_resqueue_priority_cpucores_per_segment has an invalid value. That seems
excessive, because those assertions are just extra sanity checks. The GUC's
minimum and maximum values prevent it from having invalid values in the
first place.

Linking each mock test program takes a few seconds, so this shaves some
time off the build+unit test process.

While we're at it, add a regression test and refactor the code a bit to
make it more readable.

No need to create a whole new library for this. It's included in every
installation anyway, as part of gp_toolkit. Might as well just include
it in the backend, and make the build slightly simpler.

The olap_schedule file it referenced hasn't existed for a very long time.

COPTX and BLD_DEPLOYMENT_SETTING variables were not set anywhere. I think
these were put in place to be able to pass some options from whatever calls
the Makefile, but we have so many of these variables now that no-one knows
what was used for what anymore. Let's remove it. We can always put
something like these back later, if someone needs it.

I don't understand what these targets did. Used rsync to refresh something
from somewhere, but I couldn't figure out how it works. In any case, I
don't think anyone uses them, so just remove.

Thanks to Corbin Halliwill for identifying these.

* Changing pipeline to incorporate gpaddon repository.
* Modifying pipeline to add gpaddon to packaging stage.
* Modify install script path for gpaddon.
Signed-off-by: NShoaib Lari <slari@pivotal.io>
Signed-off-by: NFoyzur Rahman <foyzur@gmail.com>
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

WAL-logging an insert record was extracted to a separate function in GPDB,
but as long as there's only one caller of it, that just increases our diff
footprint vs upstream. Move it back.

To reduce our diff vs. upstream.

Not sure why we had changed that. Perhaps 2328 was in use for some other
object? In any case, its free now.

Macros like this might be a good idea, but all the upstream code just uses
the "offsetof(<struct>, <member>) + sizeof(<elem type>) * <count>" idiom
directly, so let's follow the example.

The SIZEOF_FIELD and VARELEMENTS_TO_FIT macros were outright unused.