Ants Aasma and Jeff Davis

(cherry picked from commit fdea2530)

Isolate checksum calculation to its own module, so that bufpage
knows little if anything about the details of the calculation.

This implementation is a modified FNV-1a hash checksum, details
of which are given in the new checksum.c header comments.

Basic implementation only, so we fix the output value.

Later related commits will add version numbers to pg_control,
compiler optimization flags and memory barriers.

Ants Aasma, reviewed by Jeff Davis and Simon Riggs

(cherry picked from commit 43e7a668)

Heikki reported comment was wrong, so fixed
code to match the comment: we only need to
take additional locking precautions when we
have a shared lock on the buffer.

(cherry picked from commit 5787c673)

We copy the buffer before inserting an XLOG_HINT to avoid WAL CRC errors
caused by concurrent hint writes to buffer while share locked. To make this work
we refactor RestoreBackupBlock() to allow an XLOG_HINT to avoid the normal
path for backup blocks, which assumes the underlying buffer is exclusive locked.
Resulting code completely changes layout of XLOG_HINT WAL records, but
this isn't even beta code, so this is a low impact change.
In passing, avoid taking WALInsertLock for full page writes on checksummed
hints, remove related cruft from XLogInsert() and improve xlog_desc record for
XLOG_HINT.

Andres Freund

Bug report by Fujii Masao, testing by Jeff Janes and Jaime Casanova,
review by Jeff Davis and Simon Riggs. Applied with changes from review
and some comment editing.

(cherry picked from commit 47c43331)

(cherry picked from commit a4b94b85)

From performance analysis by Heikki Linnakangas

(cherry picked from commit 1be20351)

Also exclude the block backup for temp relations
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

With `gp_disable_tuple_hints` off, buffer will be marked dirty for hint
bit changes. The XLOG should capture the full page image if the hint bit
is the first change after checkpoint.

During recovery, the full page image should be replayed to override the
page, even if the page is corrupted on disk.

This test verified the table with corrupted page full fixed after
recovery if the `gp_disable_tuple_hints` off.
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

Originally, we didn't verify the correct error level in our helper method.

Using errcode 0 will cause ereport() to treat it as an internal
error and print the filename/line. Since this is a userfacing
error it should have a proper errcode to avoid this. This also
allows the gpdiff rule to be removed.

Commit 82329ca1 introduced this var
but never ended up using it. Remove to avoid compiler warnings on
unused var.

1)add pipeline job for resource group test
2)make test cases independent of segment count and hosts count

This test was asserting on a higher level mock which wasn't being used.
This commit uses the correct mock and the tests are passing.
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

It looks like this is passing in concourse, however, this test suite is
having issues running properly. One of the mocks is not returning the
proper behavior causing the test to fail.
Signed-off-by: NNadeem Ghani <nghani@pivotal.io>

Refactor similar usage to share code with gpperfmon behave tests
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

Resource group may have memory overuse in below case:

	CREATE RESOURCE GROUP rg_concurrency_test WITH
	(concurrency=1, cpu_rate_limit=20, memory_limit=60,
	 memory_shared_quota=0, memory_spill_ratio=10);
	CREATE ROLE role_concurrency_test RESOURCE GROUP rg_concurrency_test;

	11:SET ROLE role_concurrency_test;
	11:BEGIN;

	21:SET ROLE role_concurrency_test;
	22:SET ROLE role_concurrency_test;
	21&:BEGIN;
	22&:BEGIN;

	ALTER RESOURCE GROUP rg_concurrency_test SET CONCURRENCY 2;

	11:END;

The cause is that we didn't check overall memory quota usage in the
past, so pending queries can be waken up as long as the concurrency
limit is not reached, in such a case if the currently running tranctions
have used all the memory quota in the resource group then the overall
memory usage will be exceeded.

To fix this issue we now checks both concurrency limit and memory quota
usage to decide whether to wake up pending queries.
Signed-off-by: NZhenghua Lyu <zlv@pivotal.io>

This bug is caused by:
For COPY FROM ON SEGMENT, on QE, process the data stream (empty) dispatched from QD at first, then re-do the same workflow to read and process the local segment data file. Before redo, all flags need to be reset to initial values too. However we missed one flag.

Add the checksum masking and clearly separate where that masking is applied
separately from when the lp array masking is applied.

This will ensure the data in buf only updated once.
Signed-off-by: NC.J. Jameson <cjameson@pivotal.io>

Validate every BufferPool page sent to the mirror by the
primary prior to writing.
Signed-off-by: NTaylor Vesely <tvesely@pivotal.io>

During resync, filerep copied block with out-of-date checksum over from primary
to mirror. This caused checksum verification failure later on the mirror side,
and also caused inconsistency between the two on disk images of primary and
mirror.

The fix introduced here will always recompute the checksum during resync.

The performance impact is very low, since we only recompute the checksum for
changed blocks. However, for the full copy, we will recompute checksum for all
the blocks to be copied. We have to do it, because there is no easy way to
gurantee there no other changes like hint bit change during resync, since it's
an online operation.

Also fixed wrong comments regarding to page lsn.
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

In CLUSTER, VACUUM FULL and ALTER TABLE SET TABLESPACE
I erroneously set checksum before log_newpage, which
sets the LSN and invalidates the checksum. So set
checksum immediately *after* log_newpage.

Bug report Fujii Masao, Fix and patch by Jeff Davis

(cherry picked from commit cf8dc9e1)

Some compilers understand that this coding is safe, and some don't.

(cherry picked from commit 4912385b)
Signed-off-by: NTaylor Vesely <tvesely@pivotal.io>

(cherry picked from commit 9df56f6d)

Signed-off-by: NXin Zhang <xzhang@pivotal.io>

This test is based on ao_checksum_corruption.sql.

We added new UDF `invalidate_buffers()` to invalidate buffers for given
relation, so that we can read the content from the corrupted file again.

We tested corruptions on heap table, toast table, btree and bitmap indexes.
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

This is a new config option, on by default.
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

Checksums are set immediately prior to flush out of shared buffers
and checked when pages are read in again. Hint bit setting will
require full page write when block is dirtied, which causes various
infrastructure changes. Extensive comments, docs and README.

WARNING message thrown if checksum fails on non-all zeroes page;
ERROR thrown but can be disabled with ignore_checksum_failure = on.

Feature enabled by an initdb option, since transition from option off
to option on is long and complex and has not yet been implemented.
Default is not to use checksums.

Checksum used is WAL CRC-32 truncated to 16-bits.

Simon Riggs, Jeff Davis, Greg Smith
Wide input and assistance from many community members. Thank you.

(cherry picked from commit 96ef3b8f)

Remove use of PageSetTLI() from all page manipulation functions
and adjust README to indicate change in the way we make changes
to pages. Repurpose those bytes into the pd_checksum field and
explain how that works in comments about page header.

Refactoring ahead of actual feature patch which would make use
of the checksum field, arriving later.

Jeff Davis, with comments and doc changes by Simon Riggs
Direction suggested by Robert Haas; many others providing
review comments.

(cherry picked from bb7cc262)

Relation metadata reference was added twice due to which
memory leak is detected and PQO fallsback to planner. This patch
removes redundant AddRef for Relation Metadata and fixes fallback.
Signed-off-by: NEkta Khanna <ekhanna@pivotal.io>

We saw the following on macOS and this mock patch solves it:

`[Errno 13] Permission denied: '/usr/local/gpdb/share/package`
Signed-off-by: NC.J. Jameson <cjameson@pivotal.io>

Resource group memory spill is similar to 'statement_mem' in
resource queue, the difference is memory spill is calculated
according to the memory quota of the resource group.

The related GUCs, variables and functions shared by both resource
queue and resource group are moved to the namespace resource manager.

Also codes of resource queue relating to memory policy are refactored in this commit.
Signed-off-by: NPengzhou Tang <ptang@pivotal.io>
Signed-off-by: NNing Yu <nyu@pivotal.io>

We suspect that the gpperfmon test to be flakey.
This is not acceptable in a PR pipeline.
gpperfmon tests needs more work to be stable.

* Add sequence overflow documentation and example.

Discussed w/ CJ - we'll work offline on a better way to address groups but will temporarily accept this 

The job runs FTS tests and has failed intermittently.  By increasing
gp_segment_connect_timeout, we reduce the chance that test environment will
cause failures.
Signed-off-by: NAsim R P <apraveen@pivotal.io>