Summary: include/posix/io_posix.h is not a public API. Although include/posix/ is not a public header directory, it is confusing to put non-public headers to under include/. Move it to util/ to be clearer.

Test Plan: Run all tests

Reviewers: rven, IslamAbdelRahman, anthony, kradhakrishnan, yhchiang, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D49611

This reverts commit c37223c0.

Fix MockTable ID storage

  On Windows two tests fail that use MockTable:
  flush_job_test and compaction_job_test with the following message:
  compaction_job_test_je.exe : Assertion failed: result.size() == 4,
  file c:\dev\rocksdb\rocksdb\table\mock_table.cc, line 110

  Investigation reveals that this failure occurs when a 4 byte
  ID written to a beginning of the physically open file (main
  contents remains in a in-memory map) can not be read back.

  The reason for the failure is that the ID is written directly
  to a WritableFile bypassing WritableFileWriter. The side effect of that
  is that pending_sync_ never becomes true so the file is never flushed,
  however, the direct cause of the failure is that the filesize_ member
  of the WritableFileWriter remains zero. At Close() the file is truncated
  to that size and the file becomes empty so the ID can not be read back.

Summary: It looks like WritableFileWriter::Append() was returning OK() even when there is an error

Test Plan: make check

Reviewers: sdong, yhchiang, anthony, rven, kradhakrishnan, igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D49569

Fix unused parameter warnings in db.h

Summary: include/posix/io_posix.h should not depend on ROCKSDB_FALLOCATE_PRESENT. Remove it.

Test Plan: Build it with both of ROCKSDB_FALLOCATE_PRESENT defined and not defined.

Reviewers: rven, yhchiang, anthony, kradhakrishnan, IslamAbdelRahman, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D49563

Summary: Mac build breaks as include/posix/io_posix.h doesn't include errno. Move the exact function declaration to io_posix.cc

Test Plan: Run all test. Will run on Mac

Reviewers: rven, anthony, yhchiang, IslamAbdelRahman, igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D49551

 Adding wal filter to inspect and filter wal records on recovery 

Summary: crash_test still has a very low chance to hit some crash point. Have another mode for covering them more likely.

Test Plan: Run crash_test and see db_stress is called with expected prameters.

Reviewers: kradhakrishnan, igor, anthony, rven, IslamAbdelRahman, yhchiang

Reviewed By: yhchiang

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D49473

Error while cmake by building from zip-archive

Fail recovery if filter provides more records than original and corresponding unit-test, fix naming conventions

Summary: IO Posix depends on too many .h files. Move most of them to .cc files.

Test Plan: make all

Reviewers: anthony, rven, IslamAbdelRahman, yhchiang, kradhakrishnan, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D49479

Add Options.recycle_log_file_num for Recycling WAL Files

Summary: Update rocksdb-lego-determinator to include running make check under ROCKSDB_LITE

Test Plan: will be tested after landing in fbcode

Reviewers: sdong, yhchiang, igor, kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D49065

Summary: include/posix/io_posix.h doesn't not prevent multiple includes. Need to fix it. It is also breaking unity build.

Test Plan: Run unity build and see error go away.

Reviewers: rven, igor, IslamAbdelRahman, kradhakrishnan, anthony

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D49281

No need to #ifdef test only code on windows

Split posix storage backend into Env and library

Summary: This patch splits the posix storage backend into Env and
the actual *File implementations. The motivation is to allow other Envs
to use posix as a library. This enables a storage backend different from
posix to split its secondary storage between a normal file system
partition managed by posix, and it own media.

Test Plan: No new functionality is added to posix Env or the library,
thus the current tests should suffice.

Summary: Need to pass through the memtable parameter.

Test Plan: built, tested through myrocks

Reviewers: igor, sdong, rven

Reviewed By: rven

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D49167

Summary:
in tools/db_crashtest.py, cmd_params['db'] by default is a lambda expression, not the actual db_name.
fix by get the db_name before passing it to gen_cmd.

Test Plan: run `make crashtest`

Reviewers: sdong

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D49119

Summary: Use DEBUG_LEVEL=0 in make release and make clean

Test Plan:
make clean
make release -j32

Reviewers: MarkCallaghan, sdong, anthony, IslamAbdelRahman, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D49125

Do not build test only code and unit tests in Release builds

  Test code errors are currently blocking Windows Release builew
  We do not want spend time building in Release what we can not run
  We want to eliminate a source of most frequent errors when people
  check-in test only code which can not be built in Release.
  This feature will work only if you invoke msbuild against rocksdb.sln
  Invoking it against ALL_BUILD target will attempt to build everything.

* add validation is git found
* add validation is .git folder exists in project (project zip archive download without .git folder)
* get head commit SHA if git found and .git folder exists

Tested:
* configure project by CMake 3.0.0 successfully (with and without git), with project zip archive (without .git folder) and with project cloned from github
* configure project by command: cmake -G "Visual Studio 12 Win64"
* build solution by Visual Studio
* manually validate that file utils/build_version.cc contains valid head revision value

Summary: Manual compaction should not fill block cache. Add the verification in unit test

Test Plan: Run the test

Reviewers: yhchiang, kradhakrishnan, rven, IslamAbdelRahman, anthony, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D49089

Summary: dbname => cmd_params['db']

Test Plan: Run `make crash_test`

Reviewers: sdong

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D49077

Summary: Adding the ability to upload logs and db content to storage after the
completion of the job

Test Plan: Manual run

Reviewers:

CC: leveldb@

Task ID: #8754201

Blame Rev:

Summary: Before this diff `arc lint` on non-fb machine issued warnings. Now it doesn't.

Test Plan: `arc lint` is quiet.

Reviewers: yhchiang, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D49071

Summary: crash_test sometimes fails, hitting the add file overlapping assert. Add information in info logs help us to find the bug.

Test Plan: Run all test suites. Do some manual tests to make sure printing is correct.

Reviewers: kradhakrishnan, yhchiang, anthony, IslamAbdelRahman, rven, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D49017

Fix build error using Visual Studio 12