Better management of unreleased HISTORY (#11481)

Summary:
See new NOTE in HISTORY.md and unreleased_history/README.txt

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11481

Test Plan: some manual testing on my CentOS 8 system

Reviewed By: jaykorean

Differential Revision: D46233342

Pulled By: pdillinger

fbshipit-source-id: daf59cf3dc907f450b469090dcc481a30a7d7c0d

HISTORY.md

 # Rocksdb Change Log
-## Unreleased
-### New Features
-* Add a new option OptimisticTransactionDBOptions::shared_lock_buckets that enables sharing mutexes for validating transactions between DB instances, for better balancing memory efficiency and validation contention across DB instances. Different column families and DBs also now use different hash seeds in this validation, so that the same set of key names will not contend across DBs or column families.
-* Add `WriteBatch::Release()` that releases the batch's serialized data to the caller.
-* Add an API NewTieredVolatileCache() in include/rocksdb/cache.h to allocate an instance of a block cache with a primary block cache tier and a compressed secondary cache tier. A cache of this type distributes memory reservations against the block cache, such as WriteBufferManager, table reader memory etc., proportionally across both the primary and compressed secondary cache.
-
-### Public API Changes
-* Add `WaitForCompact()` to wait for all flush and compactions jobs to finish. Jobs to wait include the unscheduled (queued, but not scheduled yet).
-
-### Behavior changes
-* Statistics `rocksdb.sst.read.micros` scope is expanded to all SST reads except for file ingestion and column family import (some compaction reads were previously excluded).
+> NOTE: Entries for next release do not go here. Follow instructions in `unreleased_history/README.txt`
 
 ## 8.3.0 (05/19/2023)
 ### New Features

unreleased_history/README.txt

+Adding release notes
+--------------------
+
+When adding release notes for the next release, add a file to one of these
+directories:
+
+unreleased_history/new_features
+unreleased_history/behavior_changes
+unreleased_history/public_api_changes
+unreleased_history/bug_fixes
+
+with a unique name that makes sense for your change, preferably using the .md
+extension for syntax highlighting.
+
+There is a script to help, as in
+
+$ unreleased_history/add.sh unreleased_history/bug_fixes/crash_in_feature.md
+
+or simply
+
+$ unreleased_history/add.sh
+
+will take you through some prompts.
+
+The file should usually contain one line of markdown, and "* " is not
+required, as it will automatically be inserted later if not included at the
+start of the first line in the file. Extra newlines or missing trailing
+newlines will also be corrected.
+
+The only times release notes should be added directly to HISTORY are if
+* A release is being amended or corrected after it is already "cut" but not
+tagged, which should be rare.
+* A single commit contains a noteworthy change and a patch release version bump
+
+
+Ordering of entries
+-------------------
+
+Within each group, entries will be included using ls sort order, so important
+entries could start their file name with a small three digit number like
+100pretty_important.md.
+
+The ordering of groups such as new_features vs. public_api_changes is
+hard-coded in unreleased_history/release.sh
+
+
+Updating HISTORY.md with release notes
+--------------------------------------
+
+The script unreleased_history/release.sh does this. Run the script before
+updating version.h to the next develpment release, so that the script will pick
+up the version being released. You might want to start with
+
+$ DRY_RUN=1 unreleased_history/release.sh | less
+
+to check for problems and preview the output. Then run
+
+$ unreleased_history/release.sh
+
+which will git rm some files and modify HISTORY.md. You still need to commit the
+changes, or revert with the command reported in the output.
+
+
+Why not update HISTORY.md directly?
+-----------------------------------
+
+First, it was common to hit unnecessary merge conflicts when adding entries to
+HISTORY.md, which slowed development. Second, when a PR was opened before a
+release cut and landed after the release cut, it was easy to add the HISTORY
+entry to the wrong version's history. This new setup completely fixes both of
+those issues, with perhaps slighly more initial work to create each entry.
+There is also now an extra step in using `git blame` to map a release note
+to its source code implementation, but that is a relatively rare operation.

unreleased_history/add.sh

+#! /usr/bin/env bash
+
+set -e
+set -o pipefail
+
+if [ "$1" ]; then
+  # Target file specified on command line
+  TARGET="$1"
+else
+  # Interactively choose a group and file name
+  DIRS="`find unreleased_history/ -mindepth 1 -maxdepth 1 -type d`"
+  echo "Choose a group for new release note:"
+  echo "$DIRS" | grep -nEo '[^/]+$'
+  echo -n "Enter a number: "
+  while [ ! "$DIRNUM" ]; do read -r DIRNUM; done
+  DIR="$(echo "$DIRS" | head -n "$DIRNUM" | tail -1)"
+  echo "Choose a file name for new release note (e.g. improved_whatever.md):"
+  while [ ! "$FNAME" ]; do read -re FNAME; done
+  # Replace spaces with underscores
+  TARGET="$(echo "$DIR/$FNAME" | tr ' ' '_')"
+fi
+
+# Edit/create the file
+${EDITOR:-nano} "$TARGET"
+# Add to version control (easy to forget!)
+git add "$TARGET"

unreleased_history/behavior_changes/sst_read_micros.md

+* Statistics `rocksdb.sst.read.micros` scope is expanded to all SST reads except for file ingestion and column family import (some compaction reads were previously excluded).

unreleased_history/new_features/shared_lock_buckets.md

+* Add a new option OptimisticTransactionDBOptions::shared_lock_buckets that enables sharing mutexes for validating transactions between DB instances, for better balancing memory efficiency and validation contention across DB instances. Different column families and DBs also now use different hash seeds in this validation, so that the same set of key names will not contend across DBs or column families.

unreleased_history/new_features/tiered_volatile_cache.md

+Add an API NewTieredVolatileCache() in include/rocksdb/cache.h to allocate an instance of a block cache with a primary block cache tier and a compressed secondary cache tier. A cache of this type distributes memory reservations against the block cache, such as WriteBufferManager, table reader memory etc., proportionally across both the primary and compressed secondary cache.

unreleased_history/new_features/wait_for_compact.md

+Add `WaitForCompact()` to wait for all flush and compactions jobs to finish. Jobs to wait include the unscheduled (queued, but not scheduled yet).

unreleased_history/new_features/writebatchrelease.md

+Add `WriteBatch::Release()` that releases the batch's serialized data to the caller.

unreleased_history/release.sh

+#! /usr/bin/env bash
+
+set -e
+
+if [ ! -d unreleased_history ]; then
+  echo "Can't find unreleased_history/ directory"
+  exit 1
+fi
+
+GIT_PATHS="unreleased_history/ HISTORY.md"
+if [ ! "$DRY_RUN" ]; then
+  # Check for uncommitted changes
+  UNCOMMITTED="$(git diff -- $GIT_PATHS)"
+  if [ "$UNCOMMITTED" ]; then
+    echo 'Uncommitted changes to files to be modified. Please commit first to'
+    echo 'ensure a clean revert path. You can always `git commit -a --amend`'
+    echo 'to add more changes to your commit.'
+    exit 2
+  fi
+fi
+
+# Add first part of existing HISTORY file to new version
+awk '{ print } /NOTE/ { exit(0) }' < HISTORY.md > HISTORY.new
+
+# And a blank line separator
+echo >> HISTORY.new
+
+# Add new version header
+awk '/#define ROCKSDB_MAJOR/ { major = $3 }
+     /#define ROCKSDB_MINOR/ { minor = $3 }
+     /#define ROCKSDB_PATCH/ { patch = $3 }
+     END { print "## " major "." minor "." patch " (" strftime("%F") ")" }' < include/rocksdb/version.h >> HISTORY.new
+
+function process_file () {
+  # use awk to correct extra or missing newlines, missing '* ' on first line
+  awk '/./ { if (notfirstline || $1 == "*") print;
+             else print "* " $0;
+             notfirstline=1; }' < $1 >> HISTORY.new
+  echo git rm $1
+  if [ ! "$DRY_RUN" ]; then
+    git rm $1
+  fi
+}
+
+PROCESSED_DIRECTORIES=""
+
+function process_dir () {
+  PROCESSED_DIRECTORIES="$PROCESSED_DIRECTORIES $1"
+  # ls will sort the files, including the permanent header file
+  FILES="$(ls unreleased_history/$1/)"
+  if [ "$FILES" ]; then
+    echo "### $2" >> HISTORY.new
+    for FILE in $FILES; do
+      process_file "unreleased_history/$1/$FILE"
+    done
+    echo >> HISTORY.new
+    echo "Saved entries from $1"
+  else
+    echo "Nothing new in $1"
+  fi
+}
+
+# Process dirs and files
+process_dir new_features "New Features"
+process_dir public_api_changes "Public API Changes"
+process_dir behavior_changes "Behavior Changes"
+process_dir bug_fixes "Bug Fixes"
+
+# Check for unexpected files or dirs at top level. process_dir/process_file
+# will deal with contents of these directories
+EXPECTED_REGEX="[^/]*[.]sh|README[.]txt|$(echo $PROCESSED_DIRECTORIES | tr ' ' '|')"
+UNEXPECTED="$(find unreleased_history/ -mindepth 1 -maxdepth 1 -regextype egrep -not -regex "[^/]*/($EXPECTED_REGEX)")"
+if [ "$UNEXPECTED" ]; then
+  echo "Unexpected files I don't know how to process:"
+  echo "$UNEXPECTED"
+  rm HISTORY.new
+  exit 3
+fi
+
+# Add rest of existing HISTORY file to new version (collapsing newlines)
+awk '/./ { if (note) pr=1 }
+     /NOTE/ { note=1 }
+     { if (pr) print }' < HISTORY.md >> HISTORY.new
+
+if [ "$DRY_RUN" ]; then
+  echo '==========================================='
+  diff -U3 HISTORY.md HISTORY.new || true
+  rm HISTORY.new
+else
+  mv HISTORY.new HISTORY.md
+  echo "Done. Revert command: git checkout HEAD -- $GIT_PATHS"
+fi