issue 182: provide docx file content as github markdown for easier reading for...

issue 182: provide docx file content as github markdown for easier reading for those without a .docx file viewer.

Redis on Windows Release Notes.md

+MSOpenTech Redis 2.8.17 Release Notes
+=====================================
+
+Welcome to the binary release of Redis from Microsoft Open Technologies, Inc.
+
+What is Redis?
+--------------
+
+Redis is an open source, high performance, key-value store. Values may contain strings, hashes, lists, sets and sorted sets. Redis has been developed primarily for UNIX-like operating systems.
+
+Porting Goals
+-------------
+
+Our goal is to provide a version of Redis that runs on Windows with a performance essentially equal to the performance of Redis on an equivalent UNIX machine.
+
+What is new with the 2.8.17 release
+-----------------------------------
+
+Our last official release was 2.8.12. We have merged in the changes up to 2.8.17. Please see the [release notes for the UNIX 2.8 branch](http://download.redis.io/redis-stable/00-RELEASENOTES) to understand how this impacts Redis functionality.
+
+### Network layer changes
+
+There have been significant changes to the networking layer for this version. Likely there will be a few weeks before there is another official (Chocolatey and Nuget) release. Most of these changes target IPv6.
+
+### persistence-available flag 
+
+If Redis is to be used as an in-memory-only cache without any kind of persistence, then the fork() mechanism used by the background AOF/RDB persistence is unnecessary. As an optimization, all persistence can be turned off in the Windows version of Redis in this scenario. This will disable the creation of the memory mapped heap file, redirect heap allocations to the system heap allocator, and disable commands that would otherwise cause fork() operations: BGSAVE and BGREWRITEAOF. This flag may not be combined with any of the other flags that configure AOF and RDB operations.
+
+persistence-available [(yes)|no]
+
+How to develop for Redis
+------------------------
+
+You will need a client library for accessing Redis. There are a wide variety of client libraries available as listed at <http://redis.io/clients>.
+

Redis on Windows.md

+MSOpenTech’s Redis on Windows
+=============================
+
+We strive to have a stable, functionally equivalent and comparably performing version of Redis on Windows. We have achieved performance nearly identical to the POSIX version running head-to-head on identical hardware across the network. Aside from feature differences that help Redis take advantage of the Windows infrastructure, our version of Redis should work in most situations with the identical setup and configuration that one would use on a POSIX operating system.
+
+How is Redis on Windows implemented?
+====================================
+
+Redis is a C code base that compiles under Visual Studio. Most of the code compiles with only minor changes to the code due to syntactical differences between compilers and low level API differences on Windows. There are a few areas where there are significant differences in how efficient Windows programs operate relative to POSIX programs. We have encapsulated most these differences in a platform specific library. The areas where there are significant differences are:
+
+-   Networking APIs
+
+-   POSIX File Descriptors
+
+-   POSIX fork()
+
+-   Logging
+
+-   Windows Services API
+
+Networking Differences
+----------------------
+
+The Windows networking stack is split between user mode code and kernel mode code. Transitions between user and kernel mode are expensive operations. The POSIX networking APIs on Windows utilize a programming model that incurs significant performance loss due to the kernel/user mode transitions. Efficient Windows networking code instead uses the IO Completion Port model to reduce the impact of this behavior. The APIs used and the programming model for IO Completion is different enough that we were forced to implement a new networking layer in in Redis.
+
+File Descriptors
+----------------
+
+In a POSIX operating system all data sources (files, pipes, sockets, mail slots, etc.) are referenced in code with a handle called a file descriptor. These are low value integers that increment by one with each successive file descriptor created in a given program. All POSIX APIs that work with file descriptors will function without the programmer having to know what kind of data source a file descriptor represents. On Windows generally each kind of data source has a separate kind of HANDLE. APIs that work with one HANDLE type will not work with another kind of HANDLE. In order to make Redis operate with its assumptions about file descriptor values and data source agnosticism, we implemented a Redis File Descriptor API layer.
+
+fork()
+------
+
+The POSIX version of Redis uses the fork() API. There is no equivalent in Windows, and it is an exceedingly difficult API to completely simulate. For most of the uses of fork() we have used Windows specific programming idioms to bypass the need to use a fork()-like API. The one case where we could not do so was with the point-in-time heap snapshot behavior that the Redis persistence model is based on. We tried several different approaches to work around the need for a fork()-like API, but always ran into significant performance penalties and stability issues.
+
+Our current approach is to simulate the point-in-time snapshot behavior aspect of fork() without doing a complete simulation of fork(). We do this with a memory mapped file that contains the Redis heap. When a fork() operation is required we do the following:
+
+-   Mark every page in the memory mapped file with the Copy on Write page protection
+
+-   Start a child process and pass it the handle to the memory mapped file
+
+-   Signal the child to start the AOF or RDB persistence process on the memory shared via the memory mapped file
+
+-   Wait (asynchronously) for the child process to finish
+
+-   Map the changes in the Redis heap that occurred during the fork() operation back into the memory mapped file.
+
+The upside with this implementation is that our performance and stability is now on par with the POSIX version of Redis. The down side is that we have a runtime disk space requirement for Redis equal to the size of the Redis memory mapped heap. The disk space requirement defaults to:
+
+-   The size specified by the –maxheap flag if present, otherwise
+
+-   50% more than the --maxmemory setting if present, otherwise
+
+-   The size of physical RAM
+
+We also have a runtime page file commit requirement that varies depending on the amount data in the Redis heap during the quasi-fork operation. The maximum for this is about 3 times the size of the memory mapped file. This is usually not a problem because the default configuration of Windows allows the page file to grow to 3.5 times the size of physical memory. There are scenarios where 3<sup>rd</sup> party programs also compete for system swap space at runtime.
+
+Logging
+-------
+
+In addition to file based logging, the POSIX version of Redis supports logging via the syslog facility. The equivalent in Windows is the Event Log. With the recent addition of the Windows Service code we have added support for logging to the Event Log. We have mapped the –syslogxxxx flags for this purpose.
+
+Windows Service
+---------------
+
+In version 2.8.9 we are adding support to make Redis operate as a service. See the RedisService.docx file included with the GitHub binary distribution for a description of the service commands available.
+
+Redis on Windows Best Practices
+===============================
+
+Binary Distributions
+--------------------
+
+The GitHub repository should be considered a work in progress until we release the NuGet and Chocolatey packages and tag the repository at that released version.
+
+For instance, the Windows Service feature has taken many iterations with community input to get right. The initial Windows service code was checked in on April 3. Since that time we have added the following to the service based on community input:
+
+-   Preshutdown notification in order to clean up the memory mapped file consistently.
+
+-   Code to identify and clean up orphaned memory mapped files left behind when a machine running Redis as a service crashes or loses power.
+
+-   Self elevation of the Redis executable so that service commands would work from a non-elevated command prompt.
+
+-   Service naming so that multiple instances of the Redis service could be installed on one machine.
+
+-   Automatically adjusting folder permissions so that when Redis is run under the NETWORK SERVICE account it could modify the files in the installation directory.
+
+-   Moved all of the pre-main() error reporting code (service and quasi-fork code) that could write errors to stdout to use the Redis logging code. This allows service initialization errors to reach the Event Log. This required intercepting all of the command line and conf file arguments before main() in order to properly initialize the logging engine. There were several fixes related to the intricacies of how to interpret the arguments passed to Redis.
+
+The final service code has been released as of June 25 in the 2.8.9 Redis-64 packages.
+
+Heap Sizing
+-----------
+
+Native heaps are prone to fragmentation. If we are not able to allocate more heap space due to fragmentation Redis will flag the problem and exit. Unlike the POSIX version of Redis, our heap size is constrained by both by disk space and by swap file space. It is important to consider the how much data you are expecting to put into Redis, and how much fragmentation you are likely to see in the Redis heap. With very high levels of fragmentation the 50% overhead that –maxmemory imposes may not be enough to prevent running out of heap space. In this case, the use of –maxheap will supersede the –maxmemory default heap setting.
+
+Installation and Maintenance
+----------------------------
+
+Since Redis uses system swap space, the most stable configurations will only have Redis running on essentially a virgin operating system install.
+
+Redis is xcopy deployable. There should be no problem upgrading versions by simply copying new binaries over old ones (assuming they are not currently in use).
+
+Service Account
+---------------
+
+When using Redis as a Windows service, the default installation configures Redis to run under the system’s NETWORK SERVICE account. There are some environments where another account must be used (perhaps a domain service account). Configuration of this account needs to be done manually at this point with the service control manager. If this is done, it is also important to give read/write/create permission to the folder that the Redis executable is in to this user identity.
+

Windows Service Documentation.md

+Running Redis as a Service
+==========================
+
+In order to better integrate with the Windows Services model, new command line arguments have been introduced to Redis. These service arguments require an elevated user context in order to connect to the service control manager. If these commands are invoked from a non-elevated context, Redis will attempt to create an elevated context in which to execute these commands. This will cause a User Account Control dialog to be displayed by Windows and may require Administrative user credentials in order to proceed.
+
+Installing the Service
+----------------------
+
+*--service-install*
+
+This must be the first argument on the redis-server command line. Arguments after this are passed in the order they occur to Redis when the service is launched. The service will be configured as Autostart and will be launched as "NT AUTHORITY\\NetworkService". Upon successful installation a success message will be displayed and Redis will exit.
+
+This command does not start the service.
+
+For instance:
+
+redis-server --service-install redis.windows.conf --loglevel verbose
+
+Uninstalling the Service
+------------------------
+
+*--service-uninstall*
+
+This will remove the Redis service configuration information from the registry. Upon successful uninstallation a success message will be displayed and Redis will exit.
+
+This does command not stop the service.
+
+For instance:
+
+redis-server --service-uninstall
+
+Starting the Service
+--------------------
+
+*--service-start*
+
+This will start the Redis service. Upon successful start, a success message will be displayed and Redis will begin running.
+
+For instance:
+
+redis-server --service-start
+
+Stopping the Service
+--------------------
+
+*--service-stop*
+
+This will stop the Redis service. Upon successful termination a success message will be displayed and Redis will exit.
+
+For instance:
+
+redis-server --service-stop
+
+Naming the Service
+------------------
+
+*--service-name **name***
+
+This optional argument may be used with any of the preceding commands to set the name of the installed service. This argument should follow the service-install, service-start, service-stop or service-uninstall commands, and precede any arguments to be passed to Redis via the service-install command.
+
+The following would install and start three separate instances of Redis as a service:
+
+redis-server --service-install –service-name redisService1 –port 10001
+
+redis-server --service-start –service-name redisService1
+
+redis-server --service-install –service-name redisService2 –port 10002
+
+redis-server --service-start –service-name redisService2
+
+redis-server --service-install –service-name redisService3 –port 10003
+
+redis-server --service-start –service-name redisService3
+

msvs/ReleasePackagingTool/Program.cs

@@ -6,6 +6,7 @@ using System.IO;
 using System.IO.Compression;
 using System.Xml;
 using System.Reflection;
+using System.Diagnostics;
 
 namespace ReleasePackagingTool
 {
@@ -30,6 +31,7 @@ namespace ReleasePackagingTool
                 p.UpdateReleaseNotes(version);
                 p.UpdateNuSpecFiles(version);
                 p.BuildReleasePackage(version);
+                p.DocxToMd();
                 Console.Write("Release packaging complete.");
                 Environment.ExitCode = 0;
             }
@@ -49,9 +51,43 @@ namespace ReleasePackagingTool
             return line.Substring(start + 1, last - start - 1);
         }
 
+        void DocxToMd()
+        {
+            // locate converter (pandoc v1.13.0+) 
+            string pandocToolPath = Path.Combine(rootPath, @"msvs\tools\pandoc\pandoc.exe");
+            if (File.Exists(pandocToolPath))
+            {
+                var files = Directory.EnumerateFiles(Path.Combine(rootPath, @"msvs\setups\documentation"), "*.docx");
+                foreach (string sourceFile in files)
+                {
+                    string fileName = Path.GetFileName(sourceFile);
+                    string destFile = Path.ChangeExtension(Path.Combine(rootPath, fileName), "md");
+                    System.Diagnostics.Process p = new Process();
+                    p.StartInfo.FileName = pandocToolPath;
+                    p.StartInfo.UseShellExecute = false;
+                    p.StartInfo.Arguments = string.Format(
+                                                "-f {0} -t {1} -o \"{2}\" \"{3}\"",
+                                                "docx",
+                                                "markdown_github",
+                                                destFile,
+                                                sourceFile);
+                    p.Start();
+                    p.WaitForExit();
+                    if (p.ExitCode != 0)
+                    {
+                        Console.WriteLine("conversion of'{0}' to '{1}' failed.", sourceFile, destFile);
+                    }
+                }
+            }
+            else
+            {
+                Console.WriteLine("pandoc tool not found. docx-->md conversion will not take place.");
+            }
+        }
+
         void UpdateReleaseNotes(string redisVersion)
         {
-            string releaseNotesPath = Path.Combine(rootPath, @"msvs\setups\documentation\Redis Release Notes.docx");
+            string releaseNotesPath = Path.Combine(rootPath, @"msvs\setups\documentation\Redis on Windows Release Notes.docx");
             string templatePath = Path.Combine(rootPath, @"msvs\setups\documentation\templates\Redis Release Notes Template.docx");
 
             ForceFileErase(releaseNotesPath);
@@ -134,8 +170,8 @@ namespace ReleasePackagingTool
             List<string> docuementNames = new List<string>()
             {
                 "Redis on Windows.docx",
-                "Redis Release Notes.docx",
-                "RedisService.docx",
+                "Redis on Windows Release Notes.docx",
+                "Windows Service Documentation.docx",
                 "redis.windows.conf"
             };
 

msvs/tools/pandoc/pandoc tool.txt

+We convert the .docx files to github markdown for easier viewing for users without a .docx viewer. The is done with the pandoc tool (1.13.0+) from https://github.com/jgm/pandoc. In order to avoid the licensing issues with this package, we are making this an optional part of the build. If you want to modify the source .docx files, you should get the pandoc tool and place it in this directory. 
\ No newline at end of file