7003845: README-builds document proper location of forest extension, provide alternatives

Reviewed-by: robilad

README

 README:
-  This file should be located at the top of the OpenJDK Mercurial repository
-  forest. This top or enclosing repository will include a "make" directory,
-  and a Makefile at the very top of the repository.
-  It should also include the 6 repositories: "jdk", "hotspot", "langtools",
-  "corba", "jaxws"  and "jaxp".
+  This file should be located at the top of the OpenJDK Mercurial root
+  repository. This root repository will include a "make" directory,
+  and a Makefile for building the entire OpenJDK.
+  A full OpenJDK repository set (forest) should also include the following
+  6 nested repositories:
+    "jdk", "hotspot", "langtools", "corba", "jaxws"  and "jaxp".
+  There are also several source downloads for the jax* repositories that
+  will be needed.
+
+  This one root repository can be obtained with something like:
+    hg clone http://hg.openjdk.java.net/jdk7/jdk7 openjdk7
+  To make sure you have all the nested repositories, you can run:
+    cd openjdk7 && sh ./get_source.sh
+  (This is identical to using the Mercurial Forest Extension command 
+   'hg fclone http://hg.openjdk.java.net/jdk7/jdk7 openjdk7').
+  People unfamiliar with Mercurial should read the first few chapters of
+  the Mercurial book: http://hgbook.red-bean.com/read/
 
   See http://openjdk.java.net/ for more information about the OpenJDK.
 
 Simple Build Instructions:
   
-  1. Download and install a JDK 6 from
-     http://java.sun.com/javase/downloads/index.jsp
-     Set the environment variable ALT_BOOTDIR to the location of this JDK 6.
+  0. Get the necessary system software/packages installed on your system, see
+     http://hg.openjdk.java.net/jdk7/build/raw-file/tip/README-builds.html
 
-  2. Download and install the Binary Plugs for the most recent JDK7 from
-     http://download.java.net/openjdk/jdk7/
-     Set the environment variable ALT_BINARY_PLUGS_PATH to the location of
-     these binary plugs.
+  1. If you don't have a jdk6 installed, download and install a JDK 6 from
+     http://java.sun.com/javase/downloads/index.jsp
+     Set the environment variable ALT_BOOTDIR to the location of JDK 6.
 
-  3. Check the sanity of doing a build with the current machine:
+  2. Check the sanity of doing a build with your current system:
        gnumake sanity
      See README-builds.html if you run into problems.
   
-  4. Do a complete build of the jdk:
+  3. Do a complete build of the OpenJDK:
        gnumake all
      The resulting JDK image should be found in build/*/j2sdk-image
 

README-builds.html

@@ -37,6 +37,11 @@
         <blockquote>
             <ul>
                 <li><a href="#introduction">Introduction</a></li>
+                <li><a href="#hg">Use of Mercurial</a>
+                    <ul>
+                        <li><a href="#get_source">Getting the Source</a></li>
+                    </ul>
+                </li>
                 <li><a href="#MBE">Minimum Build Environments</a></li>
                 <li><a href="#SDBE">Specific Developer Build Environments</a>
                     <ul>
@@ -56,7 +61,7 @@
                         <li><a href="#linux">Basic Linux System Setup</a> </li>
                         <li><a href="#solaris">Basic Solaris System Setup</a> </li>
                         <li><a href="#windows">Basic Windows System Setup</a> </li>
-                        <li><a href="#dependencies">Build Dependencies</a> </li>
+                        <li><a href="#dependencies">Build Dependencies</a>
                             <ul>
                                 <li><a href="#bootjdk">Bootstrap JDK</a> </li>
                                 <li><a href="#binaryplugs">Binary Plugs</a> </li>
@@ -92,6 +97,7 @@
                                     </ul>
                                 </li>
                             </ul>
+                        </li>
                     </ul>
                 </li>
                 <li><a href="#creating">Creating the Build</a> </li>
@@ -100,6 +106,76 @@
                 <li><a href="#troubleshooting">Troubleshooting</a></li>
             </ul>
         </blockquote>
+
+        <!-- ------------------------------------------------------ -->
+        <hr>
+        <h2><a name="hg">Use of Mercurial</a></h2>
+        <blockquote>
+            The OpenJDK sources are maintained with the revision control system
+            <a href="http://mercurial.selenic.com/wiki/Mercurial">Mercurial</a>.
+            If you are new to Mercurial, please see the
+            <a href="http://mercurial.selenic.com/wiki/BeginnersGuides">Beginner Guides</a>
+            or refer to the <a href=""http://hgbook.red-bean.com/">Mercurial Book</a>.
+            The first few chapters of the book provide an excellent overview of
+            Mercurial, what it is and how it works.
+            <br>
+            For using Mercurial with the OpenJDK refer to the
+            <a href=""http://openjdk.java.net/guide/repositories.html#installConfig">
+               Developer Guide: Installing and Configuring Mercurial</a>
+            section for more information.
+            The Forest Extension is not part of the Mercurial install,
+            and is optional,
+            but can be obtained with the following commands:
+            <blockquote>
+                <tt>
+                    hg clone https://bitbucket.org/pmezard/hgforest-crew/overview/ <i>YourHgForest</i>
+                </tt>
+            </blockquote>
+            Once you have the file <tt>forest.py</tt>, you need to add these
+            lines to your <tt>${HOME}/.hgrc</tt> file:
+            <blockquote>
+                <tt>
+                    [extensions]
+                    <br>forest = <i>YourHgForest</i>/forest.py
+                </tt>
+            </blockquote>
+
+            <!-- ------------------------------------------------------ -->
+            <h3><a name="get_source">Getting the Source</a></h3>
+            <blockquote>
+                To get the entire set of OpenJDK Mercurial repositories
+                using the Forest Extension:
+                <blockquote>
+                    <tt>
+                        hg fclone http://openjdk.java.net/jdk7/jdk7 <i>YourOpenJDK</i>
+                    </tt>
+                </blockquote>
+                To get the entire set of OpenJDK Mercurial repositories
+                without using the Forest Extension:
+                <blockquote>
+                    <tt>
+                        hg clone http://openjdk.java.net/jdk7/jdk7 <i>YourOpenJDK</i>
+                        <br>cd <i>YourOpenJDK</i>
+                        <br>sh ./get_source.sh
+                    </tt>
+                </blockquote>
+                Once you have all the repositories, the
+                script <tt>make/scripts/hgforest.sh</tt>
+                can be used to repeat the same <tt>hg</tt>
+                command on every repository in the forest, e.g.
+                <blockquote>
+                    <tt>
+                        cd <i>YourOpenJDK</i>
+                        <br>sh ./make/scripts/hgforest.sh pull -u
+                    </tt>
+                </blockquote>
+                You may find this script <tt>make/scripts/hgforest.sh</tt> faster
+                than the <tt>hg</tt> forest commands provided by the
+                Forest Extension.
+            </blockquote>
+
+        </blockquote>
+
         <!-- ------------------------------------------------------ -->
         <hr>
         <h2><a name="MBE">Minimum Build Environments</a></h2>
@@ -1018,20 +1094,20 @@
                 </blockquote>
                 <strong><a name="msvc32">Windows i586: Microsoft Visual Studio 2010 Compilers</a></strong>
                 <blockquote>
-<p>
-<b>BEGIN WARNING</b>: At this time (Spring/Summer 2010) JDK 7 is starting a transition to
-use the newest VS2010 Microsoft compilers. These build instructions are updated
-to show where we are going. We have a QA process to go through before
-official builds actually use VS2010. So for now, official builds are
-still using VS2003. No other compilers are known to build the entire JDK,
-including non-open portions.
-So for now you should be able to build with either VS2003 or VS2010.
-We do not guarantee that VS2008 will work, although there is sufficient
-makefile support to make at least basic JDK builds plausible.
-Visual Studio 2010 Express compilers are now able to build all the
-open source repositories, but this is 32 bit only. To build 64 bit
-Windows binaries use the the 7.1 Windows SDK.<b>END WARNING.</b>
-<p>
+                    <p>
+                        <b>BEGIN WARNING</b>: At this time (Spring/Summer 2010) JDK 7 is starting a transition to
+                        use the newest VS2010 Microsoft compilers. These build instructions are updated
+                        to show where we are going. We have a QA process to go through before
+                        official builds actually use VS2010. So for now, official builds are
+                        still using VS2003. No other compilers are known to build the entire JDK,
+                        including non-open portions.
+                        So for now you should be able to build with either VS2003 or VS2010.
+                        We do not guarantee that VS2008 will work, although there is sufficient
+                        makefile support to make at least basic JDK builds plausible.
+                        Visual Studio 2010 Express compilers are now able to build all the
+                        open source repositories, but this is 32 bit only. To build 64 bit
+                        Windows binaries use the the 7.1 Windows SDK.<b>END WARNING.</b>
+                    <p>
                         The 32-bit OpenJDK Windows build
                         requires
                         Microsoft Visual Studio C++ 2010 (VS2010) Professional
@@ -1071,20 +1147,20 @@ Windows binaries use the the 7.1 Windows SDK.<b>END WARNING.</b>
                 <strong><a name="mssdk64">Windows x64: Microsoft Windows 7.1 SDK 64 bit compilers.</a></strong>
                 For a free alternative for 64 bit builds, use the 7.1 SDK.
                 Microsoft say that to set up your paths for this run
-<pre>
+                <pre>
     c:\Program Files\Microsoft SDKs\Windows\v7.1\bin\setenv.cmd /x64.
-</pre>
+                </pre>
                 What was tested is just directly setting up LIB, INCLUDE,
                 PATH and based on the installation directories using the
                 DOS short name appropriate for the system, (you will
                 need to set them for yours, not just blindly copy this) eg :
-<pre>
+                <pre>
     set VSINSTALLDIR=c:\PROGRA~2\MICROS~1.0
     set WindowsSdkDir=c:\PROGRA~1\MICROS~1\Windows\v7.1
     set PATH=%VSINSTALLDIR%\vc\bin\amd64;%VSINSTALLDIR%\Common7\IDE;%WindowsSdkDir%\bin;%PATH%
     set INCLUDE=%VSINSTALLDIR%\vc\include;%WindowsSdkDir%\include
     set LIB=%VSINSTALLDIR%\vc\lib\amd64;%WindowsSdkDir%\lib\x64
-</pre>
+                </pre>
             </blockquote>
             <!-- ------------------------------------------------------ --> 
             <h4><a name="zip">Zip and Unzip</a></h4>
@@ -1450,16 +1526,18 @@ Windows binaries use the the 7.1 Windows SDK.<b>END WARNING.</b>
             The default output directory is
             <tt>build/<i>platform</i></tt>,
             where <tt><i>platform</i></tt> is one of
-            <tt><ul>
-                    <li>solaris-sparc</li>
-                    <li>solaris-sparcv9</li>
-                    <li>solaris-i586</li>
-                    <li>solaris-amd64</li>
-                    <li>linux-i586</li>
-                    <li>linux-amd64</li>
-                    <li>windows-i586</li>
-                    <li>windows-amd64</li>
-            </ul></tt>
+            <blockquote>
+                <ul>
+                    <li><tt>solaris-sparc</tt></li>
+                    <li><tt>solaris-sparcv9</tt></li>
+                    <li><tt>solaris-i586</tt></li>
+                    <li><tt>solaris-amd64</tt></li>
+                    <li><tt>linux-i586</tt></li>
+                    <li><tt>linux-amd64</tt></li>
+                    <li><tt>windows-i586</tt></li>
+                    <li><tt>windows-amd64</tt></li>
+                </ul>
+            </blockquote>
             In particular, the 
             <tt>build/<i>platform</i>/j2sdk-image/bin</tt>
             directory should contain executables for the 
@@ -1781,18 +1859,15 @@ Windows binaries use the the 7.1 Windows SDK.<b>END WARNING.</b>
                         performance reasons.
                         <p>
                             To completely disable SELinux:
-                        <tt>
                         <ol>
-                                <li>$ su root</li>
-                                <li># system-config-securitylevel</li>
-                                <li>In the window that appears, select the SELinux tab</li>
-                                <li>Disable SELinux</li>
+                            <li><tt>$ su root</tt></li>
+                            <li><tt># system-config-securitylevel</tt></li>
+                            <li><tt>In the window that appears, select the SELinux tab</tt></li>
+                            <li><tt>Disable SELinux</tt></li>
                         </ol>
-                        </tt>
                         <p>
                             Alternatively, instead of completely disabling it you could
                             disable just this one check.
-                        <tt>
                         <ol>
                             <li>Select System->Administration->SELinux Management</li>
                             <li>In the SELinux Management Tool which appears,
@@ -1801,7 +1876,6 @@ Windows binaries use the the 7.1 Windows SDK.<b>END WARNING.</b>
                             <li>Check the first item, labeled
                                 "Allow all unconfined executables to use libraries requiring text relocation ..."</li>
                         </ol>
-                        </tt>
                     </blockquote>
                 </li>
                 <li>

get_source.sh

+#!/bin/sh
+
+#
+# Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
+# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
+#
+# This code is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 2 only, as
+# published by the Free Software Foundation.  Oracle designates this
+# particular file as subject to the "Classpath" exception as provided
+# by Oracle in the LICENSE file that accompanied this code.
+#
+# This code is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+# version 2 for more details (a copy is included in the LICENSE file that
+# accompanied this code).
+#
+# You should have received a copy of the GNU General Public License version
+# 2 along with this work; if not, write to the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
+# or visit www.oracle.com if you need additional information or have any
+# questions.
+#
+
+# Get clones of all nested repositories
+sh ./make/scripts/hgforest.sh clone
+
+# Update all existing repositories to the latest sources
+sh ./make/scripts/hgforest.sh pull -u
+

make/scripts/hgforest.sh

+#!/bin/sh
+
+#
+# Copyright (c) 2009, 2010, Oracle and/or its affiliates. All rights reserved.
+# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
+#
+# This code is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 2 only, as
+# published by the Free Software Foundation.
+#
+# This code is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+# version 2 for more details (a copy is included in the LICENSE file that
+# accompanied this code).
+#
+# You should have received a copy of the GNU General Public License version
+# 2 along with this work; if not, write to the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
+# or visit www.oracle.com if you need additional information or have any
+# questions.
+#
+
+# Shell script for a fast parallel forest command
+
+tmp=/tmp/forest.$$
+rm -f -r ${tmp}
+mkdir -p ${tmp}
+
+# Remove tmp area on A. B. Normal termination
+trap 'rm -f -r ${tmp}' KILL
+trap 'rm -f -r ${tmp}' EXIT
+
+# Only look in specific locations for possible forests (avoids long searches)
+pull_default=""
+if [ "$1" = "clone" -o "$1" = "fclone" ] ; then
+  subrepos="corba jaxp jaxws langtools jdk hotspot"
+  if [ -f .hg/hgrc ] ; then
+    pull_default=`hg paths default`
+  fi
+  if [ "${pull_default}" = "" ] ; then
+    echo "ERROR: Need initial clone with 'hg paths default' defined"
+    exit 1
+  fi
+  repos=""
+  for i in ${subrepos} ; do
+    if [ ! -f ${i}/.hg/hgrc ] ; then
+      repos="${repos} ${i}"
+    fi
+  done
+  at_a_time=2
+else
+  hgdirs=`ls -d ./.hg ./*/.hg ./*/*/.hg ./*/*/*/.hg ./*/*/*/*/.hg 2>/dev/null`
+  # Derive repository names from the .hg directory locations
+  repos=""
+  for i in ${hgdirs} ; do
+    repos="${repos} `echo ${i} | sed -e 's@/.hg$@@'`"
+  done
+  at_a_time=8
+fi
+
+# Any repos to deal with?
+if [ "${repos}" = "" ] ; then
+  echo "No repositories to process."
+  exit
+fi
+
+# Echo out what repositories we will process
+echo "# Repos: ${repos}"
+
+# Run the supplied command on all repos in parallel, save output until end
+n=0
+for i in ${repos} ; do
+  echo "Starting on ${i}"
+  n=`expr ${n} '+' 1`
+  (
+    (
+      if [ "$1" = "clone" -o "$1" = "fclone" ] ; then
+        cline="hg $* ${pull_default}/${i} ${i}"
+        echo "# ${cline}"
+        ( eval "${cline}" )
+      else
+        cline="hg $*"
+        echo "# cd ${i} && ${cline}"
+        ( cd ${i} && eval "${cline}" )
+      fi
+      echo "# exit code $?"
+    ) > ${tmp}/repo.${n} 2>&1 ; cat ${tmp}/repo.${n} ) &
+  if [ `expr ${n} '%' ${at_a_time}` -eq 0 ] ; then
+    sleep 5
+  fi
+done
+
+# Wait for all hg commands to complete
+wait
+
+# Cleanup
+rm -f -r ${tmp}
+
+# Terminate with exit 0 all the time (hard to know when to say "failed")
+exit 0
+