8011675: adding compilation level to replay data

Reviewed-by: kvn, vlivanov

8011675: adding compilation level to replay data
Reviewed-by: kvn, vlivanov
a9a37533 · iignatyev · 9d5b10bb · a9a37533 · a9a37533 · a9a37533
14 changed file
--- a/agent/doc/c2replay.html
+++ b/agent/doc/c2replay.html
 <html>
 <head>
 <title>
-C2 Replay
+Replay
 </title>
 </head>
 <body>

-<h1>C2 compiler replay</h1>
+<h1>Compiler replay</h1>
 <p>
-The C2 compiler replay is a function to repeat the compiling process from a crashed java process in compiled method<br>
+The compiler replay is a function to repeat the compiling process from a crashed java process in compiled method<br>
 This function only exists in debug version of VM
 </p>
 <h2>Usage</h2>
 <pre>
-First, use SA to attach to the core file, if suceeded, do
-       clhsdb>dumpreplaydata <address> | -a | <thread_id> [> replay.txt]
+First, use SA to attach to the core file, if succeeded, do
+       hsdb&gt; dumpreplaydata &lt;address&gt; | -a | &lt;thread_id&gt; [&gt; replay.txt]
       create file replay.txt, address is address of Method, or nmethod(CodeBlob)
-       clhsdb>buildreplayjars [all | boot | app]
+       hsdb&gt; buildreplayjars [all | boot | app]
       create files:
         all:
           app.jar, boot.jar
@@ -26,16 +26,16 @@ First, use SA to attach to the core file, if suceeded, do
           app.jar
       exit SA now.
 Second, use the obtained replay text file, replay.txt and jar files, app.jar and boot.jar, using debug version of java
-       java -Xbootclasspath/p:boot.jar -cp app.jar -XX:ReplayDataFile=<datafile> -XX:+ReplayCompiles ....
+       java -Xbootclasspath/p:boot.jar -cp app.jar -XX:ReplayDataFile=&lt;datafile&gt; -XX:+ReplayCompiles ....
       This will replay the compiling process.

       With ReplayCompiles, the replay will recompile all the methods in app.jar, and in boot.jar to emulate the process in java app.

 notes:
       1) Most time, we don't need the boot.jar which is the classes loaded from JDK. It will be only modified when an agent(JVMDI) is running and modifies the classes.
-       2) If encounter error as "<flag>" not found, that means the SA is using a VMStructs which is different from the one with corefile. In this case, SA has a utility tool vmstructsdump which is located at agent/src/os/<os>/proc/<os_platform>
+       2) If encounter error as "&lt;flag&gt;" not found, that means the SA is using a VMStructs which is different from the one with corefile. In this case, SA has a utility tool vmstructsdump which is located at agent/src/os/&lt;os&gt;/proc/&lt;os_platform&gt;

       Use this tool to dump VM type library:
-       vmstructsdump libjvm.so > <type_name>.db
+       vmstructsdump libjvm.so &gt; &lt;type_name&gt;.db

-       set env SA_TYPEDB=<type_name>.db (refer different shell for set envs)
+       set env SA_TYPEDB=&lt;type_name&gt;.db (refer different shell for set envs)
--- a/agent/doc/clhsdb.html
+++ b/agent/doc/clhsdb.html
@@ -15,7 +15,7 @@ GUI tools. Command line HSDB (CLHSDB) tool is alternative to SA GUI tool HSDB.
 <p>
 There is also JavaScript based SA command line interface called <a href="jsdb.html">jsdb</a>.
 But, CLHSDB supports Unix shell-like (or dbx/gdb-like) command line interface with
-support for output redirection/appending (familiar >, >>), command history and so on.
+support for output redirection/appending (familiar &gt;, &gt;&gt;), command history and so on.
 Each CLHSDB command can have zero or more arguments and optionally end with output redirection
 (or append) to a file. Commands may be stored in a file and run using <b>source</b> command.
 <b>help</b> command prints usage message for all supported commands (or a specific command)
@@ -49,7 +49,7 @@ Available commands:
  dumpheap [ file ] <font color="red">dump heap in hprof binary format</font>
  dumpideal -a | id <font color="red">dump ideal graph like debug flag -XX:+PrintIdeal</font>
  dumpilt -a | id <font color="red">dump inline tree for C2 compilation</font>
-  dumpreplaydata <address> | -a | <thread_id> [>replay.txt] <font color="red">dump replay data into a file</font>
+  dumpreplaydata &lt;address&gt; | -a | &lt;thread_id&gt; [&gt;replay.txt] <font color="red">dump replay data into a file</font>
  echo [ true | false ] <font color="red">turn on/off command echo mode</font>
  examine [ address/count ] | [ address,address] <font color="red">show contents of memory from given address</font>
  field [ type [ name fieldtype isStatic offset address ] ] <font color="red">print info about a field of HotSpot type</font>
@@ -127,11 +127,11 @@ hsdb&gt; jsload test.js
 </code>
 </pre>

-<h3>C2 Compilation Replay</h3>
+<h3>Compilation Replay</h3>
 <p>
 When a java process crashes in compiled method, usually a core file is saved.
-The C2 replay function can reproduce the compiling process in the core.
-<a href="c2replay.html">c2replay.html</a>
+The replay function can reproduce the compiling process in the core.
+<a href="cireplay.html">cireplay.html</a>

 </body>
 </html>
--- a/agent/src/share/classes/sun/jvm/hotspot/ci/ciEnv.java
+++ b/agent/src/share/classes/sun/jvm/hotspot/ci/ciEnv.java
@@ -93,10 +93,11 @@ public class ciEnv extends VMObject {
    CompileTask task = task();
    Method method = task.method();
    int entryBci = task.osrBci();
+    int compLevel = task.compLevel();
    Klass holder = method.getMethodHolder();
    out.println("compile " + holder.getName().asString() + " " +
                OopUtilities.escapeString(method.getName().asString()) + " " +
                method.getSignature().asString() + " " +
-                entryBci);
+                entryBci + " " + compLevel);
  }
 }
--- a/agent/src/share/classes/sun/jvm/hotspot/code/NMethod.java
+++ b/agent/src/share/classes/sun/jvm/hotspot/code/NMethod.java
@@ -78,6 +78,8 @@ public class NMethod extends CodeBlob {
      current sweep traversal index. */
  private static CIntegerField stackTraversalMarkField;

+  private static CIntegerField compLevelField;
+
  static {
    VM.registerVMInitializedObserver(new Observer() {
        public void update(Observable o, Object data) {
@@ -113,7 +115,7 @@ public class NMethod extends CodeBlob {
    osrEntryPointField          = type.getAddressField("_osr_entry_point");
    lockCountField              = type.getJIntField("_lock_count");
    stackTraversalMarkField     = type.getCIntegerField("_stack_traversal_mark");
-
+    compLevelField              = type.getCIntegerField("_comp_level");
    pcDescSize = db.lookupType("PcDesc").getSize();
  }

@@ -530,7 +532,7 @@ public class NMethod extends CodeBlob {
    out.println("compile " + holder.getName().asString() + " " +
                OopUtilities.escapeString(method.getName().asString()) + " " +
                method.getSignature().asString() + " " +
-                getEntryBCI());
+                getEntryBCI() + " " + getCompLevel());

  }

@@ -551,4 +553,5 @@ public class NMethod extends CodeBlob {
  private int getHandlerTableOffset() { return (int) handlerTableOffsetField.getValue(addr); }
  private int getNulChkTableOffset()  { return (int) nulChkTableOffsetField .getValue(addr); }
  private int getNMethodEndOffset()   { return (int) nmethodEndOffsetField  .getValue(addr); }
+  private int getCompLevel()          { return (int) compLevelField         .getValue(addr); }
 }
--- a/agent/src/share/classes/sun/jvm/hotspot/compiler/CompileTask.java
+++ b/agent/src/share/classes/sun/jvm/hotspot/compiler/CompileTask.java
@@ -46,10 +46,12 @@ public class CompileTask extends VMObject {
    Type type      = db.lookupType("CompileTask");
    methodField = type.getAddressField("_method");
    osrBciField = new CIntField(type.getCIntegerField("_osr_bci"), 0);
+    compLevelField = new CIntField(type.getCIntegerField("_comp_level"), 0);
  }

  private static AddressField methodField;
  private static CIntField osrBciField;
+  private static CIntField compLevelField;

  public CompileTask(Address addr) {
    super(addr);
@@ -63,4 +65,8 @@ public class CompileTask extends VMObject {
  public int osrBci() {
    return (int)osrBciField.getValue(getAddress());
  }
+
+  public int compLevel() {
+      return (int)compLevelField.getValue(getAddress());
+  }
 }
--- a/src/share/vm/ci/ciEnv.cpp
+++ b/src/share/vm/ci/ciEnv.cpp
@@ -1164,13 +1164,15 @@ void ciEnv::dump_replay_data(outputStream* out) {
  for (int i = 0; i < objects->length(); i++) {
    objects->at(i)->dump_replay_data(out);
  }
-  Method* method = task()->method();
-  int entry_bci = task()->osr_bci();
+  CompileTask* task = this->task();
+  Method* method = task->method();
+  int entry_bci = task->osr_bci();
+  int comp_level = task->comp_level();
  // Klass holder = method->method_holder();
-  out->print_cr("compile %s %s %s %d",
+  out->print_cr("compile %s %s %s %d %d",
                method->klass_name()->as_quoted_ascii(),
                method->name()->as_quoted_ascii(),
                method->signature()->as_quoted_ascii(),
-                entry_bci);
+                entry_bci, comp_level);
  out->flush();
 }
--- a/src/share/vm/ci/ciReplay.cpp
+++ b/src/share/vm/ci/ciReplay.cpp
@@ -369,11 +369,47 @@ class CompileReplay : public StackObj {
    }
  }

-  // compile <klass> <name> <signature> <entry_bci>
+  // validation of comp_level
+  bool is_valid_comp_level(int comp_level) {
+    const int msg_len = 256;
+    char* msg = NULL;
+    if (!is_compile(comp_level)) {
+      msg = NEW_RESOURCE_ARRAY(char, msg_len);
+      jio_snprintf(msg, msg_len, "%d isn't compilation level", comp_level);
+    } else if (!TieredCompilation && (comp_level != CompLevel_highest_tier)) {
+      msg = NEW_RESOURCE_ARRAY(char, msg_len);
+      switch (comp_level) {
+        case CompLevel_simple:
+          jio_snprintf(msg, msg_len, "compilation level %d requires Client VM or TieredCompilation", comp_level);
+          break;
+        case CompLevel_full_optimization:
+          jio_snprintf(msg, msg_len, "compilation level %d requires Server VM", comp_level);
+          break;
+        default:
+          jio_snprintf(msg, msg_len, "compilation level %d requires TieredCompilation", comp_level);
+      }
+    }
+    if (msg != NULL) {
+      report_error(msg);
+      return false;
+    }
+    return true;
+  }
+
+  // compile <klass> <name> <signature> <entry_bci> <comp_level>
  void process_compile(TRAPS) {
    // methodHandle method;
    Method* method = parse_method(CHECK);
    int entry_bci = parse_int("entry_bci");
+    const char* comp_level_label = "comp_level";
+    int comp_level = parse_int(comp_level_label);
+    // old version w/o comp_level
+    if (had_error() && (error_message() == comp_level_label)) {
+      comp_level = CompLevel_full_optimization;
+    }
+    if (!is_valid_comp_level(comp_level)) {
+      return;
+    }
    Klass* k = method->method_holder();
    ((InstanceKlass*)k)->initialize(THREAD);
    if (HAS_PENDING_EXCEPTION) {
@@ -388,12 +424,12 @@ class CompileReplay : public StackObj {
      }
    }
    // Make sure the existence of a prior compile doesn't stop this one
-    nmethod* nm = (entry_bci != InvocationEntryBci) ? method->lookup_osr_nmethod_for(entry_bci, CompLevel_full_optimization, true) : method->code();
+    nmethod* nm = (entry_bci != InvocationEntryBci) ? method->lookup_osr_nmethod_for(entry_bci, comp_level, true) : method->code();
    if (nm != NULL) {
      nm->make_not_entrant();
    }
    replay_state = this;
-    CompileBroker::compile_method(method, entry_bci, CompLevel_full_optimization,
+    CompileBroker::compile_method(method, entry_bci, comp_level,
                                  methodHandle(), 0, "replay", THREAD);
    replay_state = NULL;
    reset();

--- a/src/share/vm/compiler/compileBroker.cpp
+++ b/src/share/vm/compiler/compileBroker.cpp
@@ -1842,6 +1842,8 @@ void CompileBroker::invoke_compiler_on_method(CompileTask* task) {
        }
      }
    }
+    // simulate crash during compilation
+    assert(task->compile_id() != CICrashAt, "just as planned");
  }
  pop_jni_handle_block();


--- a/src/share/vm/runtime/globals.hpp
+++ b/src/share/vm/runtime/globals.hpp
@@ -3248,7 +3248,9 @@ class CommandLineFlags {
          "Fire OutOfMemoryErrors throughout CI for testing the compiler "  \
          "(non-negative value throws OOM after this many CI accesses "     \
          "in each compile)")                                               \
-                                                                            \
+  notproduct(intx, CICrashAt, -1,                                           \
+          "id of compilation to trigger assert in compiler thread for "     \
+          "the purpose of testing, e.g. generation of replay data")         \
  notproduct(bool, CIObjectFactoryVerify, false,                            \
          "enable potentially expensive verification in ciObjectFactory")   \
                                                                            \

--- a/src/share/vm/runtime/vmStructs.cpp
+++ b/src/share/vm/runtime/vmStructs.cpp
@@ -828,6 +828,7 @@ typedef BinaryTreeDictionary<Metablock, FreeList> MetablockTreeDictionary;
  nonstatic_field(nmethod,             _lock_count,                                   jint)                                  \
  nonstatic_field(nmethod,             _stack_traversal_mark,                         long)                                  \
  nonstatic_field(nmethod,             _compile_id,                                   int)                                   \
+  nonstatic_field(nmethod,             _comp_level,                                   int)                                   \
  nonstatic_field(nmethod,             _exception_cache,                              ExceptionCache*)                       \
  nonstatic_field(nmethod,             _marked_for_deoptimization,                    bool)                                  \
                                                                                                                             \

--- a/test/compiler/ciReplay/TestSA.sh
+++ b/test/compiler/ciReplay/TestSA.sh
+#!/bin/sh
+# 
+# Copyright (c) 2013, Oracle and/or its affiliates. All rights reserved.
+# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
+# 
+# This code is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 2 only, as
+# published by the Free Software Foundation.
+# 
+# This code is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+# version 2 for more details (a copy is included in the LICENSE file that
+# accompanied this code).
+# 
+# You should have received a copy of the GNU General Public License version
+# 2 along with this work; if not, write to the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
+# 
+# Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
+# or visit www.oracle.com if you need additional information or have any
+# questions.
+# 
+# 
+
+##
+## @test
+## @bug 8011675
+## @summary testing of ciReplay with using generated by SA replay.txt 
+## @author igor.ignatyev@oracle.com
+## @run shell TestSA.sh
+##
+
+if [ "${TESTSRC}" = "" ]
+then
+  TESTSRC=${PWD}
+  echo "TESTSRC not set.  Using "${TESTSRC}" as default"
+fi
+echo "TESTSRC=${TESTSRC}"
+
+## Adding common setup Variables for running shell tests.
+. ${TESTSRC}/../../test_env.sh
+
+. ${TESTSRC}/common.sh
+
+generate_replay
+
+${MV} ${replay_data} replay_vm.txt    
+
+if [ -z "${core_file}" -o ! -r "${core_file}" ]
+then
+    # skip test if MacOS host isn't configured for core dumping
+    if [ "$OS" = "Darwin" ]
+    then
+        if [ ! -d "/cores" ]
+        then
+            echo TEST SKIPPED: \'/cores\' dir doens\'t exist
+            exit 0
+        fi
+        if [ ! -w "/cores" ]
+        then
+            echo TEST SKIPPED: \'/cores\' dir exists but is not writable
+            exit 0
+        fi
+    fi
+    test_fail 2 "CHECK :: CORE GENERATION" "core wasn't generated on $OS"
+fi
+
+echo "dumpreplaydata -a > ${replay_data}" | \
+        ${JAVA} ${TESTVMOPTS} \
+        -cp ${TESTJAVA}${FS}lib${FS}sa-jdi.jar \
+        sun.jvm.hotspot.CLHSDB  ${JAVA} ${core_file}
+
+if [ ! -s ${replay_data} ]
+then
+    test_fail 1 "CHECK :: REPLAY DATA GENERATION" \
+        "replay data wasn't generated by SA"
+fi
+
+diff --brief ${replay_data} replay_vm.txt
+if [ $? -ne 0 ]
+then
+    echo WARNING: replay.txt from SA != replay.txt from VM
+fi
+
+common_tests 10 
+${VM_TYPE}_tests 20
+
+cleanup
+
+echo TEST PASSED
+
--- a/test/compiler/ciReplay/TestVM.sh
+++ b/test/compiler/ciReplay/TestVM.sh
+#!/bin/sh
+# 
+# Copyright (c) 2013, Oracle and/or its affiliates. All rights reserved.
+# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
+# 
+# This code is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 2 only, as
+# published by the Free Software Foundation.
+# 
+# This code is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+# version 2 for more details (a copy is included in the LICENSE file that
+# accompanied this code).
+# 
+# You should have received a copy of the GNU General Public License version
+# 2 along with this work; if not, write to the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
+# 
+# Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
+# or visit www.oracle.com if you need additional information or have any
+# questions.
+# 
+# 
+
+##
+## @test
+## @bug 8011675
+## @summary testing of ciReplay with using generated by VM replay.txt 
+## @author igor.ignatyev@oracle.com
+## @run shell TestVM.sh
+##
+
+if [ "${TESTSRC}" = "" ]
+then
+  TESTSRC=${PWD}
+  echo "TESTSRC not set.  Using "${TESTSRC}" as default"
+fi
+echo "TESTSRC=${TESTSRC}"
+
+## Adding common setup Variables for running shell tests.
+. ${TESTSRC}/../../test_env.sh
+
+. ${TESTSRC}/common.sh
+
+generate_replay
+
+if [ ! -s ${replay_data} ]
+then
+    test_fail 1 "CHECK :: REPLAY DATA GENERATION" \
+        "replay data wasn't generated by VM"
+fi
+
+common_tests 10
+${VM_TYPE}_tests 20
+
+cleanup
+
+if [ $is_tiered -eq 1 ]
+then
+    stop_level=1
+    while [ $stop_level -le $server_level ]
+    do
+        generate_replay "-XX:TieredStopAtLevel=$stop_level"
+        if [ ! -s ${replay_data} ]
+        then
+            test_fail `expr $stop_level + 30` \
+                    "TIERED LEVEL $stop_level :: REPLAY DATA GENERATION" \
+                    "replay data wasn't generated by VM with stop_level=$stop_level"
+        fi
+        level=`grep "^compile " $replay_data | awk '{print $6}'`
+        if [ $level -gt $stop_level ]
+        then
+            test_fail `expr $stop_level + 40` \
+                    "TIERED LEVEL $stop_level :: COMP_LEVEL VERIFICATION" \
+                    "comp_level in replay[$level] is greater than stop_level[$stop_level]"
+        fi
+        positive_test `expr $stop_level + 50` "TIERED LEVEL $stop_level :: REPLAY" \
+                "-XX:TieredStopAtLevel=$stop_level"
+        stop_level=`expr $stop_level + 1`
+    done
+    cleanup
+fi
+
+echo TEST PASSED
+
--- a/test/compiler/ciReplay/TestVM_no_comp_level.sh
+++ b/test/compiler/ciReplay/TestVM_no_comp_level.sh
+#!/bin/sh
+# 
+# Copyright (c) 2013, Oracle and/or its affiliates. All rights reserved.
+# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
+# 
+# This code is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 2 only, as
+# published by the Free Software Foundation.
+# 
+# This code is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+# version 2 for more details (a copy is included in the LICENSE file that
+# accompanied this code).
+# 
+# You should have received a copy of the GNU General Public License version
+# 2 along with this work; if not, write to the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
+# 
+# Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
+# or visit www.oracle.com if you need additional information or have any
+# questions.
+# 
+# 
+
+##
+## @test
+## @bug 8011675
+## @summary testing of ciReplay with using generated by VM replay.txt w/o comp_level
+## @author igor.ignatyev@oracle.com
+## @run shell TestVM_no_comp_level.sh
+##
+
+if [ "${TESTSRC}" = "" ]
+then
+  TESTSRC=${PWD}
+  echo "TESTSRC not set.  Using "${TESTSRC}" as default"
+fi
+echo "TESTSRC=${TESTSRC}"
+
+## Adding common setup Variables for running shell tests.
+. ${TESTSRC}/../../test_env.sh
+
+. ${TESTSRC}/common.sh
+
+generate_replay
+
+if [ ! -s ${replay_data} ]
+then
+    test_fail 1 "CHECK :: REPLAY DATA GENERATION" \
+            "replay data wasn't generated by VM"
+fi
+
+${CP} ${replay_data} replay_vm.txt
+
+sed 's/^\(compile *[^ ][^ ]* *[^ ][^ ]* [^ ][^ ]* [^ ][^ ]*\).*$/\1/' \
+        replay_vm.txt > ${replay_data}
+
+if [ $client_available -eq 1 ]
+then
+    # tiered is unavailable in client vm, so results w/ flags will be the same as w/o flags
+    negative_test 10 "CLIENT" -client
+fi
+
+if [ $server_available -eq 1 ]
+then
+    positive_test 21 "SERVER :: NON-TIERED" -XX:-TieredCompilation -server
+    positive_test 22 "SERVER :: TIERED" -XX:+TieredCompilation -server
+fi
+
+cleanup
+
+echo TEST PASSED
+
--- a/test/compiler/ciReplay/common.sh
+++ b/test/compiler/ciReplay/common.sh
+#!/bin/sh
+# 
+# Copyright (c) 2013, Oracle and/or its affiliates. All rights reserved.
+# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
+# 
+# This code is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 2 only, as
+# published by the Free Software Foundation.
+# 
+# This code is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+# version 2 for more details (a copy is included in the LICENSE file that
+# accompanied this code).
+# 
+# You should have received a copy of the GNU General Public License version
+# 2 along with this work; if not, write to the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
+# 
+# Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
+# or visit www.oracle.com if you need additional information or have any
+# questions.
+# 
+# 
+
+# $1 - error code
+# $2 - test name
+# $3,.. - decription
+test_fail() {
+    error=$1
+    shift
+    name=$1
+    shift
+    echo "TEST [$name] FAILED:"
+    echo "$@"
+    exit $error
+}
+
+# $@ - additional vm opts
+start_test() {
+    # disable core dump on *nix
+    ulimit -S -c 0
+    # disable core dump on windows
+    VMOPTS="$@ -XX:-CreateMinidumpOnCrash"
+    cmd="${JAVA} ${VMOPTS} -XX:+ReplayCompiles -XX:ReplayDataFile=${replay_data}"
+    echo $cmd
+    $cmd
+    return $?
+}
+
+# $1 - error_code
+# $2 - test name
+# $3,.. - additional vm opts
+positive_test() {
+    error=$1
+    shift
+    name=$1
+    shift
+    VMOPTS="${TESTVMOPTS} $@"
+    echo "POSITIVE TEST [$name]"
+    start_test ${VMOPTS}
+    exit_code=$?
+    if [ ${exit_code} -ne 0 ]
+    then
+        test_fail $error "$name" "exit_code[${exit_code}] != 0 during replay "\
+                "w/ vmopts: ${VMOPTS}"
+    fi
+}
+
+# $1 - error_code
+# $2 - test name
+# $2,.. - additional vm opts
+negative_test() {
+    error=$1
+    shift
+    name=$1
+    shift
+    VMOPTS="${TESTVMOPTS} $@"
+    echo "NEGATIVE TEST [$name]"
+    start_test ${VMOPTS}
+    exit_code=$?
+    if [ ${exit_code} -eq 0 ]
+    then
+        test_fail $error "$name" "exit_code[${exit_code}] == 0 during replay "\
+                "w/ vmopts: ${VMOPTS}"
+    fi
+}
+
+# $1 - initial error_code
+common_tests() {
+    positive_test $1 "COMMON :: THE SAME FLAGS"
+    positive_test `expr $1 + 1` "COMMON :: TIERED" -XX:+TieredCompilation
+}
+
+# $1 - initial error_code
+# $2 - non-tiered comp_level 
+nontiered_tests() {
+    level=`grep "^compile " $replay_data | awk '{print $6}'`
+    # is level available in non-tiere
+    if [ "$level" -eq $2 ]
+    then
+        positive_test $1 "NON-TIERED :: AVAILABLE COMP_LEVEL" \
+                -XX:-TieredCompilation
+    else
+        negative_test `expr $1 + 1` "NON-TIERED :: UNAVAILABLE COMP_LEVEL" \
+        negative_test `expr $1 + 1` "NON-TIERED :: UNAVAILABLE COMP_LEVEL" \
+                -XX:-TieredCompilation
+    fi
+}
+
+# $1 - initial error_code
+client_tests() {
+    # testing in opposite VM
+    if [ $server_available -eq 1 ]
+    then
+        negative_test $1 "SERVER :: NON-TIERED" -XX:-TieredCompilation \
+                -server
+        positive_test `expr $1 + 1` "SERVER :: TIERED" -XX:+TieredCompilation \
+                -server
+    fi
+    nontiered_tests `expr $1 + 2` $client_level 
+}
+
+# $1 - initial error_code
+server_tests() {
+    # testing in opposite VM
+    if [ $client_available -eq 1 ]
+    then
+        # tiered is unavailable in client vm, so results w/ flags will be the same as w/o flags
+        negative_test $1 "CLIENT" -client
+    fi
+    nontiered_tests `expr $1 + 2` $server_level
+}
+
+cleanup() {
+    ${RM} -f core*
+    ${RM} -f replay*.txt
+    ${RM} -f hs_err_pid*.log
+    ${RM} -f test_core
+    ${RM} -f test_replay.txt
+}
+
+JAVA=${TESTJAVA}${FS}bin${FS}java
+
+replay_data=test_replay.txt
+
+${JAVA} ${TESTVMOPTS} -Xinternalversion 2>&1 | grep debug
+
+# Only test fastdebug 
+if [ $? -ne 0 ]
+then
+    echo TEST SKIPPED: product build
+    exit 0
+fi
+
+is_int=`${JAVA} ${TESTVMOPTS} -version 2>&1 | grep -c "interpreted mode"`
+# Not applicable for Xint
+if [ $is_int -ne 0 ]
+then
+    echo TEST SKIPPED: interpreted mode
+    exit 0
+fi
+
+cleanup
+
+client_available=`${JAVA} ${TESTVMOPTS} -client -Xinternalversion 2>&1 | \
+        grep -c Client`
+server_available=`${JAVA} ${TESTVMOPTS} -server -Xinternalversion 2>&1 | \
+        grep -c Server`
+is_tiered=`${JAVA} ${TESTVMOPTS} -XX:+PrintFlagsFinal -version | \
+        grep TieredCompilation | \
+        grep -c true`
+# CompLevel_simple -- C1
+client_level=1
+# CompLevel_full_optimization -- C2 or Shark 
+server_level=4
+
+echo "client_available=$client_available"
+echo "server_available=$server_available"
+echo "is_tiered=$is_tiered"
+
+# crash vm in compiler thread with generation replay data and 'small' dump-file
+# $@ - additional vm opts
+generate_replay() {
+    # enable core dump
+    ulimit -c unlimited
+
+    cmd="${JAVA} ${TESTVMOPTS} $@ \
+            -Xms8m \
+            -Xmx32m \
+            -XX:MetaspaceSize=4m \
+            -XX:MaxMetaspaceSize=16m \
+            -XX:InitialCodeCacheSize=512k \
+            -XX:ReservedCodeCacheSize=4m \
+            -XX:ThreadStackSize=512 \
+            -XX:VMThreadStackSize=512 \
+            -XX:CompilerThreadStackSize=512 \
+            -XX:ParallelGCThreads=1 \
+            -XX:CICompilerCount=1 \
+            -Xcomp \
+            -XX:CICrashAt=1 \
+            -XX:+CreateMinidumpOnCrash \
+            -XX:+DumpReplayDataOnError \
+            -XX:ReplayDataFile=${replay_data} \
+            -version"
+    echo GENERATION OF REPLAY.TXT:
+    echo $cmd
+
+    ${cmd} 2>&1 > crash.out
+    
+    core_locations=`grep -i core crash.out | grep "location:" | \
+            sed -e 's/.*location: //'`
+    rm crash.out 
+    # processing core locations for *nix
+    if [ $OS != "windows" ]
+    then
+        # remove 'or' between '/core.<pid>' and 'core'
+        core_locations=`echo $core_locations | \
+                sed -e 's/\([^ ]*\) or \([^ ]*\)/\1 \2/'`
+        # add <core_path>/core.<pid> core.<pid>
+        core=`echo $core_locations | awk '{print $1}'`
+        dir=`dirname $core`
+        core=`basename $core`
+        if [ -n ${core} ]
+        then
+            core_locations="$core_locations $dir${FS}$core"
+        fi
+        core=`echo $core_locations | awk '{print $2}'`
+        if [ -n ${core} ]
+        then
+            core_locations="$core_locations $dir${FS}$core"
+        fi
+    fi
+
+    echo "LOOKING FOR CORE IN ${core_locations}"
+    for core in $core_locations
+    do
+        if [ -r "$core" ]
+        then
+            core_file=$core
+        fi
+    done
+
+    # core-file was found
+    if [ -n "$core_file" ]
+    then
+        ${MV} "${core_file}" test_core
+        core_file=test_core
+    fi
+
+    ${RM} -f hs_err_pid*.log
+}
+