Update the ENGINE README, and stock it up with extra verbeage for good

measure (including info and instructions on "dynamic" ENGINEs).

README.ENGINE

-
   ENGINE
   ======
 
-  With OpenSSL 0.9.6, a new component has been added to support external 
-  crypto devices, for example accelerator cards.  The component is called
-  ENGINE, and has still a pretty experimental status and almost no
-  documentation.  It's designed to be faily easily extensible by the
-  calling programs.
+  With OpenSSL 0.9.6, a new component was added to support alternative
+  cryptography implementations, most commonly for interfacing with external
+  crypto devices (eg. accelerator cards). This component is called ENGINE,
+  and its presence in OpenSSL 0.9.6 (and subsequent bug-fix releases)
+  caused a little confusion as 0.9.6** releases were rolled in two
+  versions, a "standard" and an "engine" verion. In development for 0.9.7,
+  the ENGINE code has been merged into the main branch and will be present
+  in the standard releases from 0.9.7 forwards.
 
-  There's currently built-in support for the following crypto devices:
+  There are currently built-in ENGINE implementations for the following
+  crypto devices:
 
       o CryptoSwift
       o Compaq Atalla
       o nCipher CHIL
       o Nuron
+      o Broadcom uBSec
+
+  In addition, dynamic binding to external ENGINE implementations is now
+  provided by a special ENGINE called "dynamic". See the "DYNAMIC ENGINE"
+  section below for details.
+
+  At this stage, a number of things are still needed and are being worked on:
+
+      1 Integration of EVP support.
+      2 Configuration support.
+      3 Documentation!
+
+1 With respect to EVP, this relates to support for ciphers and digests in
+  the ENGINE model so that alternative implementations of existing
+  algorithms/modes (or previously unimplemented ones) can be provided by
+  ENGINE implementations.
+
+2 Configuration support currently exists in the ENGINE API itself, in the
+  form of "control commands". These allow an application to expose to the
+  user/admin the set of commands and parameter types a given ENGINE
+  implementation supports, and for an application to directly feed string
+  based input to those ENGINEs, in the form of name-value pairs. This is an
+  extensible way for ENGINEs to define their own "configuration" mechanisms
+  that are specific to a given ENGINE (eg. for a particular hardware
+  device) but that should be consistent across *all* OpenSSL-based
+  applications when they use that ENGINE. Work is in progress (or at least
+  in planning) for supporting these control commands from the CONF (or
+  NCONF) code so that applications using OpenSSL's existing configuration
+  file format can have ENGINE settings specified in much the same way.
+  Presently however, applications must use the ENGINE API itself to provide
+  such functionality. To see first hand the types of commands available
+  with the various compiled-in ENGINEs (see further down for dynamic
+  ENGINEs), use the "engine" openssl utility with full verbosity, ie;
+       openssl engine -vvvv
 
-  A number of things are still needed and are being worked on:
+3 Documentation? Volunteers welcome! The source code is reasonably well
+  self-documenting, but some summaries and usage instructions are needed -
+  moreover, they are needed in the same POD format the existing OpenSSL
+  documentation is provided in. Any complete or incomplete contributions
+  would help make this happen.
 
-      o A better way of handling the methods that are handled by the
-        engines.
-      o Documentation!
+  STABILITY & BUG-REPORTS
+  =======================
 
   What already exists is fairly stable as far as it has been tested, but
-  the test base has been a bit small most of the time.
+  the test base has been a bit small most of the time. For the most part,
+  the vendors of the devices these ENGINEs support have contributed to the
+  development and/or testing of the implementations, and *usually* (with no
+  guarantees) have experience in using the ENGINE support to drive their
+  devices from common OpenSSL-based applications. Bugs and/or inexplicable
+  behaviour in using a specific ENGINE implementation should be sent to the
+  author of that implementation (if it is mentioned in the corresponding C
+  file), and in the case of implementations for commercial hardware
+  devices, also through whatever vendor support channels are available.  If
+  none of this is possible, or the problem seems to be something about the
+  ENGINE API itself (ie. not necessarily specific to a particular ENGINE
+  implementation) then you should mail complete details to the relevant
+  OpenSSL mailing list. For a definition of "complete details", refer to
+  the OpenSSL "README" file. As for which list to send it to;
+
+     openssl-users: if you are *using* the ENGINE abstraction, either in an
+          pre-compiled application or in your own application code.
+
+     openssl-dev: if you are discussing problems with OpenSSL source code.
+
+  USAGE
+  =====
+
+  The default "openssl" ENGINE is always chosen when performing crypto
+  operations unless you specify otherwise. You must actively tell the
+  openssl utility commands to use anything else through a new command line
+  switch called "-engine". Also, if you want to use the ENGINE support in
+  your own code to do something similar, you must likewise explicitly
+  select the ENGINE implementation you want.
+
+  Depending on the type of hardware, system, and configuration, "settings"
+  may need to be applied to an ENGINE for it to function as expected/hoped.
+  The recommended way of doing this is for the application to support
+  ENGINE "control commands" so that each ENGINE implementation can provide
+  whatever configuration primitives it might require and the application
+  can allow the user/admin (and thus the hardware vendor's support desk
+  also) to provide any such input directly to the ENGINE implementation.
+  This way, applications do not need to know anything specific to any
+  device, they only need to provide the means to carry such user/admin
+  input through to the ENGINE in question. Ie. this connects *you* (and
+  your helpdesk) to the specific ENGINE implementation (and device), and
+  allows application authors to not get buried in hassle supporting
+  arbitrary devices they know (and care) nothing about.
+
+  A new "openssl" utility, "openssl engine", has been added in that allows
+  for testing and examination of ENGINE implementations. Basic usage
+  instructions are available by specifying the "-?" command line switch.
+
+  DYNAMIC ENGINES
+  ===============
+
+  The new "dynamic" ENGINE provides a low-overhead way to support ENGINE
+  implementations that aren't pre-compiled and linked into OpenSSL-based
+  applications. This could be because existing compiled-in implementations
+  have known problems and you wish to use a newer version with an existing
+  application. It could equally be because the application (or OpenSSL
+  library) you are using simply doesn't have support for the ENGINE you
+  wish to use, and the ENGINE provider (eg. hardware vendor) is providing
+  you with a self-contained implementation in the form of a shared-library.
+  The other use-case for "dynamic" is with applications that wish to
+  maintain the smallest foot-print possible and so do not link in various
+  ENGINE implementations from OpenSSL, but instead leaves you to provide
+  them, if you want them, in the form of "dynamic"-loadable
+  shared-libraries. It should be possible for hardware vendors to provide
+  their own shared-libraries to support arbitrary hardware to work with
+  applications based on OpenSSL 0.9.7 or later. If you're using an
+  application based on 0.9.7 (or later) and the support you desire is only
+  announced for versions later than the one you need, ask the vendor to
+  backport their ENGINE to the version you need.
+
+  How does "dynamic" work?
+  ------------------------
+    The dynamic ENGINE has a special flag in its implementation such that
+    every time application code asks for the 'dynamic' ENGINE, it in fact
+    gets its own copy of it. As such, multi-threaded code (or code that
+    multiplexes multiple uses of 'dynamic' in a single application in any
+    way at all) does not get confused by 'dynamic' being used to do many
+    independant things. Other ENGINEs typically don't do this so there is
+    only ever 1 ENGINE structure of its type (and reference counts are used
+    to keep order). The dynamic ENGINE itself provides absolutely no
+    cryptographic functionality, and any attempt to "initialise" the ENGINE
+    automatically fails. All it does provide are a few "control commands"
+    that can be used to control how it will load an external ENGINE
+    implementation from a shared-library. To see these control commands,
+    use the command-line;
+
+       openssl engine -vvvv dynamic
+
+    The "SO_PATH" control command should be used to identify the
+    shared-library that contains the ENGINE implementation, and "NO_VCHECK"
+    might possibly be useful if there is a minor version conflict and you
+    (or a vendor helpdesk) is convinced you can safely ignore it.
+    "ENGINE_ID" is probably only needed if a shared-library implements
+    multiple ENGINEs, but if you know the engine id you expect to be using,
+    it doesn't hurt to specify it (and this provides a sanity check if
+    nothing else). "LIST_ADD" is only required if you actually wish the
+    loaded ENGINE to be discoverable by application code later on using the
+    ENGINE's "id". For most applications, this isn't necessary - but some
+    application authors may have nifty reasons for using it. The "LOAD"
+    command is the only one that takes no parameters and is the command
+    that uses the settings from any previous commands to actually *load*
+    the shared-library ENGINE implementation. If this command succeeds, the
+    (copy of the) 'dynamic' ENGINE will magically morph into the ENGINE
+    that has been loaded from the shared-library. As such, any control
+    commands supported by the loaded ENGINE could then be executed as per
+    normal. Eg. if ENGINE "foo" is implemented in the shared-library
+    "libfoo.so" and it supports some special control command "CMD_FOO", the
+    following code would load and use it (NB: obviously this code has no
+    error checking);
+
+       ENGINE *e = ENGINE_by_id("dynamic");
+       ENGINE_ctrl_cmd_string(e, "SO_PATH", "/lib/libfoo.so", 0);
+       ENGINE_ctrl_cmd_string(e, "ENGINE_ID", "foo", 0);
+       ENGINE_ctrl_cmd_string(e, "LOAD", NULL, 0);
+       ENGINE_ctrl_cmd_string(e, "CMD_FOO", "some input data", 0);
+
+    For testing, the "openssl engine" utility can be useful for this sort
+    of thing. For example the above code excerpt would achieve much the
+    same result as;
+
+       openssl engine dynamic \
+                 -pre SO_PATH:/lib/libfoo.so \
+                 -pre ENGINE_ID:foo \
+                 -pre LOAD \
+                 -pre "CMD_FOO:some input data"
+
+    Or to simply see the list of commands supported by the "foo" ENGINE;
+
+       openssl engine -vvvv dynamic \
+                 -pre SO_PATH:/lib/libfoo.so \
+                 -pre ENGINE_ID:foo \
+                 -pre LOAD
+
+    Applications that support the ENGINE API and more specifically, the
+    "control commands" mechanism, will provide some way for you to pass
+    such commands through to ENGINEs. As such, you would select "dynamic"
+    as the ENGINE to use, and the parameters/commands you pass would
+    control the *actual* ENGINE used. Each command is actually a name-value
+    pair and the value can sometimes be omitted (eg. the "LOAD" command).
+    Whilst the syntax demonstrated in "openssl engine" uses a colon to
+    separate the command name from the value, applications may provide
+    their own syntax for making that separation (eg. a win32 registry
+    key-value pair may be used by some applications). The reason for the
+    "-pre" syntax in the "openssl engine" utility is that some commands
+    might be issued to an ENGINE *after* it has been initialised for use.
+    Eg. if an ENGINE implementation requires a smart-card to be inserted
+    during intialisation (or a PIN to be typed, or whatever), there may be
+    a control command you can issue afterwards to "forget" the smart-card
+    so that additional initialisation is no longer possible. In
+    applications such as web-servers, where potentially volatile code may
+    run on the same host system, this may provide some arguable security
+    value. In such a case, the command would be passed to the ENGINE after
+    it has been initialised for use, and so the "-post" switch would be
+    used instead. Applications may provide a different syntax for
+    supporting this distinction, and some may simply not provide it at all
+    ("-pre" is almost always what you're after, in reality).
 
+  How do I build a "dynamic" ENGINE?
+  ----------------------------------
+    This question is trickier - currently OpenSSL bundles various ENGINE
+    implementations that are statically built in, and any application that
+    calls the "ENGINE_load_builtin_engines()" function will automatically
+    have all such ENGINEs available (and occupying memory). Applications
+    that don't call that function have no ENGINEs available like that and
+    would have to use "dynamic" to load any such ENGINE - but on the other
+    hand such applications would only have the memory footprint of any
+    ENGINEs explicitly loaded using user/admin provided control commands.
+    The main advantage of not statically linking ENGINEs and only using
+    "dynamic" for hardare support is that any installation using no
+    "external" ENGINE suffers no unecessary memory footprint from unused
+    ENGINEs. Likewise, installations that do require an ENGINE incur the
+    overheads from only *that* ENGINE once it has been loaded.
 
-  No external crypto device is chosen unless you say so.  You have actively
-  tell the openssl utility commands to use it through a new command line
-  switch called "-engine".  And if you want to use the ENGINE library to
-  do something similar, you must also explicitely choose an external crypto
-  device, or the built-in crypto routines will be used, just as in the
-  default OpenSSL distribution.
+    Sounds good? Maybe, but currently building an ENGINE implementation as
+    a shared-library that can be loaded by "dynamic" isn't automated in
+    OpenSSL's build process. It can be done manually quite easily however.
+    Such a shared-library can either be built with any OpenSSL code it
+    needs statically linked in, or it can link dynamically against OpenSSL
+    if OpenSSL itself is built as a shared library. The instructions are
+    the same in each case, but in the former (statically linked any
+    dependencies on OpenSSL) you must ensure OpenSSL is built with
+    position-independant code ("PIC"). The default OpenSSL compilation may
+    already specify the relevant flags to do this, but you should consult
+    with your compiler documentation if you are in any doubt.
 
+    This example will show building the "atalla" ENGINE in the
+    crypto/engine/ directory as a shared-library for use via the "dynamic"
+    ENGINE.
+    1) "cd" to the crypto/engine/ directory of a pre-compiled OpenSSL
+       source tree.
+    2) Recompile at least one source file so you can see all the compiler
+       flags (and syntax) being used to build normally. Eg;
+           touch hw_atalla.c ; make
+       will rebuild "hw_atalla.o" using all such flags.
+    3) Manually enter the same compilation line to compile the
+       "hw_atalla.c" file but with the following two changes;
+         (a) add "-DENGINE_DYNAMIC_SUPPORT" to the command line switches,
+	 (b) change the output file from "hw_atalla.o" to something new,
+             eg. "tmp_atalla.o"
+    4) Link "tmp_atalla.o" into a shared-library using the top-level
+       OpenSSL libraries to resolve any dependencies. The syntax for doing
+       this depends heavily on your system/compiler and is a nightmare
+       known well to anyone who has worked with shared-library portability
+       before. 'gcc' on Linux, for example, would use the following syntax;
+          gcc -shared -o dyn_atalla.so tmp_atalla.o -L../.. -lcrypto
+    5) Test your shared library using "openssl engine" as explained in the
+       previous section. Eg. from the top-level directory, you might try;
+          apps/openssl engine -vvvv dynamic \
+              -pre SO_PATH:./crypto/engine/dyn_atalla.so -pre LOAD
+       If the shared-library loads successfully, you will see both "-pre"
+       commands marked as "SUCCESS" and the list of control commands
+       displayed (because of "-vvvv") will be the control commands for the
+       *atalla* ENGINE (ie. *not* the 'dynamic' ENGINE). You can also add
+       the "-t" switch to the utility if you want it to try and initialise
+       the atalla ENGINE for use to test any possible hardware/driver
+       issues.
 
   PROBLEMS
   ========
 
-  It seems like the ENGINE part doesn't work too well with Cryptoswift on
-  Win32.  A quick test done right before the release showed that trying
-  "openssl speed -engine cswift" generated errors.  If the DSO gets enabled,
-  an attempt is made to write at memory address 0x00000002.
+  It seems like the ENGINE part doesn't work too well with Cryptoswift on Win32.
+  A quick test done right before the release showed that trying "openssl speed
+  -engine cswift" generated errors. If the DSO gets enabled, an attempt is made
+  to write at memory address 0x00000002.